Nov 03 13:07:43.670742 ip-10-0-131-36 systemd[1]: Starting Kubernetes Kubelet... Nov 03 13:07:44.075681 ip-10-0-131-36 kubenswrapper[2573]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 13:07:44.075681 ip-10-0-131-36 kubenswrapper[2573]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 03 13:07:44.075681 ip-10-0-131-36 kubenswrapper[2573]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 13:07:44.075681 ip-10-0-131-36 kubenswrapper[2573]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 13:07:44.081179 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.081098 2573 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 13:07:44.081179 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.081179 2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 13:07:44.081248 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.081182 2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 13:07:44.081248 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.081187 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 13:07:44.081248 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.081190 2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 13:07:44.081248 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.081192 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 13:07:44.081248 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.081195 2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 13:07:44.081248 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.081198 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 13:07:44.081248 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.081200 2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 13:07:44.081248 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.081203 2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 13:07:44.081248 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.081206 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 13:07:44.081248 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.081209 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 13:07:44.081248 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.081212 2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 13:07:44.081248 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.081215 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 13:07:44.081248 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.081217 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 13:07:44.081248 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.081220 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 13:07:44.081248 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.081223 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 13:07:44.081248 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.081230 2573 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 13:07:44.081248 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.081233 2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 13:07:44.081248 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.081236 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 13:07:44.081248 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.081239 2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 13:07:44.081248 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.081241 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 13:07:44.081742 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.081244 2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 13:07:44.081742 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.081246 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 13:07:44.081742 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.081249 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 13:07:44.081742 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.081251 2573 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 13:07:44.081742 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.081254 2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 13:07:44.081742 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.081257 2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 13:07:44.081742 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.081260 2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 13:07:44.081742 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.081263 2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 13:07:44.081742 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.081266 2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 13:07:44.081742 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.081268 2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 13:07:44.081742 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.081271 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 13:07:44.081742 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.081273 2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 13:07:44.081742 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.081276 2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 13:07:44.081742 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.081279 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 13:07:44.081742 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.081281 2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 13:07:44.081742 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.081284 2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 13:07:44.081742 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.081286 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 13:07:44.081742 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.081288 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 13:07:44.081742 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.081291 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 13:07:44.081742 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.081293 2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 13:07:44.082198 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.081296 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 13:07:44.082198 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.081299 2573 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 13:07:44.082198 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.081301 2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 13:07:44.082198 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.081304 2573 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 13:07:44.082198 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.081306 2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 13:07:44.082198 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.081309 2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 13:07:44.082198 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.081312 2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 13:07:44.082198 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.081314 2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 13:07:44.082198 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.081317 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 13:07:44.082198 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.081319 2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 13:07:44.082198 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.081321 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 13:07:44.082198 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.081324 2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 13:07:44.082198 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.081327 2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 13:07:44.082198 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.081330 2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 13:07:44.082198 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.081333 2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 13:07:44.082198 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.081335 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 13:07:44.082198 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.081338 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 13:07:44.082198 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.081341 2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 13:07:44.082198 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.081343 2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 13:07:44.082651 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.081346 2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 13:07:44.082651 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.081348 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 13:07:44.082651 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.081351 2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 13:07:44.082651 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.081353 2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 13:07:44.082651 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.081356 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 13:07:44.082651 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.081358 2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 13:07:44.082651 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.081361 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 13:07:44.082651 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.081363 2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 13:07:44.082651 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.081366 2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 13:07:44.082651 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.081370 2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 13:07:44.082651 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.081374 2573 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 13:07:44.082651 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.081377 2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 13:07:44.082651 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.081381 2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 13:07:44.082651 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.081398 2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 13:07:44.082651 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.081401 2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 13:07:44.082651 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.081404 2573 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 13:07:44.082651 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.081407 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 13:07:44.082651 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.081409 2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 13:07:44.082651 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.081412 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 13:07:44.083108 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.081415 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 13:07:44.083108 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.081418 2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 13:07:44.083108 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.081421 2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 13:07:44.083108 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.081423 2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 13:07:44.083108 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.081426 2573 feature_gate.go:386] unrecognized feature gate: Example Nov 03 13:07:44.083108 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.081428 2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 13:07:44.083108 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.081431 2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 13:07:44.083108 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.081434 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 13:07:44.083108 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.081437 2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 13:07:44.083108 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.081440 2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 13:07:44.083108 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.081443 2573 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 13:07:44.083108 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.081446 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 13:07:44.083108 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.081449 2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 13:07:44.083108 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.081452 2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 13:07:44.083108 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.081454 2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 13:07:44.083108 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.081458 2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 13:07:44.083108 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.081462 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 13:07:44.083108 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.081464 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 13:07:44.083108 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.081467 2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 13:07:44.083108 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.081470 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 13:07:44.083589 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.081472 2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 13:07:44.083589 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.081475 2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 13:07:44.083589 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.081478 2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 13:07:44.083589 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.081480 2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 13:07:44.083589 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.081484 2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 13:07:44.083589 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.081489 2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 13:07:44.083589 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.082289 2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 13:07:44.083589 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.082297 2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 13:07:44.083589 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.082300 2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 13:07:44.083589 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.082303 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 13:07:44.083589 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.082305 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 13:07:44.083589 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.082308 2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 13:07:44.083589 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.082311 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 13:07:44.083589 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.082314 2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 13:07:44.083589 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.082316 2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 13:07:44.083589 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.082319 2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 13:07:44.083589 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.082322 2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 13:07:44.083589 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.082324 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 13:07:44.083589 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.082328 2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 13:07:44.083589 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.082331 2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 13:07:44.084066 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.082334 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 13:07:44.084066 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.082337 2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 13:07:44.084066 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.082339 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 13:07:44.084066 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.082342 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 13:07:44.084066 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.082345 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 13:07:44.084066 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.082347 2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 13:07:44.084066 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.082350 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 13:07:44.084066 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.082352 2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 13:07:44.084066 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.082355 2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 13:07:44.084066 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.082358 2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 13:07:44.084066 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.082360 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 13:07:44.084066 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.082363 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 13:07:44.084066 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.082365 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 13:07:44.084066 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.082368 2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 13:07:44.084066 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.082370 2573 feature_gate.go:386] unrecognized feature gate: Example Nov 03 13:07:44.084066 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.082373 2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 13:07:44.084066 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.082376 2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 13:07:44.084066 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.082379 2573 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 13:07:44.084066 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.082382 2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 13:07:44.084531 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.082399 2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 13:07:44.084531 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.082402 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 13:07:44.084531 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.082405 2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 13:07:44.084531 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.082408 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 13:07:44.084531 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.082411 2573 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 13:07:44.084531 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.082414 2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 13:07:44.084531 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.082416 2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 13:07:44.084531 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.082419 2573 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 13:07:44.084531 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.082422 2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 13:07:44.084531 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.082425 2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 13:07:44.084531 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.082427 2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 13:07:44.084531 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.082430 2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 13:07:44.084531 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.082433 2573 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 13:07:44.084531 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.082436 2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 13:07:44.084531 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.082438 2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 13:07:44.084531 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.082441 2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 13:07:44.084531 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.082444 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 13:07:44.084531 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.082446 2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 13:07:44.084531 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.082449 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 13:07:44.084531 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.082451 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 13:07:44.085005 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.082453 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 13:07:44.085005 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.082456 2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 13:07:44.085005 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.082458 2573 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 13:07:44.085005 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.082461 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 13:07:44.085005 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.082463 2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 13:07:44.085005 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.082466 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 13:07:44.085005 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.082468 2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 13:07:44.085005 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.082471 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 13:07:44.085005 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.082473 2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 13:07:44.085005 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.082476 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 13:07:44.085005 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.082479 2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 13:07:44.085005 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.082481 2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 13:07:44.085005 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.082484 2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 13:07:44.085005 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.082487 2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 13:07:44.085005 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.082491 2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 13:07:44.085005 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.082495 2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 13:07:44.085005 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.082498 2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 13:07:44.085005 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.082501 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 13:07:44.085005 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.082504 2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 13:07:44.085468 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.082508 2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 13:07:44.085468 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.082511 2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 13:07:44.085468 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.082514 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 13:07:44.085468 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.082517 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 13:07:44.085468 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.082520 2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 13:07:44.085468 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.082522 2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 13:07:44.085468 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.082525 2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 13:07:44.085468 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.082527 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 13:07:44.085468 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.082530 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 13:07:44.085468 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.082533 2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 13:07:44.085468 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.082535 2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 13:07:44.085468 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.082538 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 13:07:44.085468 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.082540 2573 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 13:07:44.085468 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.082543 2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 13:07:44.085468 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.082545 2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 13:07:44.085468 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.082548 2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 13:07:44.085468 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.082550 2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 13:07:44.085468 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.082553 2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 13:07:44.085909 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.082555 2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 13:07:44.085909 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.082558 2573 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 13:07:44.085909 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.082560 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 13:07:44.085909 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.082563 2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 13:07:44.085909 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.082566 2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 13:07:44.085909 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.082569 2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 13:07:44.085909 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.082571 2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 13:07:44.085909 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.082574 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 13:07:44.085909 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.082577 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 13:07:44.085909 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.082579 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 13:07:44.085909 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.082584 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 13:07:44.085909 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.082586 2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 13:07:44.085909 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.082589 2573 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 13:07:44.085909 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.082591 2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 13:07:44.085909 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.082594 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 13:07:44.085909 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.082596 2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 13:07:44.085909 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.083083 2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 13:07:44.085909 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.083089 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 13:07:44.085909 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.083093 2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 13:07:44.085909 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.083096 2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 13:07:44.086407 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.083098 2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 13:07:44.086407 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.083101 2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 13:07:44.086407 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.083104 2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 13:07:44.086407 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.083108 2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 13:07:44.086407 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.083110 2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 13:07:44.086407 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.083113 2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 13:07:44.086407 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.083116 2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 13:07:44.086407 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.083119 2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 13:07:44.086407 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.083121 2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 13:07:44.086407 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.083124 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 13:07:44.086407 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.083127 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 13:07:44.086407 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.083130 2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 13:07:44.086407 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.083132 2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 13:07:44.086407 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.083135 2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 13:07:44.086407 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.083138 2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 13:07:44.086407 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.083141 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 13:07:44.086407 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.083144 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 13:07:44.086407 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.083147 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 13:07:44.086407 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.083150 2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 13:07:44.086407 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.083152 2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 13:07:44.086865 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.083155 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 13:07:44.086865 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.083157 2573 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 13:07:44.086865 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.083160 2573 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 13:07:44.086865 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.083163 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 13:07:44.086865 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.083165 2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 13:07:44.086865 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.083168 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 13:07:44.086865 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.083170 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 13:07:44.086865 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.083173 2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 13:07:44.086865 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.083177 2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 13:07:44.086865 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.083181 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 13:07:44.086865 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.083184 2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 13:07:44.086865 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.083186 2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 13:07:44.086865 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.083190 2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 13:07:44.086865 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.083193 2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 13:07:44.086865 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.083197 2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 13:07:44.086865 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.083200 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 13:07:44.086865 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.083203 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 13:07:44.086865 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.083206 2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 13:07:44.086865 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.083208 2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 13:07:44.087312 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.083210 2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 13:07:44.087312 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.083213 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 13:07:44.087312 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.083215 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 13:07:44.087312 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.083218 2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 13:07:44.087312 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.083221 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 13:07:44.087312 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.083223 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 13:07:44.087312 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.083226 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 13:07:44.087312 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.083232 2573 feature_gate.go:386] unrecognized feature gate: Example Nov 03 13:07:44.087312 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.083235 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 13:07:44.087312 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.083237 2573 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 13:07:44.087312 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.083240 2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 13:07:44.087312 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.083242 2573 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 13:07:44.087312 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.083245 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 13:07:44.087312 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.083248 2573 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 13:07:44.087312 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.083252 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 13:07:44.087312 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.083254 2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 13:07:44.087312 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.083257 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 13:07:44.087312 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.083259 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 13:07:44.087312 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.083262 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 13:07:44.087312 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.083264 2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 13:07:44.087837 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.083267 2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 13:07:44.087837 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.083270 2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 13:07:44.087837 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.083272 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 13:07:44.087837 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.083274 2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 13:07:44.087837 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.083277 2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 13:07:44.087837 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.083279 2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 13:07:44.087837 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.083282 2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 13:07:44.087837 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.083284 2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 13:07:44.087837 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.083287 2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 13:07:44.087837 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.083289 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 13:07:44.087837 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.083292 2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 13:07:44.087837 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.083294 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 13:07:44.087837 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.083297 2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 13:07:44.087837 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.083299 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 13:07:44.087837 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.083302 2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 13:07:44.087837 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.083304 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 13:07:44.087837 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.083307 2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 13:07:44.087837 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.083309 2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 13:07:44.087837 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.083312 2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 13:07:44.087837 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.083315 2573 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 13:07:44.088305 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.083318 2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 13:07:44.088305 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.083321 2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 13:07:44.088305 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.083323 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 13:07:44.088305 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.083326 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 13:07:44.088305 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.083328 2573 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 13:07:44.088305 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.083331 2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 13:07:44.088305 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.083333 2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 13:07:44.088305 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.083336 2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 13:07:44.088305 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.083338 2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 13:07:44.088305 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.083341 2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 13:07:44.088305 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.083343 2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 13:07:44.088305 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.083346 2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 13:07:44.088305 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.083349 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 13:07:44.088305 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.083351 2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 13:07:44.088305 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.083354 2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 13:07:44.088305 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.083357 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 13:07:44.088305 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.083359 2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 13:07:44.088305 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.083362 2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 13:07:44.088305 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.083364 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 13:07:44.088305 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.083367 2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 13:07:44.088793 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.083369 2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 13:07:44.088793 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.083371 2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 13:07:44.088793 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.083374 2573 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 13:07:44.088793 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.083471 2573 flags.go:64] FLAG: --address="0.0.0.0" Nov 03 13:07:44.088793 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.083479 2573 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 03 13:07:44.088793 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.083487 2573 flags.go:64] FLAG: --anonymous-auth="true" Nov 03 13:07:44.088793 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.083493 2573 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 03 13:07:44.088793 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.083500 2573 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 03 13:07:44.088793 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.083505 2573 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 03 13:07:44.088793 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.083511 2573 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 03 13:07:44.088793 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.083516 2573 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 03 13:07:44.088793 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.083521 2573 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 03 13:07:44.088793 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.083525 2573 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 03 13:07:44.088793 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.083528 2573 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 03 13:07:44.088793 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.083532 2573 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 03 13:07:44.088793 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.083536 2573 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 03 13:07:44.088793 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.083538 2573 flags.go:64] FLAG: --cgroup-root="" Nov 03 13:07:44.088793 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.083542 2573 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 03 13:07:44.088793 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.083545 2573 flags.go:64] FLAG: --client-ca-file="" Nov 03 13:07:44.088793 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.083548 2573 flags.go:64] FLAG: --cloud-provider="external" Nov 03 13:07:44.088793 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.083555 2573 flags.go:64] FLAG: --cluster-dns="[]" Nov 03 13:07:44.088793 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.083560 2573 flags.go:64] FLAG: --cluster-domain="" Nov 03 13:07:44.088793 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.083563 2573 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 03 13:07:44.088793 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.083566 2573 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 03 13:07:44.089380 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.083569 2573 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 03 13:07:44.089380 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.083573 2573 flags.go:64] FLAG: --container-log-max-files="5" Nov 03 13:07:44.089380 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.083576 2573 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 03 13:07:44.089380 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.083579 2573 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 03 13:07:44.089380 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.083582 2573 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 03 13:07:44.089380 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.083586 2573 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 03 13:07:44.089380 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.083589 2573 flags.go:64] FLAG: --contention-profiling="false" Nov 03 13:07:44.089380 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.083592 2573 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 03 13:07:44.089380 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.083595 2573 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 03 13:07:44.089380 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.083598 2573 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 03 13:07:44.089380 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.083601 2573 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 03 13:07:44.089380 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.083605 2573 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 03 13:07:44.089380 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.083608 2573 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 03 13:07:44.089380 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.083611 2573 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 03 13:07:44.089380 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.083615 2573 flags.go:64] FLAG: --enable-load-reader="false" Nov 03 13:07:44.089380 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.083618 2573 flags.go:64] FLAG: --enable-server="true" Nov 03 13:07:44.089380 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.083621 2573 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 03 13:07:44.089380 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.083647 2573 flags.go:64] FLAG: --event-burst="100" Nov 03 13:07:44.089380 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.083651 2573 flags.go:64] FLAG: --event-qps="50" Nov 03 13:07:44.089380 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.083656 2573 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 03 13:07:44.089380 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.083660 2573 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 03 13:07:44.089380 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.083663 2573 flags.go:64] FLAG: --eviction-hard="" Nov 03 13:07:44.089380 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.083667 2573 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 03 13:07:44.089380 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.083671 2573 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 03 13:07:44.090042 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.083674 2573 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 03 13:07:44.090042 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.083677 2573 flags.go:64] FLAG: --eviction-soft="" Nov 03 13:07:44.090042 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.083680 2573 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 03 13:07:44.090042 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.083683 2573 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 03 13:07:44.090042 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.083686 2573 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 03 13:07:44.090042 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.083690 2573 flags.go:64] FLAG: --experimental-mounter-path="" Nov 03 13:07:44.090042 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.083693 2573 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 03 13:07:44.090042 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.083696 2573 flags.go:64] FLAG: --fail-swap-on="true" Nov 03 13:07:44.090042 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.083699 2573 flags.go:64] FLAG: --feature-gates="" Nov 03 13:07:44.090042 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.083703 2573 flags.go:64] FLAG: --file-check-frequency="20s" Nov 03 13:07:44.090042 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.083707 2573 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 03 13:07:44.090042 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.083710 2573 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 03 13:07:44.090042 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.083713 2573 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 03 13:07:44.090042 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.083716 2573 flags.go:64] FLAG: --healthz-port="10248" Nov 03 13:07:44.090042 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.083719 2573 flags.go:64] FLAG: --help="false" Nov 03 13:07:44.090042 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.083722 2573 flags.go:64] FLAG: --hostname-override="ip-10-0-131-36.us-east-2.compute.internal" Nov 03 13:07:44.090042 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.083725 2573 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 03 13:07:44.090042 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.083728 2573 flags.go:64] FLAG: --http-check-frequency="20s" Nov 03 13:07:44.090042 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.083731 2573 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 03 13:07:44.090042 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.083735 2573 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 03 13:07:44.090042 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.083738 2573 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 03 13:07:44.090042 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.083741 2573 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 03 13:07:44.090042 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.083747 2573 flags.go:64] FLAG: --image-service-endpoint="" Nov 03 13:07:44.090595 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.083750 2573 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 03 13:07:44.090595 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.083753 2573 flags.go:64] FLAG: --kube-api-burst="100" Nov 03 13:07:44.090595 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.083756 2573 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 03 13:07:44.090595 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.083759 2573 flags.go:64] FLAG: --kube-api-qps="50" Nov 03 13:07:44.090595 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.083763 2573 flags.go:64] FLAG: --kube-reserved="" Nov 03 13:07:44.090595 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.083767 2573 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 03 13:07:44.090595 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.083770 2573 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 03 13:07:44.090595 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.083773 2573 flags.go:64] FLAG: --kubelet-cgroups="" Nov 03 13:07:44.090595 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.083776 2573 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 03 13:07:44.090595 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.083779 2573 flags.go:64] FLAG: --lock-file="" Nov 03 13:07:44.090595 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.083782 2573 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 03 13:07:44.090595 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.083785 2573 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 03 13:07:44.090595 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.083788 2573 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 03 13:07:44.090595 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.083793 2573 flags.go:64] FLAG: --log-json-split-stream="false" Nov 03 13:07:44.090595 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.083796 2573 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 03 13:07:44.090595 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.083800 2573 flags.go:64] FLAG: --log-text-split-stream="false" Nov 03 13:07:44.090595 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.083803 2573 flags.go:64] FLAG: --logging-format="text" Nov 03 13:07:44.090595 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.083806 2573 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 03 13:07:44.090595 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.083809 2573 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 03 13:07:44.090595 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.083812 2573 flags.go:64] FLAG: --manifest-url="" Nov 03 13:07:44.090595 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.083815 2573 flags.go:64] FLAG: --manifest-url-header="" Nov 03 13:07:44.090595 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.083820 2573 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 03 13:07:44.090595 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.083823 2573 flags.go:64] FLAG: --max-open-files="1000000" Nov 03 13:07:44.090595 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.083827 2573 flags.go:64] FLAG: --max-pods="110" Nov 03 13:07:44.090595 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.083830 2573 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 03 13:07:44.091183 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.083833 2573 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 03 13:07:44.091183 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.083836 2573 flags.go:64] FLAG: --memory-manager-policy="None" Nov 03 13:07:44.091183 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.083839 2573 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 03 13:07:44.091183 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.083842 2573 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 03 13:07:44.091183 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.083845 2573 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 03 13:07:44.091183 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.083847 2573 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 03 13:07:44.091183 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.083856 2573 flags.go:64] FLAG: --node-status-max-images="50" Nov 03 13:07:44.091183 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.083859 2573 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 03 13:07:44.091183 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.083862 2573 flags.go:64] FLAG: --oom-score-adj="-999" Nov 03 13:07:44.091183 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.083866 2573 flags.go:64] FLAG: --pod-cidr="" Nov 03 13:07:44.091183 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.083869 2573 flags.go:64] FLAG: --pod-manifest-path="" Nov 03 13:07:44.091183 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.083874 2573 flags.go:64] FLAG: --pod-max-pids="-1" Nov 03 13:07:44.091183 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.083877 2573 flags.go:64] FLAG: --pods-per-core="0" Nov 03 13:07:44.091183 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.083880 2573 flags.go:64] FLAG: --port="10250" Nov 03 13:07:44.091183 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.083883 2573 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 03 13:07:44.091183 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.083886 2573 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-013559e5d3bed1ebf" Nov 03 13:07:44.091183 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.083889 2573 flags.go:64] FLAG: --qos-reserved="" Nov 03 13:07:44.091183 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.083892 2573 flags.go:64] FLAG: --read-only-port="10255" Nov 03 13:07:44.091183 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.083895 2573 flags.go:64] FLAG: --register-node="true" Nov 03 13:07:44.091183 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.083898 2573 flags.go:64] FLAG: --register-with-taints="" Nov 03 13:07:44.091183 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.083902 2573 flags.go:64] FLAG: --registry-burst="10" Nov 03 13:07:44.091183 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.083905 2573 flags.go:64] FLAG: --registry-qps="5" Nov 03 13:07:44.091183 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.083908 2573 flags.go:64] FLAG: --reserved-cpus="" Nov 03 13:07:44.091183 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.083911 2573 flags.go:64] FLAG: --reserved-memory="" Nov 03 13:07:44.091183 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.083919 2573 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 03 13:07:44.091183 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.083923 2573 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 03 13:07:44.091797 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.083926 2573 flags.go:64] FLAG: --rotate-certificates="false" Nov 03 13:07:44.091797 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.083929 2573 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 03 13:07:44.091797 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.083932 2573 flags.go:64] FLAG: --runonce="false" Nov 03 13:07:44.091797 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.083935 2573 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 03 13:07:44.091797 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.083938 2573 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 03 13:07:44.091797 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.083941 2573 flags.go:64] FLAG: --seccomp-default="false" Nov 03 13:07:44.091797 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.083944 2573 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 03 13:07:44.091797 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.083947 2573 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 03 13:07:44.091797 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.083950 2573 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 03 13:07:44.091797 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.083953 2573 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 03 13:07:44.091797 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.083956 2573 flags.go:64] FLAG: --storage-driver-password="root" Nov 03 13:07:44.091797 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.083959 2573 flags.go:64] FLAG: --storage-driver-secure="false" Nov 03 13:07:44.091797 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.083964 2573 flags.go:64] FLAG: --storage-driver-table="stats" Nov 03 13:07:44.091797 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.083967 2573 flags.go:64] FLAG: --storage-driver-user="root" Nov 03 13:07:44.091797 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.083969 2573 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 03 13:07:44.091797 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.083973 2573 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 03 13:07:44.091797 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.083976 2573 flags.go:64] FLAG: --system-cgroups="" Nov 03 13:07:44.091797 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.083980 2573 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 03 13:07:44.091797 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.083986 2573 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 03 13:07:44.091797 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.083989 2573 flags.go:64] FLAG: --tls-cert-file="" Nov 03 13:07:44.091797 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.083992 2573 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 03 13:07:44.091797 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.083996 2573 flags.go:64] FLAG: --tls-min-version="" Nov 03 13:07:44.091797 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.083999 2573 flags.go:64] FLAG: --tls-private-key-file="" Nov 03 13:07:44.091797 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.084002 2573 flags.go:64] FLAG: --topology-manager-policy="none" Nov 03 13:07:44.091797 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.084004 2573 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 03 13:07:44.092371 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.084007 2573 flags.go:64] FLAG: --topology-manager-scope="container" Nov 03 13:07:44.092371 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.084010 2573 flags.go:64] FLAG: --v="2" Nov 03 13:07:44.092371 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.084015 2573 flags.go:64] FLAG: --version="false" Nov 03 13:07:44.092371 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.084019 2573 flags.go:64] FLAG: --vmodule="" Nov 03 13:07:44.092371 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.084023 2573 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 03 13:07:44.092371 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.084027 2573 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 03 13:07:44.092371 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.084165 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 13:07:44.092371 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.084169 2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 13:07:44.092371 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.084173 2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 13:07:44.092371 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.084190 2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 13:07:44.092371 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.084194 2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 13:07:44.092371 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.084197 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 13:07:44.092371 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.084201 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 13:07:44.092371 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.084204 2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 13:07:44.092371 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.084206 2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 13:07:44.092371 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.084209 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 13:07:44.092371 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.084212 2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 13:07:44.092371 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.084215 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 13:07:44.092371 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.084217 2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 13:07:44.092371 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.084221 2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 13:07:44.092371 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.084224 2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 13:07:44.092912 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.084226 2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 13:07:44.092912 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.084229 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 13:07:44.092912 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.084232 2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 13:07:44.092912 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.084236 2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 13:07:44.092912 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.084239 2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 13:07:44.092912 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.084242 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 13:07:44.092912 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.084245 2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 13:07:44.092912 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.084247 2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 13:07:44.092912 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.084250 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 13:07:44.092912 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.084252 2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 13:07:44.092912 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.084255 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 13:07:44.092912 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.084258 2573 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 13:07:44.092912 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.084260 2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 13:07:44.092912 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.084263 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 13:07:44.092912 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.084266 2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 13:07:44.092912 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.084269 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 13:07:44.092912 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.084272 2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 13:07:44.092912 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.084275 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 13:07:44.092912 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.084277 2573 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 13:07:44.092912 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.084280 2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 13:07:44.093423 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.084282 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 13:07:44.093423 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.084285 2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 13:07:44.093423 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.084288 2573 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 13:07:44.093423 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.084290 2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 13:07:44.093423 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.084293 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 13:07:44.093423 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.084295 2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 13:07:44.093423 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.084297 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 13:07:44.093423 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.084300 2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 13:07:44.093423 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.084302 2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 13:07:44.093423 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.084305 2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 13:07:44.093423 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.084309 2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 13:07:44.093423 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.084311 2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 13:07:44.093423 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.084314 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 13:07:44.093423 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.084317 2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 13:07:44.093423 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.084319 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 13:07:44.093423 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.084323 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 13:07:44.093423 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.084325 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 13:07:44.093423 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.084328 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 13:07:44.093423 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.084331 2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 13:07:44.093423 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.084333 2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 13:07:44.093894 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.084336 2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 13:07:44.093894 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.084338 2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 13:07:44.093894 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.084341 2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 13:07:44.093894 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.084343 2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 13:07:44.093894 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.084346 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 13:07:44.093894 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.084349 2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 13:07:44.093894 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.084351 2573 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 13:07:44.093894 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.084354 2573 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 13:07:44.093894 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.084356 2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 13:07:44.093894 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.084359 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 13:07:44.093894 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.084362 2573 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 13:07:44.093894 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.084364 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 13:07:44.093894 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.084367 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 13:07:44.093894 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.084370 2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 13:07:44.093894 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.084372 2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 13:07:44.093894 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.084375 2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 13:07:44.093894 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.084378 2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 13:07:44.093894 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.084380 2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 13:07:44.093894 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.084383 2573 feature_gate.go:386] unrecognized feature gate: Example Nov 03 13:07:44.093894 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.084398 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 13:07:44.094372 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.084400 2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 13:07:44.094372 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.084404 2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 13:07:44.094372 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.084410 2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 13:07:44.094372 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.084414 2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 13:07:44.094372 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.084418 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 13:07:44.094372 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.084421 2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 13:07:44.094372 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.084424 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 13:07:44.094372 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.084428 2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 13:07:44.094372 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.084430 2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 13:07:44.094372 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.084433 2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 13:07:44.094372 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.084436 2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 13:07:44.094372 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.084438 2573 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 13:07:44.094372 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.084441 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 13:07:44.094372 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.084443 2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 13:07:44.094372 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.084446 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 13:07:44.094372 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.084449 2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 13:07:44.094372 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.084451 2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 13:07:44.094372 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.084454 2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 13:07:44.094807 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.084456 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 13:07:44.094807 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.084459 2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 13:07:44.094807 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.084461 2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 13:07:44.094807 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.084464 2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 13:07:44.094807 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.084466 2573 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 13:07:44.094807 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.084469 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 13:07:44.094807 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.084472 2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 13:07:44.094807 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.084475 2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 13:07:44.094807 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.084478 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 13:07:44.094807 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.084481 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 13:07:44.094807 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.084483 2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 13:07:44.094807 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.084486 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 13:07:44.094807 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.084488 2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 13:07:44.094807 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.084514 2573 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 13:07:44.094807 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.084552 2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 13:07:44.094807 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.084556 2573 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 13:07:44.095182 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.084559 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 13:07:44.095182 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.084563 2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 13:07:44.095182 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.084566 2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 13:07:44.095182 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.084568 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 13:07:44.095182 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.084571 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 13:07:44.095182 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.084574 2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 13:07:44.095182 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.084577 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 13:07:44.095182 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.084579 2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 13:07:44.095182 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.084582 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 13:07:44.095182 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.084584 2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 13:07:44.095182 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.084587 2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 13:07:44.095182 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.084590 2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 13:07:44.095182 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.084592 2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 13:07:44.095182 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.084595 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 13:07:44.095182 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.084597 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 13:07:44.095182 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.084599 2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 13:07:44.095182 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.084602 2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 13:07:44.095182 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.084605 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 13:07:44.095182 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.084607 2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 13:07:44.095641 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.084610 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 13:07:44.095641 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.084612 2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 13:07:44.095641 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.084615 2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 13:07:44.095641 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.084618 2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 13:07:44.095641 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.084621 2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 13:07:44.095641 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.084623 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 13:07:44.095641 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.084626 2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 13:07:44.095641 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.084628 2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 13:07:44.095641 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.084631 2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 13:07:44.095641 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.084633 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 13:07:44.095641 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.084636 2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 13:07:44.095641 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.084638 2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 13:07:44.095641 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.084641 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 13:07:44.095641 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.084643 2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 13:07:44.095641 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.084647 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 13:07:44.095641 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.084649 2573 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 13:07:44.095641 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.084652 2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 13:07:44.095641 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.084654 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 13:07:44.095641 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.084657 2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 13:07:44.095641 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.084660 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 13:07:44.096111 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.084662 2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 13:07:44.096111 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.084665 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 13:07:44.096111 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.084668 2573 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 13:07:44.096111 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.084670 2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 13:07:44.096111 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.084673 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 13:07:44.096111 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.084675 2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 13:07:44.096111 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.084678 2573 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 13:07:44.096111 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.084680 2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 13:07:44.096111 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.084683 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 13:07:44.096111 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.084685 2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 13:07:44.096111 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.084688 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 13:07:44.096111 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.084690 2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 13:07:44.096111 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.084693 2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 13:07:44.096111 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.084695 2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 13:07:44.096111 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.084698 2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 13:07:44.096111 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.084700 2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 13:07:44.096111 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.084703 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 13:07:44.096111 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.084706 2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 13:07:44.096111 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.084709 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 13:07:44.096111 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.084711 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 13:07:44.096591 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.084714 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 13:07:44.096591 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.084716 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 13:07:44.096591 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.084719 2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 13:07:44.096591 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.084721 2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 13:07:44.096591 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.084724 2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 13:07:44.096591 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.084726 2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 13:07:44.096591 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.084730 2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 13:07:44.096591 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.084733 2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 13:07:44.096591 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.084736 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 13:07:44.096591 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.084738 2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 13:07:44.096591 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.084741 2573 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 13:07:44.096591 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.084744 2573 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 13:07:44.096591 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.084746 2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 13:07:44.096591 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.084749 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 13:07:44.096591 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.084751 2573 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 13:07:44.096591 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.084754 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 13:07:44.096591 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.084756 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 13:07:44.096591 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.084759 2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 13:07:44.096591 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.084761 2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 13:07:44.096591 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.084764 2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 13:07:44.097144 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.084768 2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 13:07:44.097144 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.084771 2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 13:07:44.097144 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.084773 2573 feature_gate.go:386] unrecognized feature gate: Example Nov 03 13:07:44.097144 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.084776 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 13:07:44.097144 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.084778 2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 13:07:44.097144 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.084782 2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 13:07:44.097144 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.084786 2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 13:07:44.097144 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.084790 2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 13:07:44.097144 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.084793 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 13:07:44.097144 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.084797 2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 13:07:44.097144 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.084801 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 13:07:44.097144 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.084804 2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 13:07:44.097144 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.084806 2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 13:07:44.097144 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.084809 2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 13:07:44.097144 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.084812 2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 13:07:44.097144 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.084815 2573 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 13:07:44.097144 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.084817 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 13:07:44.097144 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.084820 2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 13:07:44.097679 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.084824 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 13:07:44.097679 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.084827 2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 13:07:44.097679 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.084829 2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 13:07:44.097679 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.084832 2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 13:07:44.097679 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.084834 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 13:07:44.097679 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.084837 2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 13:07:44.097679 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.084839 2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 13:07:44.097679 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.091958 2573 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 03 13:07:44.097679 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.091975 2573 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 03 13:07:44.097679 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.091993 2573 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 03 13:07:44.097679 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.091998 2573 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 03 13:07:44.097679 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.092087 2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 13:07:44.097679 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.092094 2573 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 13:07:44.097679 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.092097 2573 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 13:07:44.097679 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.092100 2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 13:07:44.097679 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.092103 2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 13:07:44.097679 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.092106 2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 13:07:44.097679 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.092109 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 13:07:44.097679 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.092112 2573 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 13:07:44.097679 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.092115 2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 13:07:44.098175 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.092118 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 13:07:44.098175 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.092121 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 13:07:44.098175 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.092123 2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 13:07:44.098175 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.092132 2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 13:07:44.098175 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.092136 2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 13:07:44.098175 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.092141 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 13:07:44.098175 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.092145 2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 13:07:44.098175 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.092149 2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 13:07:44.098175 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.092153 2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 13:07:44.098175 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.092155 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 13:07:44.098175 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.092158 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 13:07:44.098175 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.092160 2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 13:07:44.098175 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.092163 2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 13:07:44.098175 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.092165 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 13:07:44.098175 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.092168 2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 13:07:44.098175 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.092172 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 13:07:44.098175 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.092174 2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 13:07:44.098175 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.092177 2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 13:07:44.098175 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.092179 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 13:07:44.098175 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.092182 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 13:07:44.098669 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.092185 2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 13:07:44.098669 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.092188 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 13:07:44.098669 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.092190 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 13:07:44.098669 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.092193 2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 13:07:44.098669 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.092196 2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 13:07:44.098669 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.092198 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 13:07:44.098669 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.092201 2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 13:07:44.098669 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.092203 2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 13:07:44.098669 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.092206 2573 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 13:07:44.098669 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.092208 2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 13:07:44.098669 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.092211 2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 13:07:44.098669 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.092214 2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 13:07:44.098669 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.092219 2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 13:07:44.098669 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.092223 2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 13:07:44.098669 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.092227 2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 13:07:44.098669 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.092232 2573 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 13:07:44.098669 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.092242 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 13:07:44.098669 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.092245 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 13:07:44.098669 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.092248 2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 13:07:44.099117 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.092250 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 13:07:44.099117 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.092253 2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 13:07:44.099117 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.092255 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 13:07:44.099117 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.092259 2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 13:07:44.099117 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.092263 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 13:07:44.099117 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.092266 2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 13:07:44.099117 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.092269 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 13:07:44.099117 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.092272 2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 13:07:44.099117 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.092274 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 13:07:44.099117 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.092277 2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 13:07:44.099117 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.092279 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 13:07:44.099117 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.092282 2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 13:07:44.099117 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.092285 2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 13:07:44.099117 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.092288 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 13:07:44.099117 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.092291 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 13:07:44.099117 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.092295 2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 13:07:44.099117 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.092299 2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 13:07:44.099117 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.092303 2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 13:07:44.099117 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.092308 2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 13:07:44.099564 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.092312 2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 13:07:44.099564 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.092314 2573 feature_gate.go:386] unrecognized feature gate: Example Nov 03 13:07:44.099564 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.092317 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 13:07:44.099564 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.092319 2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 13:07:44.099564 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.092322 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 13:07:44.099564 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.092325 2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 13:07:44.099564 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.092327 2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 13:07:44.099564 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.092330 2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 13:07:44.099564 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.092332 2573 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 13:07:44.099564 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.092335 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 13:07:44.099564 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.092337 2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 13:07:44.099564 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.092346 2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 13:07:44.099564 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.092348 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 13:07:44.099564 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.092351 2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 13:07:44.099564 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.092353 2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 13:07:44.099564 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.092358 2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 13:07:44.099564 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.092361 2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 13:07:44.099564 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.092365 2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 13:07:44.099564 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.092369 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 13:07:44.099564 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.092372 2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 13:07:44.100019 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.092375 2573 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 13:07:44.100019 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.092380 2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 13:07:44.100019 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.092398 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 13:07:44.100019 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.092403 2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 13:07:44.100019 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.092406 2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 13:07:44.100019 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.092410 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 13:07:44.100019 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.092412 2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 13:07:44.100019 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.092415 2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 13:07:44.100019 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.092418 2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 13:07:44.100019 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.092420 2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 13:07:44.100019 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.092423 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 13:07:44.100019 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.092425 2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 13:07:44.100019 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.092428 2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 13:07:44.100019 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.092430 2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 13:07:44.100019 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.092433 2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 13:07:44.100019 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.092436 2573 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 13:07:44.100019 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.092439 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 13:07:44.100019 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.092441 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 13:07:44.100019 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.092443 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 13:07:44.100476 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.092477 2573 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 13:07:44.100476 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.092698 2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 13:07:44.100476 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.092704 2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 13:07:44.100476 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.092707 2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 13:07:44.100476 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.092711 2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 13:07:44.100476 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.092731 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 13:07:44.100476 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.092735 2573 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 13:07:44.100476 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.092738 2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 13:07:44.100476 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.092741 2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 13:07:44.100476 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.092743 2573 feature_gate.go:386] unrecognized feature gate: Example Nov 03 13:07:44.100476 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.092746 2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 13:07:44.100476 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.092748 2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 13:07:44.100476 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.092751 2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 13:07:44.100476 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.092754 2573 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 13:07:44.100476 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.092756 2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 13:07:44.100476 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.092759 2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 13:07:44.100893 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.092762 2573 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 13:07:44.100893 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.092764 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 13:07:44.100893 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.092767 2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 13:07:44.100893 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.092769 2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 13:07:44.100893 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.092772 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 13:07:44.100893 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.092774 2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 13:07:44.100893 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.092777 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 13:07:44.100893 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.092779 2573 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 13:07:44.100893 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.092782 2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 13:07:44.100893 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.092786 2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 13:07:44.100893 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.092790 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 13:07:44.100893 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.092797 2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 13:07:44.100893 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.092803 2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 13:07:44.100893 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.092808 2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 13:07:44.100893 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.092811 2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 13:07:44.100893 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.092814 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 13:07:44.100893 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.092817 2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 13:07:44.100893 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.092819 2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 13:07:44.100893 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.092822 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 13:07:44.101609 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.092825 2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 13:07:44.101609 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.092827 2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 13:07:44.101609 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.092830 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 13:07:44.101609 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.092839 2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 13:07:44.101609 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.092842 2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 13:07:44.101609 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.092845 2573 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 13:07:44.101609 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.092847 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 13:07:44.101609 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.092850 2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 13:07:44.101609 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.092853 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 13:07:44.101609 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.092856 2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 13:07:44.101609 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.092858 2573 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 13:07:44.101609 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.092861 2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 13:07:44.101609 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.092863 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 13:07:44.101609 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.092866 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 13:07:44.101609 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.092868 2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 13:07:44.101609 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.092872 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 13:07:44.101609 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.092876 2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 13:07:44.101609 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.092880 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 13:07:44.101609 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.092884 2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 13:07:44.101609 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.092888 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 13:07:44.102112 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.092891 2573 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 13:07:44.102112 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.092893 2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 13:07:44.102112 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.092896 2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 13:07:44.102112 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.092898 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 13:07:44.102112 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.092901 2573 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 13:07:44.102112 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.092903 2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 13:07:44.102112 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.092906 2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 13:07:44.102112 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.092908 2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 13:07:44.102112 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.092911 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 13:07:44.102112 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.092913 2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 13:07:44.102112 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.092916 2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 13:07:44.102112 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.092919 2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 13:07:44.102112 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.092922 2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 13:07:44.102112 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.092924 2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 13:07:44.102112 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.092927 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 13:07:44.102112 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.092929 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 13:07:44.102112 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.092938 2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 13:07:44.102112 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.092940 2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 13:07:44.102112 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.092943 2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 13:07:44.102638 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.092945 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 13:07:44.102638 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.092955 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 13:07:44.102638 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.092960 2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 13:07:44.102638 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.092964 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 13:07:44.102638 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.092968 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 13:07:44.102638 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.092971 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 13:07:44.102638 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.092973 2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 13:07:44.102638 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.092976 2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 13:07:44.102638 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.092979 2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 13:07:44.102638 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.092981 2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 13:07:44.102638 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.092984 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 13:07:44.102638 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.092986 2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 13:07:44.102638 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.092989 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 13:07:44.102638 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.092991 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 13:07:44.102638 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.092993 2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 13:07:44.102638 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.092996 2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 13:07:44.102638 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.092998 2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 13:07:44.102638 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.093001 2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 13:07:44.102638 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.093003 2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 13:07:44.103080 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.093006 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 13:07:44.103080 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.093008 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 13:07:44.103080 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.093011 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 13:07:44.103080 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.093013 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 13:07:44.103080 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.093015 2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 13:07:44.103080 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.093019 2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 13:07:44.103080 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.093022 2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 13:07:44.103080 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.093024 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 13:07:44.103080 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.093027 2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 13:07:44.103080 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.093029 2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 13:07:44.103080 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.093032 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 13:07:44.103080 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.093048 2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 13:07:44.103080 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.093052 2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 13:07:44.103080 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.093055 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 13:07:44.103080 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.093080 2573 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 13:07:44.103080 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.093132 2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 13:07:44.103756 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.093137 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 13:07:44.103756 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.093139 2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 13:07:44.103756 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.093142 2573 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 13:07:44.103756 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.093145 2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 13:07:44.103756 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.093147 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 13:07:44.103756 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.093150 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 13:07:44.103756 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.093152 2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 13:07:44.103756 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.093155 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 13:07:44.103756 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.093157 2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 13:07:44.103756 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.093160 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 13:07:44.103756 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.093163 2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 13:07:44.103756 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.093165 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 13:07:44.103756 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.093168 2573 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 13:07:44.103756 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.093170 2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 13:07:44.103756 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.093173 2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 13:07:44.103756 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.093175 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 13:07:44.103756 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.093178 2573 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 13:07:44.103756 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.093180 2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 13:07:44.103756 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.093183 2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 13:07:44.103756 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.093185 2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 13:07:44.104269 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.093188 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 13:07:44.104269 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.093194 2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 13:07:44.104269 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.093198 2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 13:07:44.104269 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.093202 2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 13:07:44.104269 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.093209 2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 13:07:44.104269 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.093213 2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 13:07:44.104269 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.093216 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 13:07:44.104269 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.093219 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 13:07:44.104269 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.093228 2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 13:07:44.104269 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.093230 2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 13:07:44.104269 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.093233 2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 13:07:44.104269 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.093236 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 13:07:44.104269 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.093239 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 13:07:44.104269 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.093241 2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 13:07:44.104269 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.093244 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 13:07:44.104269 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.093246 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 13:07:44.104269 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.093249 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 13:07:44.104269 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.093252 2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 13:07:44.104269 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.093254 2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 13:07:44.104768 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.093256 2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 13:07:44.104768 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.093259 2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 13:07:44.104768 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.093261 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 13:07:44.104768 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.093264 2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 13:07:44.104768 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.093266 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 13:07:44.104768 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.093269 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 13:07:44.104768 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.093271 2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 13:07:44.104768 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.093274 2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 13:07:44.104768 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.093278 2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 13:07:44.104768 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.093282 2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 13:07:44.104768 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.093287 2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 13:07:44.104768 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.093291 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 13:07:44.104768 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.093295 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 13:07:44.104768 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.093297 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 13:07:44.104768 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.093302 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 13:07:44.104768 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.093304 2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 13:07:44.104768 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.093307 2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 13:07:44.104768 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.093311 2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 13:07:44.104768 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.093314 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 13:07:44.105258 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.093317 2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 13:07:44.105258 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.093319 2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 13:07:44.105258 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.093321 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 13:07:44.105258 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.093330 2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 13:07:44.105258 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.093333 2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 13:07:44.105258 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.093335 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 13:07:44.105258 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.093338 2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 13:07:44.105258 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.093341 2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 13:07:44.105258 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.093343 2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 13:07:44.105258 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.093346 2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 13:07:44.105258 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.093349 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 13:07:44.105258 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.093352 2573 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 13:07:44.105258 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.093355 2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 13:07:44.105258 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.093359 2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 13:07:44.105258 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.093363 2573 feature_gate.go:386] unrecognized feature gate: Example Nov 03 13:07:44.105258 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.093368 2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 13:07:44.105258 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.093372 2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 13:07:44.105258 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.093376 2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 13:07:44.105258 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.093379 2573 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 13:07:44.105258 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.093381 2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 13:07:44.105895 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.093396 2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 13:07:44.105895 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.093399 2573 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 13:07:44.105895 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.093402 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 13:07:44.105895 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.093404 2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 13:07:44.105895 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.093407 2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 13:07:44.105895 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.093409 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 13:07:44.105895 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.093412 2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 13:07:44.105895 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.093416 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 13:07:44.105895 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.093419 2573 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 13:07:44.105895 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.093421 2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 13:07:44.105895 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.093427 2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 13:07:44.105895 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.093430 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 13:07:44.105895 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.093433 2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 13:07:44.105895 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.093439 2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 13:07:44.105895 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.093445 2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 13:07:44.105895 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.093450 2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 13:07:44.105895 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.093461 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 13:07:44.105895 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.093464 2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 13:07:44.105895 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.093466 2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 13:07:44.106381 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.093469 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 13:07:44.106381 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.093472 2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 13:07:44.106381 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.093474 2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 13:07:44.106381 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.093478 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 13:07:44.106381 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.093481 2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 13:07:44.106381 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.093484 2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 13:07:44.106381 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.093486 2573 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 13:07:44.106381 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:44.093489 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 13:07:44.106381 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.093542 2573 server.go:957] "Client rotation is on, will bootstrap in background" Nov 03 13:07:44.106381 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.095630 2573 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Nov 03 13:07:44.106381 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.096397 2573 server.go:1014] "Starting client certificate rotation" Nov 03 13:07:44.106381 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.096505 2573 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 03 13:07:44.106381 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.096539 2573 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 03 13:07:44.106381 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.098567 2573 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 13:07:44.106381 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.103376 2573 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 13:07:44.114213 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.114193 2573 log.go:25] "Validated CRI v1 runtime API" Nov 03 13:07:44.118663 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.118646 2573 log.go:25] "Validated CRI v1 image API" Nov 03 13:07:44.119499 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.119472 2573 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 03 13:07:44.125115 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.125093 2573 fs.go:135] Filesystem UUIDs: map[11ac45d8-f758-4da3-834f-f7d549fb1720:/dev/nvme0n1p4 6780481b-d8e9-45eb-9edc-0ebc30424bb0:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2] Nov 03 13:07:44.125185 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.125113 2573 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Nov 03 13:07:44.130761 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.130648 2573 manager.go:217] Machine: {Timestamp:2026-11-03 13:07:44.129521962 +0000 UTC m=+0.340741239 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3126792 MemoryCapacity:33164341248 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec254c61e28ea61f2656b603524d3067 SystemUUID:ec254c61-e28e-a61f-2656-b603524d3067 BootID:2b02f133-fa94-4f95-b9a7-98c3dbc870e5 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:d0:69:7a:87 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:d0:69:7a:87 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:56:36:d1:4f:c0:a5 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164341248 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 03 13:07:44.131258 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.131248 2573 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 03 13:07:44.131353 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.131337 2573 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 03 13:07:44.132811 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.132786 2573 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 03 13:07:44.132972 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.132812 2573 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-131-36.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 03 13:07:44.133014 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.132977 2573 topology_manager.go:143] "Creating topology manager with none policy" Nov 03 13:07:44.133014 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.132985 2573 container_manager_linux.go:311] "Creating device plugin manager" Nov 03 13:07:44.133066 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.133015 2573 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 03 13:07:44.133066 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.133033 2573 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 03 13:07:44.133231 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.133214 2573 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 03 13:07:44.134000 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.133973 2573 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 03 13:07:44.137580 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.137568 2573 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 03 13:07:44.137716 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.137706 2573 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 03 13:07:44.137751 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.137740 2573 kubelet.go:493] "Attempting to sync node with API server" Nov 03 13:07:44.137778 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.137752 2573 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 03 13:07:44.137778 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.137763 2573 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 03 13:07:44.137778 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.137772 2573 kubelet.go:405] "Adding apiserver pod source" Nov 03 13:07:44.137873 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.137780 2573 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 03 13:07:44.138115 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.138104 2573 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 03 13:07:44.143565 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.143545 2573 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 03 13:07:44.148667 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.148651 2573 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 03 13:07:44.149294 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.149282 2573 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 03 13:07:44.149332 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.149313 2573 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 03 13:07:44.149497 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.149486 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 03 13:07:44.149534 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.149503 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 03 13:07:44.149534 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.149510 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 03 13:07:44.149534 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.149516 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 03 13:07:44.149534 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.149522 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 03 13:07:44.149534 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.149528 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 03 13:07:44.149859 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.149850 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 03 13:07:44.149859 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.149859 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 03 13:07:44.149911 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.149866 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 03 13:07:44.149911 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.149872 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 03 13:07:44.149911 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.149880 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 03 13:07:44.149911 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.149889 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 03 13:07:44.149911 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.149910 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 03 13:07:44.150055 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.149915 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 03 13:07:44.152544 ip-10-0-131-36 kubenswrapper[2573]: E1103 13:07:44.152526 2573 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" Nov 03 13:07:44.152595 ip-10-0-131-36 kubenswrapper[2573]: E1103 13:07:44.152575 2573 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-131-36.us-east-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" Nov 03 13:07:44.153916 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.153904 2573 server.go:1263] "Started kubelet" Nov 03 13:07:44.154025 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.153992 2573 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 03 13:07:44.154118 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.154045 2573 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 03 13:07:44.154166 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.154117 2573 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 03 13:07:44.154703 ip-10-0-131-36 systemd[1]: Started Kubernetes Kubelet. Nov 03 13:07:44.155196 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.155181 2573 server.go:317] "Adding debug handlers to kubelet server" Nov 03 13:07:44.155644 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.155622 2573 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 03 13:07:44.160027 ip-10-0-131-36 kubenswrapper[2573]: E1103 13:07:44.159212 2573 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-36.us-east-2.compute.internal.18e48ae429821c52 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-36.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-131-36.us-east-2.compute.internal,},FirstTimestamp:2026-11-03 13:07:44.153869394 +0000 UTC m=+0.365088670,LastTimestamp:2026-11-03 13:07:44.153869394 +0000 UTC m=+0.365088670,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-36.us-east-2.compute.internal,}" Nov 03 13:07:44.160124 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.160090 2573 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-131-36.us-east-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Nov 03 13:07:44.161869 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.161852 2573 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 03 13:07:44.161941 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.161913 2573 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 03 13:07:44.161997 ip-10-0-131-36 kubenswrapper[2573]: E1103 13:07:44.161985 2573 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-36.us-east-2.compute.internal\" not found" Nov 03 13:07:44.162252 ip-10-0-131-36 kubenswrapper[2573]: E1103 13:07:44.162228 2573 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Nov 03 13:07:44.162458 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.162444 2573 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 03 13:07:44.162560 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.162446 2573 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 03 13:07:44.162649 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.162623 2573 factory.go:55] Registering systemd factory Nov 03 13:07:44.162711 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.162651 2573 factory.go:223] Registration of the systemd container factory successfully Nov 03 13:07:44.162711 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.162630 2573 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 03 13:07:44.162860 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.162842 2573 factory.go:153] Registering CRI-O factory Nov 03 13:07:44.162925 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.162863 2573 factory.go:223] Registration of the crio container factory successfully Nov 03 13:07:44.162925 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.162914 2573 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 03 13:07:44.163024 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.162847 2573 reconstruct.go:97] "Volume reconstruction finished" Nov 03 13:07:44.163024 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.162944 2573 reconciler.go:29] "Reconciler: start to sync state" Nov 03 13:07:44.163024 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.162933 2573 factory.go:103] Registering Raw factory Nov 03 13:07:44.163024 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.162986 2573 manager.go:1199] Started watching for new ooms in manager Nov 03 13:07:44.163724 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.163710 2573 manager.go:319] Starting recovery of all containers Nov 03 13:07:44.176441 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.173428 2573 manager.go:324] Recovery completed Nov 03 13:07:44.176441 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.175570 2573 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-s527p" Nov 03 13:07:44.177704 ip-10-0-131-36 kubenswrapper[2573]: E1103 13:07:44.177682 2573 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-131-36.us-east-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Nov 03 13:07:44.179859 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.179846 2573 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 13:07:44.180796 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.180774 2573 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-s527p" Nov 03 13:07:44.181891 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.181870 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-36.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 13:07:44.181970 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.181906 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-36.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 13:07:44.181970 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.181922 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-36.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 03 13:07:44.182512 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.182500 2573 cpu_manager.go:226] "Starting" policy="none" Nov 03 13:07:44.182512 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.182511 2573 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 03 13:07:44.182588 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.182525 2573 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 03 13:07:44.183401 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.183371 2573 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 13:07:44.184282 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.184264 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-36.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 13:07:44.184345 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.184296 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-36.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 13:07:44.184345 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.184310 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-36.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 03 13:07:44.187599 ip-10-0-131-36 kubenswrapper[2573]: E1103 13:07:44.187575 2573 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" Nov 03 13:07:44.187663 ip-10-0-131-36 kubenswrapper[2573]: E1103 13:07:44.187550 2573 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-36.us-east-2.compute.internal.18e48ae42b2da8f1 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-36.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-131-36.us-east-2.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-131-36.us-east-2.compute.internal,},FirstTimestamp:2026-11-03 13:07:44.181889265 +0000 UTC m=+0.393108546,LastTimestamp:2026-11-03 13:07:44.181889265 +0000 UTC m=+0.393108546,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-36.us-east-2.compute.internal,}" Nov 03 13:07:44.192768 ip-10-0-131-36 kubenswrapper[2573]: E1103 13:07:44.192704 2573 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-36.us-east-2.compute.internal.18e48ae42b2e0b04 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-36.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-131-36.us-east-2.compute.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-131-36.us-east-2.compute.internal,},FirstTimestamp:2026-11-03 13:07:44.181914372 +0000 UTC m=+0.393133652,LastTimestamp:2026-11-03 13:07:44.181914372 +0000 UTC m=+0.393133652,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-36.us-east-2.compute.internal,}" Nov 03 13:07:44.196244 ip-10-0-131-36 kubenswrapper[2573]: E1103 13:07:44.196188 2573 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-36.us-east-2.compute.internal.18e48ae42b2e41ac default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-36.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-131-36.us-east-2.compute.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-131-36.us-east-2.compute.internal,},FirstTimestamp:2026-11-03 13:07:44.181928364 +0000 UTC m=+0.393147642,LastTimestamp:2026-11-03 13:07:44.181928364 +0000 UTC m=+0.393147642,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-36.us-east-2.compute.internal,}" Nov 03 13:07:44.199521 ip-10-0-131-36 kubenswrapper[2573]: E1103 13:07:44.199462 2573 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-131-36.us-east-2.compute.internal.18e48ae42b2da8f1\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-36.us-east-2.compute.internal.18e48ae42b2da8f1 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-36.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-131-36.us-east-2.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-131-36.us-east-2.compute.internal,},FirstTimestamp:2026-11-03 13:07:44.181889265 +0000 UTC m=+0.393108546,LastTimestamp:2026-11-03 13:07:44.184279982 +0000 UTC m=+0.395499259,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-36.us-east-2.compute.internal,}" Nov 03 13:07:44.208043 ip-10-0-131-36 kubenswrapper[2573]: E1103 13:07:44.207985 2573 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-131-36.us-east-2.compute.internal.18e48ae42b2e0b04\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-36.us-east-2.compute.internal.18e48ae42b2e0b04 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-36.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-131-36.us-east-2.compute.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-131-36.us-east-2.compute.internal,},FirstTimestamp:2026-11-03 13:07:44.181914372 +0000 UTC m=+0.393133652,LastTimestamp:2026-11-03 13:07:44.184302181 +0000 UTC m=+0.395521458,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-36.us-east-2.compute.internal,}" Nov 03 13:07:44.249911 ip-10-0-131-36 kubenswrapper[2573]: E1103 13:07:44.249843 2573 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-131-36.us-east-2.compute.internal.18e48ae42b2e41ac\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-36.us-east-2.compute.internal.18e48ae42b2e41ac default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-36.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-131-36.us-east-2.compute.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-131-36.us-east-2.compute.internal,},FirstTimestamp:2026-11-03 13:07:44.181928364 +0000 UTC m=+0.393147642,LastTimestamp:2026-11-03 13:07:44.184316698 +0000 UTC m=+0.395535975,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-36.us-east-2.compute.internal,}" Nov 03 13:07:44.263003 ip-10-0-131-36 kubenswrapper[2573]: E1103 13:07:44.262983 2573 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-36.us-east-2.compute.internal\" not found" Nov 03 13:07:44.360320 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.360263 2573 policy_none.go:50] "Start" Nov 03 13:07:44.360320 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.360292 2573 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 03 13:07:44.360320 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.360305 2573 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 03 13:07:44.362433 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.362419 2573 policy_none.go:44] "Start" Nov 03 13:07:44.363045 ip-10-0-131-36 kubenswrapper[2573]: E1103 13:07:44.363028 2573 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-36.us-east-2.compute.internal\" not found" Nov 03 13:07:44.386980 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.385112 2573 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 03 13:07:44.386980 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.386601 2573 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 03 13:07:44.386980 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.386622 2573 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 03 13:07:44.386980 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.386660 2573 kubelet.go:2521] "Starting kubelet main sync loop" Nov 03 13:07:44.386980 ip-10-0-131-36 kubenswrapper[2573]: E1103 13:07:44.386703 2573 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Nov 03 13:07:44.392986 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.392949 2573 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 03 13:07:44.396060 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.396046 2573 manager.go:341] "Starting Device Plugin manager" Nov 03 13:07:44.396164 ip-10-0-131-36 kubenswrapper[2573]: E1103 13:07:44.396078 2573 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 03 13:07:44.396164 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.396090 2573 server.go:87] "Starting device plugin registration server" Nov 03 13:07:44.396351 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.396337 2573 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 03 13:07:44.396431 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.396354 2573 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 03 13:07:44.396484 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.396429 2573 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 03 13:07:44.396532 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.396506 2573 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 03 13:07:44.396532 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.396513 2573 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 03 13:07:44.396641 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.396570 2573 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 13:07:44.397139 ip-10-0-131-36 kubenswrapper[2573]: E1103 13:07:44.397118 2573 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 03 13:07:44.397202 ip-10-0-131-36 kubenswrapper[2573]: E1103 13:07:44.397175 2573 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-131-36.us-east-2.compute.internal\" not found" Nov 03 13:07:44.400234 ip-10-0-131-36 kubenswrapper[2573]: E1103 13:07:44.400219 2573 nodelease.go:50] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-131-36.us-east-2.compute.internal\" not found" node="ip-10-0-131-36.us-east-2.compute.internal" Nov 03 13:07:44.487778 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.487737 2573 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-36.us-east-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-131-36.us-east-2.compute.internal"] Nov 03 13:07:44.487868 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.487819 2573 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 13:07:44.488917 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.488903 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-36.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 13:07:44.488963 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.488931 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-36.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 13:07:44.488963 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.488941 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-36.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 03 13:07:44.491121 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.491109 2573 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 13:07:44.491260 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.491245 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-36.us-east-2.compute.internal" Nov 03 13:07:44.491293 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.491282 2573 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 13:07:44.492058 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.492043 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-36.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 13:07:44.492119 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.492068 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-36.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 13:07:44.492119 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.492045 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-36.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 13:07:44.492119 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.492078 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-36.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 03 13:07:44.492119 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.492094 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-36.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 13:07:44.492119 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.492104 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-36.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 03 13:07:44.494188 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.494168 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-36.us-east-2.compute.internal" Nov 03 13:07:44.494265 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.494199 2573 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 13:07:44.495209 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.495187 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-36.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 13:07:44.495288 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.495215 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-36.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 13:07:44.495288 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.495229 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-36.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 03 13:07:44.496576 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.496560 2573 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 13:07:44.497221 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.497207 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-36.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 13:07:44.497287 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.497236 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-36.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 13:07:44.497287 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.497248 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-36.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 03 13:07:44.497287 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.497266 2573 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-131-36.us-east-2.compute.internal" Nov 03 13:07:44.506294 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.506274 2573 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-131-36.us-east-2.compute.internal" Nov 03 13:07:44.506369 ip-10-0-131-36 kubenswrapper[2573]: E1103 13:07:44.506295 2573 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-131-36.us-east-2.compute.internal\": node \"ip-10-0-131-36.us-east-2.compute.internal\" not found" Nov 03 13:07:44.514075 ip-10-0-131-36 kubenswrapper[2573]: E1103 13:07:44.514054 2573 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-36.us-east-2.compute.internal\" not found" node="ip-10-0-131-36.us-east-2.compute.internal" Nov 03 13:07:44.516735 ip-10-0-131-36 kubenswrapper[2573]: E1103 13:07:44.516719 2573 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-36.us-east-2.compute.internal\" not found" node="ip-10-0-131-36.us-east-2.compute.internal" Nov 03 13:07:44.525475 ip-10-0-131-36 kubenswrapper[2573]: E1103 13:07:44.525458 2573 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-36.us-east-2.compute.internal\" not found" Nov 03 13:07:44.563859 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.563821 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/4879f52c5c2e658a43379722bc61381e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-36.us-east-2.compute.internal\" (UID: \"4879f52c5c2e658a43379722bc61381e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-36.us-east-2.compute.internal" Nov 03 13:07:44.563859 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.563859 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4879f52c5c2e658a43379722bc61381e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-36.us-east-2.compute.internal\" (UID: \"4879f52c5c2e658a43379722bc61381e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-36.us-east-2.compute.internal" Nov 03 13:07:44.564034 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.563885 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/5ee88558fb38141d796a31757de861ab-config\") pod \"kube-apiserver-proxy-ip-10-0-131-36.us-east-2.compute.internal\" (UID: \"5ee88558fb38141d796a31757de861ab\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-36.us-east-2.compute.internal" Nov 03 13:07:44.626558 ip-10-0-131-36 kubenswrapper[2573]: E1103 13:07:44.626507 2573 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-36.us-east-2.compute.internal\" not found" Nov 03 13:07:44.665050 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.665023 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4879f52c5c2e658a43379722bc61381e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-36.us-east-2.compute.internal\" (UID: \"4879f52c5c2e658a43379722bc61381e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-36.us-east-2.compute.internal" Nov 03 13:07:44.665141 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.665056 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/5ee88558fb38141d796a31757de861ab-config\") pod \"kube-apiserver-proxy-ip-10-0-131-36.us-east-2.compute.internal\" (UID: \"5ee88558fb38141d796a31757de861ab\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-36.us-east-2.compute.internal" Nov 03 13:07:44.665141 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.665074 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/4879f52c5c2e658a43379722bc61381e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-36.us-east-2.compute.internal\" (UID: \"4879f52c5c2e658a43379722bc61381e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-36.us-east-2.compute.internal" Nov 03 13:07:44.665230 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.665128 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4879f52c5c2e658a43379722bc61381e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-36.us-east-2.compute.internal\" (UID: \"4879f52c5c2e658a43379722bc61381e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-36.us-east-2.compute.internal" Nov 03 13:07:44.665230 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.665186 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/4879f52c5c2e658a43379722bc61381e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-36.us-east-2.compute.internal\" (UID: \"4879f52c5c2e658a43379722bc61381e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-36.us-east-2.compute.internal" Nov 03 13:07:44.665301 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.665221 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/5ee88558fb38141d796a31757de861ab-config\") pod \"kube-apiserver-proxy-ip-10-0-131-36.us-east-2.compute.internal\" (UID: \"5ee88558fb38141d796a31757de861ab\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-36.us-east-2.compute.internal" Nov 03 13:07:44.727185 ip-10-0-131-36 kubenswrapper[2573]: E1103 13:07:44.727156 2573 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-36.us-east-2.compute.internal\" not found" Nov 03 13:07:44.817728 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.817701 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-36.us-east-2.compute.internal" Nov 03 13:07:44.819847 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:44.819827 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-36.us-east-2.compute.internal" Nov 03 13:07:44.828308 ip-10-0-131-36 kubenswrapper[2573]: E1103 13:07:44.828179 2573 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-36.us-east-2.compute.internal\" not found" Nov 03 13:07:44.928800 ip-10-0-131-36 kubenswrapper[2573]: E1103 13:07:44.928719 2573 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-36.us-east-2.compute.internal\" not found" Nov 03 13:07:46.266115 ip-10-0-131-36 kubenswrapper[2573]: E1103 13:07:46.266079 2573 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-36.us-east-2.compute.internal\" not found" Nov 03 13:07:46.333674 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:46.333650 2573 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Nov 03 13:07:46.334098 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:46.333801 2573 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Nov 03 13:07:46.366973 ip-10-0-131-36 kubenswrapper[2573]: E1103 13:07:46.366936 2573 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-36.us-east-2.compute.internal\" not found" Nov 03 13:07:46.467524 ip-10-0-131-36 kubenswrapper[2573]: E1103 13:07:46.467449 2573 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-36.us-east-2.compute.internal\" not found" Nov 03 13:07:46.529910 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:46.529882 2573 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 13:07:46.530827 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:46.530808 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-36.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 13:07:46.530926 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:46.530840 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-36.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 13:07:46.530926 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:46.530854 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-36.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 03 13:07:46.568369 ip-10-0-131-36 kubenswrapper[2573]: E1103 13:07:46.568331 2573 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-36.us-east-2.compute.internal\" not found" Nov 03 13:07:46.597592 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:46.597552 2573 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-02 13:02:45 +0000 UTC" deadline="2028-08-01 17:14:59.882717736 +0000 UTC" Nov 03 13:07:46.597592 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:46.597582 2573 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15292h7m13.285138319s" Nov 03 13:07:46.600178 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:46.600158 2573 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 13:07:46.699063 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:46.699027 2573 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-36.us-east-2.compute.internal" Nov 03 13:07:46.715609 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:46.715587 2573 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 13:07:46.717081 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:46.717063 2573 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-36.us-east-2.compute.internal" Nov 03 13:07:46.721982 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:46.721930 2573 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 13:07:46.725531 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:46.725516 2573 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 13:07:46.734681 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:46.734647 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4879f52c5c2e658a43379722bc61381e.slice/crio-5d20773640ce8e198ac75606401c820525fb0fe1350882ef0b9b2ed7d166824a WatchSource:0}: Error finding container 5d20773640ce8e198ac75606401c820525fb0fe1350882ef0b9b2ed7d166824a: Status 404 returned error can't find the container with id 5d20773640ce8e198ac75606401c820525fb0fe1350882ef0b9b2ed7d166824a Nov 03 13:07:46.736194 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:46.736172 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ee88558fb38141d796a31757de861ab.slice/crio-5b44ee172423f97b68d439f1263e56f195b4502d78f9d58d2899e3b7eee3152f WatchSource:0}: Error finding container 5b44ee172423f97b68d439f1263e56f195b4502d78f9d58d2899e3b7eee3152f: Status 404 returned error can't find the container with id 5b44ee172423f97b68d439f1263e56f195b4502d78f9d58d2899e3b7eee3152f Nov 03 13:07:46.739015 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:46.738997 2573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 13:07:46.993565 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:46.993484 2573 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 13:07:47.376066 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.375982 2573 apiserver.go:52] "Watching apiserver" Nov 03 13:07:47.381896 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.381866 2573 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 03 13:07:47.382270 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.382243 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/node-resolver-v8gcq","openshift-multus/multus-dsjsv","openshift-multus/network-metrics-daemon-f7hmh","openshift-network-operator/iptables-alerter-g4xgp","openshift-ovn-kubernetes/ovnkube-node-fg6vh","kube-system/global-pull-secret-syncer-5jz7t","openshift-cluster-node-tuning-operator/tuned-hmbpc","kube-system/kube-apiserver-proxy-ip-10-0-131-36.us-east-2.compute.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c9dvm","openshift-image-registry/node-ca-4tn4j","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-36.us-east-2.compute.internal","openshift-multus/multus-additional-cni-plugins-x2rm4","openshift-network-diagnostics/network-check-target-8rdgq","kube-system/konnectivity-agent-bdg96"] Nov 03 13:07:47.384931 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.384910 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-hmbpc" Nov 03 13:07:47.386906 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.386880 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-dsjsv" Nov 03 13:07:47.387910 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.387862 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 03 13:07:47.387910 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.387878 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 03 13:07:47.388034 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.387932 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-s4cl2\"" Nov 03 13:07:47.388969 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.388920 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 03 13:07:47.390722 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.389139 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 03 13:07:47.390722 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.389325 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-62cxl\"" Nov 03 13:07:47.390722 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.389431 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 03 13:07:47.390722 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.389469 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 03 13:07:47.390722 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.389604 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f7hmh" Nov 03 13:07:47.390722 ip-10-0-131-36 kubenswrapper[2573]: E1103 13:07:47.389654 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f7hmh" podUID="e0300f55-f002-4faa-bf6a-c04bea0a043e" Nov 03 13:07:47.392266 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.392245 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-g4xgp" Nov 03 13:07:47.394197 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.394176 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-x2rm4" Nov 03 13:07:47.396507 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.396488 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-4tn4j" Nov 03 13:07:47.399028 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.399012 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c9dvm" Nov 03 13:07:47.399119 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.399107 2573 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Nov 03 13:07:47.404599 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.404582 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8rdgq" Nov 03 13:07:47.404680 ip-10-0-131-36 kubenswrapper[2573]: E1103 13:07:47.404643 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8rdgq" podUID="1bfc1afc-5b9f-4447-8d40-fabedbee9c07" Nov 03 13:07:47.407725 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.407706 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-fg6vh" Nov 03 13:07:47.409598 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.409573 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c704eb02-d156-4695-a55d-7dc96654e4cd-etc-kubernetes\") pod \"tuned-hmbpc\" (UID: \"c704eb02-d156-4695-a55d-7dc96654e4cd\") " pod="openshift-cluster-node-tuning-operator/tuned-hmbpc" Nov 03 13:07:47.409685 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.409611 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c704eb02-d156-4695-a55d-7dc96654e4cd-etc-sysctl-d\") pod \"tuned-hmbpc\" (UID: \"c704eb02-d156-4695-a55d-7dc96654e4cd\") " pod="openshift-cluster-node-tuning-operator/tuned-hmbpc" Nov 03 13:07:47.409685 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.409634 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c704eb02-d156-4695-a55d-7dc96654e4cd-etc-sysctl-conf\") pod \"tuned-hmbpc\" (UID: \"c704eb02-d156-4695-a55d-7dc96654e4cd\") " pod="openshift-cluster-node-tuning-operator/tuned-hmbpc" Nov 03 13:07:47.409685 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.409657 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fd7e8b58-de31-406a-8cdb-9ccfee140d57-host-slash\") pod \"iptables-alerter-g4xgp\" (UID: \"fd7e8b58-de31-406a-8cdb-9ccfee140d57\") " pod="openshift-network-operator/iptables-alerter-g4xgp" Nov 03 13:07:47.409845 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.409722 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/35e0fb40-48ce-4223-b7d4-c9290b6861cd-system-cni-dir\") pod \"multus-additional-cni-plugins-x2rm4\" (UID: \"35e0fb40-48ce-4223-b7d4-c9290b6861cd\") " pod="openshift-multus/multus-additional-cni-plugins-x2rm4" Nov 03 13:07:47.409845 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.409754 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/35e0fb40-48ce-4223-b7d4-c9290b6861cd-cnibin\") pod \"multus-additional-cni-plugins-x2rm4\" (UID: \"35e0fb40-48ce-4223-b7d4-c9290b6861cd\") " pod="openshift-multus/multus-additional-cni-plugins-x2rm4" Nov 03 13:07:47.409845 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.409781 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/74dc4aee-10b1-463c-a9c3-522b7a6c10b7-multus-socket-dir-parent\") pod \"multus-dsjsv\" (UID: \"74dc4aee-10b1-463c-a9c3-522b7a6c10b7\") " pod="openshift-multus/multus-dsjsv" Nov 03 13:07:47.409845 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.409829 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e0300f55-f002-4faa-bf6a-c04bea0a043e-metrics-certs\") pod \"network-metrics-daemon-f7hmh\" (UID: \"e0300f55-f002-4faa-bf6a-c04bea0a043e\") " pod="openshift-multus/network-metrics-daemon-f7hmh" Nov 03 13:07:47.410044 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.409853 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c704eb02-d156-4695-a55d-7dc96654e4cd-run\") pod \"tuned-hmbpc\" (UID: \"c704eb02-d156-4695-a55d-7dc96654e4cd\") " pod="openshift-cluster-node-tuning-operator/tuned-hmbpc" Nov 03 13:07:47.410044 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.409878 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c704eb02-d156-4695-a55d-7dc96654e4cd-tmp\") pod \"tuned-hmbpc\" (UID: \"c704eb02-d156-4695-a55d-7dc96654e4cd\") " pod="openshift-cluster-node-tuning-operator/tuned-hmbpc" Nov 03 13:07:47.410044 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.409903 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/35e0fb40-48ce-4223-b7d4-c9290b6861cd-tuning-conf-dir\") pod \"multus-additional-cni-plugins-x2rm4\" (UID: \"35e0fb40-48ce-4223-b7d4-c9290b6861cd\") " pod="openshift-multus/multus-additional-cni-plugins-x2rm4" Nov 03 13:07:47.410044 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.409928 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/35e0fb40-48ce-4223-b7d4-c9290b6861cd-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-x2rm4\" (UID: \"35e0fb40-48ce-4223-b7d4-c9290b6861cd\") " pod="openshift-multus/multus-additional-cni-plugins-x2rm4" Nov 03 13:07:47.410044 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.409953 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/35e0fb40-48ce-4223-b7d4-c9290b6861cd-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-x2rm4\" (UID: \"35e0fb40-48ce-4223-b7d4-c9290b6861cd\") " pod="openshift-multus/multus-additional-cni-plugins-x2rm4" Nov 03 13:07:47.410044 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.410023 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9db7d195-220e-4644-8690-9e5ee37fc066-kubelet-dir\") pod \"aws-ebs-csi-driver-node-c9dvm\" (UID: \"9db7d195-220e-4644-8690-9e5ee37fc066\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c9dvm" Nov 03 13:07:47.410349 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.410047 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/74dc4aee-10b1-463c-a9c3-522b7a6c10b7-multus-cni-dir\") pod \"multus-dsjsv\" (UID: \"74dc4aee-10b1-463c-a9c3-522b7a6c10b7\") " pod="openshift-multus/multus-dsjsv" Nov 03 13:07:47.410349 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.410072 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/74dc4aee-10b1-463c-a9c3-522b7a6c10b7-host-run-k8s-cni-cncf-io\") pod \"multus-dsjsv\" (UID: \"74dc4aee-10b1-463c-a9c3-522b7a6c10b7\") " pod="openshift-multus/multus-dsjsv" Nov 03 13:07:47.410349 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.410099 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cwrs7\" (UniqueName: \"kubernetes.io/projected/74dc4aee-10b1-463c-a9c3-522b7a6c10b7-kube-api-access-cwrs7\") pod \"multus-dsjsv\" (UID: \"74dc4aee-10b1-463c-a9c3-522b7a6c10b7\") " pod="openshift-multus/multus-dsjsv" Nov 03 13:07:47.410349 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.410123 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bcz25\" (UniqueName: \"kubernetes.io/projected/e0300f55-f002-4faa-bf6a-c04bea0a043e-kube-api-access-bcz25\") pod \"network-metrics-daemon-f7hmh\" (UID: \"e0300f55-f002-4faa-bf6a-c04bea0a043e\") " pod="openshift-multus/network-metrics-daemon-f7hmh" Nov 03 13:07:47.410349 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.410146 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c704eb02-d156-4695-a55d-7dc96654e4cd-host\") pod \"tuned-hmbpc\" (UID: \"c704eb02-d156-4695-a55d-7dc96654e4cd\") " pod="openshift-cluster-node-tuning-operator/tuned-hmbpc" Nov 03 13:07:47.410349 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.410169 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9db7d195-220e-4644-8690-9e5ee37fc066-socket-dir\") pod \"aws-ebs-csi-driver-node-c9dvm\" (UID: \"9db7d195-220e-4644-8690-9e5ee37fc066\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c9dvm" Nov 03 13:07:47.410349 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.410192 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9db7d195-220e-4644-8690-9e5ee37fc066-etc-selinux\") pod \"aws-ebs-csi-driver-node-c9dvm\" (UID: \"9db7d195-220e-4644-8690-9e5ee37fc066\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c9dvm" Nov 03 13:07:47.410349 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.410217 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ftcqc\" (UniqueName: \"kubernetes.io/projected/9db7d195-220e-4644-8690-9e5ee37fc066-kube-api-access-ftcqc\") pod \"aws-ebs-csi-driver-node-c9dvm\" (UID: \"9db7d195-220e-4644-8690-9e5ee37fc066\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c9dvm" Nov 03 13:07:47.410349 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.410241 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/74dc4aee-10b1-463c-a9c3-522b7a6c10b7-system-cni-dir\") pod \"multus-dsjsv\" (UID: \"74dc4aee-10b1-463c-a9c3-522b7a6c10b7\") " pod="openshift-multus/multus-dsjsv" Nov 03 13:07:47.410349 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.410264 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/74dc4aee-10b1-463c-a9c3-522b7a6c10b7-cni-binary-copy\") pod \"multus-dsjsv\" (UID: \"74dc4aee-10b1-463c-a9c3-522b7a6c10b7\") " pod="openshift-multus/multus-dsjsv" Nov 03 13:07:47.410914 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.410361 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c704eb02-d156-4695-a55d-7dc96654e4cd-etc-systemd\") pod \"tuned-hmbpc\" (UID: \"c704eb02-d156-4695-a55d-7dc96654e4cd\") " pod="openshift-cluster-node-tuning-operator/tuned-hmbpc" Nov 03 13:07:47.410914 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.410393 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-snpf4\" (UniqueName: \"kubernetes.io/projected/fd7e8b58-de31-406a-8cdb-9ccfee140d57-kube-api-access-snpf4\") pod \"iptables-alerter-g4xgp\" (UID: \"fd7e8b58-de31-406a-8cdb-9ccfee140d57\") " pod="openshift-network-operator/iptables-alerter-g4xgp" Nov 03 13:07:47.410914 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.410418 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9db7d195-220e-4644-8690-9e5ee37fc066-device-dir\") pod \"aws-ebs-csi-driver-node-c9dvm\" (UID: \"9db7d195-220e-4644-8690-9e5ee37fc066\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c9dvm" Nov 03 13:07:47.410914 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.410442 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hbfgb\" (UniqueName: \"kubernetes.io/projected/1bfc1afc-5b9f-4447-8d40-fabedbee9c07-kube-api-access-hbfgb\") pod \"network-check-target-8rdgq\" (UID: \"1bfc1afc-5b9f-4447-8d40-fabedbee9c07\") " pod="openshift-network-diagnostics/network-check-target-8rdgq" Nov 03 13:07:47.410914 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.410475 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/74dc4aee-10b1-463c-a9c3-522b7a6c10b7-host-var-lib-kubelet\") pod \"multus-dsjsv\" (UID: \"74dc4aee-10b1-463c-a9c3-522b7a6c10b7\") " pod="openshift-multus/multus-dsjsv" Nov 03 13:07:47.410914 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.410519 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/74dc4aee-10b1-463c-a9c3-522b7a6c10b7-host-run-multus-certs\") pod \"multus-dsjsv\" (UID: \"74dc4aee-10b1-463c-a9c3-522b7a6c10b7\") " pod="openshift-multus/multus-dsjsv" Nov 03 13:07:47.410914 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.410570 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c704eb02-d156-4695-a55d-7dc96654e4cd-etc-modprobe-d\") pod \"tuned-hmbpc\" (UID: \"c704eb02-d156-4695-a55d-7dc96654e4cd\") " pod="openshift-cluster-node-tuning-operator/tuned-hmbpc" Nov 03 13:07:47.410914 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.410598 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c704eb02-d156-4695-a55d-7dc96654e4cd-lib-modules\") pod \"tuned-hmbpc\" (UID: \"c704eb02-d156-4695-a55d-7dc96654e4cd\") " pod="openshift-cluster-node-tuning-operator/tuned-hmbpc" Nov 03 13:07:47.410914 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.410621 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c704eb02-d156-4695-a55d-7dc96654e4cd-var-lib-kubelet\") pod \"tuned-hmbpc\" (UID: \"c704eb02-d156-4695-a55d-7dc96654e4cd\") " pod="openshift-cluster-node-tuning-operator/tuned-hmbpc" Nov 03 13:07:47.410914 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.410626 2573 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 03 13:07:47.410914 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.410650 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w4j5g\" (UniqueName: \"kubernetes.io/projected/c704eb02-d156-4695-a55d-7dc96654e4cd-kube-api-access-w4j5g\") pod \"tuned-hmbpc\" (UID: \"c704eb02-d156-4695-a55d-7dc96654e4cd\") " pod="openshift-cluster-node-tuning-operator/tuned-hmbpc" Nov 03 13:07:47.410914 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.410677 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/74dc4aee-10b1-463c-a9c3-522b7a6c10b7-multus-daemon-config\") pod \"multus-dsjsv\" (UID: \"74dc4aee-10b1-463c-a9c3-522b7a6c10b7\") " pod="openshift-multus/multus-dsjsv" Nov 03 13:07:47.410914 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.410704 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c704eb02-d156-4695-a55d-7dc96654e4cd-etc-tuned\") pod \"tuned-hmbpc\" (UID: \"c704eb02-d156-4695-a55d-7dc96654e4cd\") " pod="openshift-cluster-node-tuning-operator/tuned-hmbpc" Nov 03 13:07:47.410914 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.410731 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/fd7e8b58-de31-406a-8cdb-9ccfee140d57-iptables-alerter-script\") pod \"iptables-alerter-g4xgp\" (UID: \"fd7e8b58-de31-406a-8cdb-9ccfee140d57\") " pod="openshift-network-operator/iptables-alerter-g4xgp" Nov 03 13:07:47.410914 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.410756 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/35e0fb40-48ce-4223-b7d4-c9290b6861cd-cni-binary-copy\") pod \"multus-additional-cni-plugins-x2rm4\" (UID: \"35e0fb40-48ce-4223-b7d4-c9290b6861cd\") " pod="openshift-multus/multus-additional-cni-plugins-x2rm4" Nov 03 13:07:47.410914 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.410781 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9db7d195-220e-4644-8690-9e5ee37fc066-registration-dir\") pod \"aws-ebs-csi-driver-node-c9dvm\" (UID: \"9db7d195-220e-4644-8690-9e5ee37fc066\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c9dvm" Nov 03 13:07:47.410914 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.410806 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/74dc4aee-10b1-463c-a9c3-522b7a6c10b7-cnibin\") pod \"multus-dsjsv\" (UID: \"74dc4aee-10b1-463c-a9c3-522b7a6c10b7\") " pod="openshift-multus/multus-dsjsv" Nov 03 13:07:47.411747 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.410876 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/74dc4aee-10b1-463c-a9c3-522b7a6c10b7-hostroot\") pod \"multus-dsjsv\" (UID: \"74dc4aee-10b1-463c-a9c3-522b7a6c10b7\") " pod="openshift-multus/multus-dsjsv" Nov 03 13:07:47.411747 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.410919 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/74dc4aee-10b1-463c-a9c3-522b7a6c10b7-etc-kubernetes\") pod \"multus-dsjsv\" (UID: \"74dc4aee-10b1-463c-a9c3-522b7a6c10b7\") " pod="openshift-multus/multus-dsjsv" Nov 03 13:07:47.411747 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.410966 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mss42\" (UniqueName: \"kubernetes.io/projected/35e0fb40-48ce-4223-b7d4-c9290b6861cd-kube-api-access-mss42\") pod \"multus-additional-cni-plugins-x2rm4\" (UID: \"35e0fb40-48ce-4223-b7d4-c9290b6861cd\") " pod="openshift-multus/multus-additional-cni-plugins-x2rm4" Nov 03 13:07:47.411747 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.410985 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5jz7t" Nov 03 13:07:47.411747 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.410997 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/74dc4aee-10b1-463c-a9c3-522b7a6c10b7-host-run-netns\") pod \"multus-dsjsv\" (UID: \"74dc4aee-10b1-463c-a9c3-522b7a6c10b7\") " pod="openshift-multus/multus-dsjsv" Nov 03 13:07:47.411747 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.411032 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/74dc4aee-10b1-463c-a9c3-522b7a6c10b7-host-var-lib-cni-bin\") pod \"multus-dsjsv\" (UID: \"74dc4aee-10b1-463c-a9c3-522b7a6c10b7\") " pod="openshift-multus/multus-dsjsv" Nov 03 13:07:47.411747 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.411055 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/74dc4aee-10b1-463c-a9c3-522b7a6c10b7-host-var-lib-cni-multus\") pod \"multus-dsjsv\" (UID: \"74dc4aee-10b1-463c-a9c3-522b7a6c10b7\") " pod="openshift-multus/multus-dsjsv" Nov 03 13:07:47.411747 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.411079 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/74dc4aee-10b1-463c-a9c3-522b7a6c10b7-multus-conf-dir\") pod \"multus-dsjsv\" (UID: \"74dc4aee-10b1-463c-a9c3-522b7a6c10b7\") " pod="openshift-multus/multus-dsjsv" Nov 03 13:07:47.411747 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.411102 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zpq8b\" (UniqueName: \"kubernetes.io/projected/39e11989-e598-4cea-87d9-26b3cf78a66b-kube-api-access-zpq8b\") pod \"node-ca-4tn4j\" (UID: \"39e11989-e598-4cea-87d9-26b3cf78a66b\") " pod="openshift-image-registry/node-ca-4tn4j" Nov 03 13:07:47.411747 ip-10-0-131-36 kubenswrapper[2573]: E1103 13:07:47.411052 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5jz7t" podUID="7e280277-f29f-4a2b-8b85-2491529e5ef2" Nov 03 13:07:47.411747 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.411153 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c704eb02-d156-4695-a55d-7dc96654e4cd-etc-sysconfig\") pod \"tuned-hmbpc\" (UID: \"c704eb02-d156-4695-a55d-7dc96654e4cd\") " pod="openshift-cluster-node-tuning-operator/tuned-hmbpc" Nov 03 13:07:47.411747 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.411179 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c704eb02-d156-4695-a55d-7dc96654e4cd-sys\") pod \"tuned-hmbpc\" (UID: \"c704eb02-d156-4695-a55d-7dc96654e4cd\") " pod="openshift-cluster-node-tuning-operator/tuned-hmbpc" Nov 03 13:07:47.411747 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.411203 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9db7d195-220e-4644-8690-9e5ee37fc066-sys-fs\") pod \"aws-ebs-csi-driver-node-c9dvm\" (UID: \"9db7d195-220e-4644-8690-9e5ee37fc066\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c9dvm" Nov 03 13:07:47.411747 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.411226 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/39e11989-e598-4cea-87d9-26b3cf78a66b-host\") pod \"node-ca-4tn4j\" (UID: \"39e11989-e598-4cea-87d9-26b3cf78a66b\") " pod="openshift-image-registry/node-ca-4tn4j" Nov 03 13:07:47.411747 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.411265 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/39e11989-e598-4cea-87d9-26b3cf78a66b-serviceca\") pod \"node-ca-4tn4j\" (UID: \"39e11989-e598-4cea-87d9-26b3cf78a66b\") " pod="openshift-image-registry/node-ca-4tn4j" Nov 03 13:07:47.414319 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.413449 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-v8gcq" Nov 03 13:07:47.415758 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.415525 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-bdg96" Nov 03 13:07:47.431657 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.431566 2573 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-pbx86" Nov 03 13:07:47.439081 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.439063 2573 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-pbx86" Nov 03 13:07:47.489775 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.489650 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 03 13:07:47.489775 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.489653 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 03 13:07:47.489775 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.489739 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 03 13:07:47.490007 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.489901 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 03 13:07:47.490042 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.490007 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 03 13:07:47.490042 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.490035 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 03 13:07:47.490117 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.490104 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 03 13:07:47.490170 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.490124 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 03 13:07:47.490170 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.490161 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 03 13:07:47.490263 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.490182 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 03 13:07:47.490263 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.490194 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 03 13:07:47.490511 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.490484 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 03 13:07:47.490511 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.490503 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 03 13:07:47.490749 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.490538 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 03 13:07:47.490749 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.490543 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 03 13:07:47.490749 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.490564 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 03 13:07:47.491239 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.491213 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 03 13:07:47.491974 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.491954 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-vwt8x\"" Nov 03 13:07:47.492063 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.492039 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-psqgf\"" Nov 03 13:07:47.492163 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.492150 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-c596b\"" Nov 03 13:07:47.492236 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.492224 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 03 13:07:47.492344 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.492327 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-wv2nv\"" Nov 03 13:07:47.492442 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.492397 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-4mrd6\"" Nov 03 13:07:47.492500 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.492449 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-jqlk5\"" Nov 03 13:07:47.492549 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.492515 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 03 13:07:47.492580 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.492574 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 03 13:07:47.492613 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.492603 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 03 13:07:47.492681 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.492669 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-knm89\"" Nov 03 13:07:47.500841 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.500823 2573 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 03 13:07:47.511771 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.511745 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ftcqc\" (UniqueName: \"kubernetes.io/projected/9db7d195-220e-4644-8690-9e5ee37fc066-kube-api-access-ftcqc\") pod \"aws-ebs-csi-driver-node-c9dvm\" (UID: \"9db7d195-220e-4644-8690-9e5ee37fc066\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c9dvm" Nov 03 13:07:47.511878 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.511788 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d926d945-39f4-48f3-b3a4-0b302b15b69b-env-overrides\") pod \"ovnkube-node-fg6vh\" (UID: \"d926d945-39f4-48f3-b3a4-0b302b15b69b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fg6vh" Nov 03 13:07:47.511878 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.511817 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/74dc4aee-10b1-463c-a9c3-522b7a6c10b7-system-cni-dir\") pod \"multus-dsjsv\" (UID: \"74dc4aee-10b1-463c-a9c3-522b7a6c10b7\") " pod="openshift-multus/multus-dsjsv" Nov 03 13:07:47.511878 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.511842 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-snpf4\" (UniqueName: \"kubernetes.io/projected/fd7e8b58-de31-406a-8cdb-9ccfee140d57-kube-api-access-snpf4\") pod \"iptables-alerter-g4xgp\" (UID: \"fd7e8b58-de31-406a-8cdb-9ccfee140d57\") " pod="openshift-network-operator/iptables-alerter-g4xgp" Nov 03 13:07:47.511878 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.511867 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7e280277-f29f-4a2b-8b85-2491529e5ef2-original-pull-secret\") pod \"global-pull-secret-syncer-5jz7t\" (UID: \"7e280277-f29f-4a2b-8b85-2491529e5ef2\") " pod="kube-system/global-pull-secret-syncer-5jz7t" Nov 03 13:07:47.512067 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.511893 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/74dc4aee-10b1-463c-a9c3-522b7a6c10b7-host-run-multus-certs\") pod \"multus-dsjsv\" (UID: \"74dc4aee-10b1-463c-a9c3-522b7a6c10b7\") " pod="openshift-multus/multus-dsjsv" Nov 03 13:07:47.512067 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.511917 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c704eb02-d156-4695-a55d-7dc96654e4cd-etc-modprobe-d\") pod \"tuned-hmbpc\" (UID: \"c704eb02-d156-4695-a55d-7dc96654e4cd\") " pod="openshift-cluster-node-tuning-operator/tuned-hmbpc" Nov 03 13:07:47.512067 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.511951 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4j5g\" (UniqueName: \"kubernetes.io/projected/c704eb02-d156-4695-a55d-7dc96654e4cd-kube-api-access-w4j5g\") pod \"tuned-hmbpc\" (UID: \"c704eb02-d156-4695-a55d-7dc96654e4cd\") " pod="openshift-cluster-node-tuning-operator/tuned-hmbpc" Nov 03 13:07:47.512067 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.511976 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9db7d195-220e-4644-8690-9e5ee37fc066-registration-dir\") pod \"aws-ebs-csi-driver-node-c9dvm\" (UID: \"9db7d195-220e-4644-8690-9e5ee37fc066\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c9dvm" Nov 03 13:07:47.512067 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.512001 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/246971bc-ed35-4fc7-b5f8-941327b90a94-konnectivity-ca\") pod \"konnectivity-agent-bdg96\" (UID: \"246971bc-ed35-4fc7-b5f8-941327b90a94\") " pod="kube-system/konnectivity-agent-bdg96" Nov 03 13:07:47.512067 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.512025 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5ea5cf61-ed1c-4f5e-b1a0-5c46c5c3ee18-tmp-dir\") pod \"node-resolver-v8gcq\" (UID: \"5ea5cf61-ed1c-4f5e-b1a0-5c46c5c3ee18\") " pod="openshift-dns/node-resolver-v8gcq" Nov 03 13:07:47.512067 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.512040 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/74dc4aee-10b1-463c-a9c3-522b7a6c10b7-multus-daemon-config\") pod \"multus-dsjsv\" (UID: \"74dc4aee-10b1-463c-a9c3-522b7a6c10b7\") " pod="openshift-multus/multus-dsjsv" Nov 03 13:07:47.512067 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.512055 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/fd7e8b58-de31-406a-8cdb-9ccfee140d57-iptables-alerter-script\") pod \"iptables-alerter-g4xgp\" (UID: \"fd7e8b58-de31-406a-8cdb-9ccfee140d57\") " pod="openshift-network-operator/iptables-alerter-g4xgp" Nov 03 13:07:47.512415 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.512070 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6b28l\" (UniqueName: \"kubernetes.io/projected/d926d945-39f4-48f3-b3a4-0b302b15b69b-kube-api-access-6b28l\") pod \"ovnkube-node-fg6vh\" (UID: \"d926d945-39f4-48f3-b3a4-0b302b15b69b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fg6vh" Nov 03 13:07:47.512415 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.512093 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjhjt\" (UniqueName: \"kubernetes.io/projected/5ea5cf61-ed1c-4f5e-b1a0-5c46c5c3ee18-kube-api-access-vjhjt\") pod \"node-resolver-v8gcq\" (UID: \"5ea5cf61-ed1c-4f5e-b1a0-5c46c5c3ee18\") " pod="openshift-dns/node-resolver-v8gcq" Nov 03 13:07:47.512415 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.512117 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/74dc4aee-10b1-463c-a9c3-522b7a6c10b7-cnibin\") pod \"multus-dsjsv\" (UID: \"74dc4aee-10b1-463c-a9c3-522b7a6c10b7\") " pod="openshift-multus/multus-dsjsv" Nov 03 13:07:47.512415 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.512131 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mss42\" (UniqueName: \"kubernetes.io/projected/35e0fb40-48ce-4223-b7d4-c9290b6861cd-kube-api-access-mss42\") pod \"multus-additional-cni-plugins-x2rm4\" (UID: \"35e0fb40-48ce-4223-b7d4-c9290b6861cd\") " pod="openshift-multus/multus-additional-cni-plugins-x2rm4" Nov 03 13:07:47.512415 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.512154 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d926d945-39f4-48f3-b3a4-0b302b15b69b-log-socket\") pod \"ovnkube-node-fg6vh\" (UID: \"d926d945-39f4-48f3-b3a4-0b302b15b69b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fg6vh" Nov 03 13:07:47.512415 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.512176 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/74dc4aee-10b1-463c-a9c3-522b7a6c10b7-host-var-lib-cni-bin\") pod \"multus-dsjsv\" (UID: \"74dc4aee-10b1-463c-a9c3-522b7a6c10b7\") " pod="openshift-multus/multus-dsjsv" Nov 03 13:07:47.512415 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.512198 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c704eb02-d156-4695-a55d-7dc96654e4cd-etc-sysconfig\") pod \"tuned-hmbpc\" (UID: \"c704eb02-d156-4695-a55d-7dc96654e4cd\") " pod="openshift-cluster-node-tuning-operator/tuned-hmbpc" Nov 03 13:07:47.512415 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.512219 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9db7d195-220e-4644-8690-9e5ee37fc066-sys-fs\") pod \"aws-ebs-csi-driver-node-c9dvm\" (UID: \"9db7d195-220e-4644-8690-9e5ee37fc066\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c9dvm" Nov 03 13:07:47.512415 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.512235 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d926d945-39f4-48f3-b3a4-0b302b15b69b-var-lib-openvswitch\") pod \"ovnkube-node-fg6vh\" (UID: \"d926d945-39f4-48f3-b3a4-0b302b15b69b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fg6vh" Nov 03 13:07:47.512415 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.512257 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/39e11989-e598-4cea-87d9-26b3cf78a66b-host\") pod \"node-ca-4tn4j\" (UID: \"39e11989-e598-4cea-87d9-26b3cf78a66b\") " pod="openshift-image-registry/node-ca-4tn4j" Nov 03 13:07:47.512415 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.512273 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/39e11989-e598-4cea-87d9-26b3cf78a66b-serviceca\") pod \"node-ca-4tn4j\" (UID: \"39e11989-e598-4cea-87d9-26b3cf78a66b\") " pod="openshift-image-registry/node-ca-4tn4j" Nov 03 13:07:47.512415 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.512307 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c704eb02-d156-4695-a55d-7dc96654e4cd-etc-sysctl-conf\") pod \"tuned-hmbpc\" (UID: \"c704eb02-d156-4695-a55d-7dc96654e4cd\") " pod="openshift-cluster-node-tuning-operator/tuned-hmbpc" Nov 03 13:07:47.512415 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.512331 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fd7e8b58-de31-406a-8cdb-9ccfee140d57-host-slash\") pod \"iptables-alerter-g4xgp\" (UID: \"fd7e8b58-de31-406a-8cdb-9ccfee140d57\") " pod="openshift-network-operator/iptables-alerter-g4xgp" Nov 03 13:07:47.512415 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.512354 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/35e0fb40-48ce-4223-b7d4-c9290b6861cd-tuning-conf-dir\") pod \"multus-additional-cni-plugins-x2rm4\" (UID: \"35e0fb40-48ce-4223-b7d4-c9290b6861cd\") " pod="openshift-multus/multus-additional-cni-plugins-x2rm4" Nov 03 13:07:47.512415 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.512382 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d926d945-39f4-48f3-b3a4-0b302b15b69b-host-kubelet\") pod \"ovnkube-node-fg6vh\" (UID: \"d926d945-39f4-48f3-b3a4-0b302b15b69b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fg6vh" Nov 03 13:07:47.512415 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.512406 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d926d945-39f4-48f3-b3a4-0b302b15b69b-host-cni-bin\") pod \"ovnkube-node-fg6vh\" (UID: \"d926d945-39f4-48f3-b3a4-0b302b15b69b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fg6vh" Nov 03 13:07:47.513094 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.512432 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d926d945-39f4-48f3-b3a4-0b302b15b69b-host-cni-netd\") pod \"ovnkube-node-fg6vh\" (UID: \"d926d945-39f4-48f3-b3a4-0b302b15b69b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fg6vh" Nov 03 13:07:47.513094 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.512455 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e0300f55-f002-4faa-bf6a-c04bea0a043e-metrics-certs\") pod \"network-metrics-daemon-f7hmh\" (UID: \"e0300f55-f002-4faa-bf6a-c04bea0a043e\") " pod="openshift-multus/network-metrics-daemon-f7hmh" Nov 03 13:07:47.513094 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.512481 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/35e0fb40-48ce-4223-b7d4-c9290b6861cd-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-x2rm4\" (UID: \"35e0fb40-48ce-4223-b7d4-c9290b6861cd\") " pod="openshift-multus/multus-additional-cni-plugins-x2rm4" Nov 03 13:07:47.513094 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.512507 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9db7d195-220e-4644-8690-9e5ee37fc066-etc-selinux\") pod \"aws-ebs-csi-driver-node-c9dvm\" (UID: \"9db7d195-220e-4644-8690-9e5ee37fc066\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c9dvm" Nov 03 13:07:47.513094 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.512532 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d926d945-39f4-48f3-b3a4-0b302b15b69b-run-openvswitch\") pod \"ovnkube-node-fg6vh\" (UID: \"d926d945-39f4-48f3-b3a4-0b302b15b69b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fg6vh" Nov 03 13:07:47.513094 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.512562 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/7e280277-f29f-4a2b-8b85-2491529e5ef2-dbus\") pod \"global-pull-secret-syncer-5jz7t\" (UID: \"7e280277-f29f-4a2b-8b85-2491529e5ef2\") " pod="kube-system/global-pull-secret-syncer-5jz7t" Nov 03 13:07:47.513094 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.512586 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/74dc4aee-10b1-463c-a9c3-522b7a6c10b7-multus-cni-dir\") pod \"multus-dsjsv\" (UID: \"74dc4aee-10b1-463c-a9c3-522b7a6c10b7\") " pod="openshift-multus/multus-dsjsv" Nov 03 13:07:47.513094 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.512611 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwrs7\" (UniqueName: \"kubernetes.io/projected/74dc4aee-10b1-463c-a9c3-522b7a6c10b7-kube-api-access-cwrs7\") pod \"multus-dsjsv\" (UID: \"74dc4aee-10b1-463c-a9c3-522b7a6c10b7\") " pod="openshift-multus/multus-dsjsv" Nov 03 13:07:47.513094 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.512637 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c704eb02-d156-4695-a55d-7dc96654e4cd-host\") pod \"tuned-hmbpc\" (UID: \"c704eb02-d156-4695-a55d-7dc96654e4cd\") " pod="openshift-cluster-node-tuning-operator/tuned-hmbpc" Nov 03 13:07:47.513094 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.512660 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9db7d195-220e-4644-8690-9e5ee37fc066-socket-dir\") pod \"aws-ebs-csi-driver-node-c9dvm\" (UID: \"9db7d195-220e-4644-8690-9e5ee37fc066\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c9dvm" Nov 03 13:07:47.513094 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.512677 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d926d945-39f4-48f3-b3a4-0b302b15b69b-ovnkube-config\") pod \"ovnkube-node-fg6vh\" (UID: \"d926d945-39f4-48f3-b3a4-0b302b15b69b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fg6vh" Nov 03 13:07:47.513094 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.512695 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/7e280277-f29f-4a2b-8b85-2491529e5ef2-kubelet-config\") pod \"global-pull-secret-syncer-5jz7t\" (UID: \"7e280277-f29f-4a2b-8b85-2491529e5ef2\") " pod="kube-system/global-pull-secret-syncer-5jz7t" Nov 03 13:07:47.513094 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.512716 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/74dc4aee-10b1-463c-a9c3-522b7a6c10b7-cni-binary-copy\") pod \"multus-dsjsv\" (UID: \"74dc4aee-10b1-463c-a9c3-522b7a6c10b7\") " pod="openshift-multus/multus-dsjsv" Nov 03 13:07:47.513094 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.512738 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c704eb02-d156-4695-a55d-7dc96654e4cd-etc-systemd\") pod \"tuned-hmbpc\" (UID: \"c704eb02-d156-4695-a55d-7dc96654e4cd\") " pod="openshift-cluster-node-tuning-operator/tuned-hmbpc" Nov 03 13:07:47.513094 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.512758 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d926d945-39f4-48f3-b3a4-0b302b15b69b-ovn-node-metrics-cert\") pod \"ovnkube-node-fg6vh\" (UID: \"d926d945-39f4-48f3-b3a4-0b302b15b69b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fg6vh" Nov 03 13:07:47.513094 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.512786 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/74dc4aee-10b1-463c-a9c3-522b7a6c10b7-host-var-lib-kubelet\") pod \"multus-dsjsv\" (UID: \"74dc4aee-10b1-463c-a9c3-522b7a6c10b7\") " pod="openshift-multus/multus-dsjsv" Nov 03 13:07:47.513094 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.512812 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c704eb02-d156-4695-a55d-7dc96654e4cd-lib-modules\") pod \"tuned-hmbpc\" (UID: \"c704eb02-d156-4695-a55d-7dc96654e4cd\") " pod="openshift-cluster-node-tuning-operator/tuned-hmbpc" Nov 03 13:07:47.513779 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.512835 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c704eb02-d156-4695-a55d-7dc96654e4cd-var-lib-kubelet\") pod \"tuned-hmbpc\" (UID: \"c704eb02-d156-4695-a55d-7dc96654e4cd\") " pod="openshift-cluster-node-tuning-operator/tuned-hmbpc" Nov 03 13:07:47.513779 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.512859 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/35e0fb40-48ce-4223-b7d4-c9290b6861cd-cni-binary-copy\") pod \"multus-additional-cni-plugins-x2rm4\" (UID: \"35e0fb40-48ce-4223-b7d4-c9290b6861cd\") " pod="openshift-multus/multus-additional-cni-plugins-x2rm4" Nov 03 13:07:47.513779 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.512887 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d926d945-39f4-48f3-b3a4-0b302b15b69b-host-run-netns\") pod \"ovnkube-node-fg6vh\" (UID: \"d926d945-39f4-48f3-b3a4-0b302b15b69b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fg6vh" Nov 03 13:07:47.513779 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.512913 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d926d945-39f4-48f3-b3a4-0b302b15b69b-etc-openvswitch\") pod \"ovnkube-node-fg6vh\" (UID: \"d926d945-39f4-48f3-b3a4-0b302b15b69b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fg6vh" Nov 03 13:07:47.513779 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.512938 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c704eb02-d156-4695-a55d-7dc96654e4cd-etc-tuned\") pod \"tuned-hmbpc\" (UID: \"c704eb02-d156-4695-a55d-7dc96654e4cd\") " pod="openshift-cluster-node-tuning-operator/tuned-hmbpc" Nov 03 13:07:47.513779 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.512963 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d926d945-39f4-48f3-b3a4-0b302b15b69b-run-ovn\") pod \"ovnkube-node-fg6vh\" (UID: \"d926d945-39f4-48f3-b3a4-0b302b15b69b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fg6vh" Nov 03 13:07:47.513779 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.513017 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d926d945-39f4-48f3-b3a4-0b302b15b69b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-fg6vh\" (UID: \"d926d945-39f4-48f3-b3a4-0b302b15b69b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fg6vh" Nov 03 13:07:47.513779 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.513047 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d926d945-39f4-48f3-b3a4-0b302b15b69b-ovnkube-script-lib\") pod \"ovnkube-node-fg6vh\" (UID: \"d926d945-39f4-48f3-b3a4-0b302b15b69b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fg6vh" Nov 03 13:07:47.513779 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.513099 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/74dc4aee-10b1-463c-a9c3-522b7a6c10b7-hostroot\") pod \"multus-dsjsv\" (UID: \"74dc4aee-10b1-463c-a9c3-522b7a6c10b7\") " pod="openshift-multus/multus-dsjsv" Nov 03 13:07:47.513779 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.513123 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/74dc4aee-10b1-463c-a9c3-522b7a6c10b7-etc-kubernetes\") pod \"multus-dsjsv\" (UID: \"74dc4aee-10b1-463c-a9c3-522b7a6c10b7\") " pod="openshift-multus/multus-dsjsv" Nov 03 13:07:47.513779 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.513150 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d926d945-39f4-48f3-b3a4-0b302b15b69b-run-systemd\") pod \"ovnkube-node-fg6vh\" (UID: \"d926d945-39f4-48f3-b3a4-0b302b15b69b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fg6vh" Nov 03 13:07:47.513779 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.513174 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d926d945-39f4-48f3-b3a4-0b302b15b69b-host-run-ovn-kubernetes\") pod \"ovnkube-node-fg6vh\" (UID: \"d926d945-39f4-48f3-b3a4-0b302b15b69b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fg6vh" Nov 03 13:07:47.513779 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.513202 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/5ea5cf61-ed1c-4f5e-b1a0-5c46c5c3ee18-hosts-file\") pod \"node-resolver-v8gcq\" (UID: \"5ea5cf61-ed1c-4f5e-b1a0-5c46c5c3ee18\") " pod="openshift-dns/node-resolver-v8gcq" Nov 03 13:07:47.513779 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.513242 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/74dc4aee-10b1-463c-a9c3-522b7a6c10b7-host-run-netns\") pod \"multus-dsjsv\" (UID: \"74dc4aee-10b1-463c-a9c3-522b7a6c10b7\") " pod="openshift-multus/multus-dsjsv" Nov 03 13:07:47.513779 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.513267 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/74dc4aee-10b1-463c-a9c3-522b7a6c10b7-host-var-lib-cni-multus\") pod \"multus-dsjsv\" (UID: \"74dc4aee-10b1-463c-a9c3-522b7a6c10b7\") " pod="openshift-multus/multus-dsjsv" Nov 03 13:07:47.513779 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.513306 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/74dc4aee-10b1-463c-a9c3-522b7a6c10b7-multus-conf-dir\") pod \"multus-dsjsv\" (UID: \"74dc4aee-10b1-463c-a9c3-522b7a6c10b7\") " pod="openshift-multus/multus-dsjsv" Nov 03 13:07:47.513779 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.513333 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zpq8b\" (UniqueName: \"kubernetes.io/projected/39e11989-e598-4cea-87d9-26b3cf78a66b-kube-api-access-zpq8b\") pod \"node-ca-4tn4j\" (UID: \"39e11989-e598-4cea-87d9-26b3cf78a66b\") " pod="openshift-image-registry/node-ca-4tn4j" Nov 03 13:07:47.514447 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.513358 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c704eb02-d156-4695-a55d-7dc96654e4cd-sys\") pod \"tuned-hmbpc\" (UID: \"c704eb02-d156-4695-a55d-7dc96654e4cd\") " pod="openshift-cluster-node-tuning-operator/tuned-hmbpc" Nov 03 13:07:47.514447 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.513382 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d926d945-39f4-48f3-b3a4-0b302b15b69b-systemd-units\") pod \"ovnkube-node-fg6vh\" (UID: \"d926d945-39f4-48f3-b3a4-0b302b15b69b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fg6vh" Nov 03 13:07:47.514447 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.513409 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/246971bc-ed35-4fc7-b5f8-941327b90a94-agent-certs\") pod \"konnectivity-agent-bdg96\" (UID: \"246971bc-ed35-4fc7-b5f8-941327b90a94\") " pod="kube-system/konnectivity-agent-bdg96" Nov 03 13:07:47.514447 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.513434 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c704eb02-d156-4695-a55d-7dc96654e4cd-etc-kubernetes\") pod \"tuned-hmbpc\" (UID: \"c704eb02-d156-4695-a55d-7dc96654e4cd\") " pod="openshift-cluster-node-tuning-operator/tuned-hmbpc" Nov 03 13:07:47.514447 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.513458 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c704eb02-d156-4695-a55d-7dc96654e4cd-etc-sysctl-d\") pod \"tuned-hmbpc\" (UID: \"c704eb02-d156-4695-a55d-7dc96654e4cd\") " pod="openshift-cluster-node-tuning-operator/tuned-hmbpc" Nov 03 13:07:47.514447 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.513491 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/35e0fb40-48ce-4223-b7d4-c9290b6861cd-system-cni-dir\") pod \"multus-additional-cni-plugins-x2rm4\" (UID: \"35e0fb40-48ce-4223-b7d4-c9290b6861cd\") " pod="openshift-multus/multus-additional-cni-plugins-x2rm4" Nov 03 13:07:47.514447 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.513516 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/35e0fb40-48ce-4223-b7d4-c9290b6861cd-cnibin\") pod \"multus-additional-cni-plugins-x2rm4\" (UID: \"35e0fb40-48ce-4223-b7d4-c9290b6861cd\") " pod="openshift-multus/multus-additional-cni-plugins-x2rm4" Nov 03 13:07:47.514447 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.513540 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9db7d195-220e-4644-8690-9e5ee37fc066-kubelet-dir\") pod \"aws-ebs-csi-driver-node-c9dvm\" (UID: \"9db7d195-220e-4644-8690-9e5ee37fc066\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c9dvm" Nov 03 13:07:47.514447 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.513565 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9db7d195-220e-4644-8690-9e5ee37fc066-device-dir\") pod \"aws-ebs-csi-driver-node-c9dvm\" (UID: \"9db7d195-220e-4644-8690-9e5ee37fc066\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c9dvm" Nov 03 13:07:47.514447 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.513589 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d926d945-39f4-48f3-b3a4-0b302b15b69b-host-slash\") pod \"ovnkube-node-fg6vh\" (UID: \"d926d945-39f4-48f3-b3a4-0b302b15b69b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fg6vh" Nov 03 13:07:47.514447 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.513615 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d926d945-39f4-48f3-b3a4-0b302b15b69b-node-log\") pod \"ovnkube-node-fg6vh\" (UID: \"d926d945-39f4-48f3-b3a4-0b302b15b69b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fg6vh" Nov 03 13:07:47.514447 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.513639 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/74dc4aee-10b1-463c-a9c3-522b7a6c10b7-multus-socket-dir-parent\") pod \"multus-dsjsv\" (UID: \"74dc4aee-10b1-463c-a9c3-522b7a6c10b7\") " pod="openshift-multus/multus-dsjsv" Nov 03 13:07:47.514447 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.513663 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c704eb02-d156-4695-a55d-7dc96654e4cd-run\") pod \"tuned-hmbpc\" (UID: \"c704eb02-d156-4695-a55d-7dc96654e4cd\") " pod="openshift-cluster-node-tuning-operator/tuned-hmbpc" Nov 03 13:07:47.514447 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.513697 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c704eb02-d156-4695-a55d-7dc96654e4cd-tmp\") pod \"tuned-hmbpc\" (UID: \"c704eb02-d156-4695-a55d-7dc96654e4cd\") " pod="openshift-cluster-node-tuning-operator/tuned-hmbpc" Nov 03 13:07:47.514447 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.513722 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/35e0fb40-48ce-4223-b7d4-c9290b6861cd-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-x2rm4\" (UID: \"35e0fb40-48ce-4223-b7d4-c9290b6861cd\") " pod="openshift-multus/multus-additional-cni-plugins-x2rm4" Nov 03 13:07:47.514447 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.513750 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbfgb\" (UniqueName: \"kubernetes.io/projected/1bfc1afc-5b9f-4447-8d40-fabedbee9c07-kube-api-access-hbfgb\") pod \"network-check-target-8rdgq\" (UID: \"1bfc1afc-5b9f-4447-8d40-fabedbee9c07\") " pod="openshift-network-diagnostics/network-check-target-8rdgq" Nov 03 13:07:47.514447 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.513776 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/74dc4aee-10b1-463c-a9c3-522b7a6c10b7-host-run-k8s-cni-cncf-io\") pod \"multus-dsjsv\" (UID: \"74dc4aee-10b1-463c-a9c3-522b7a6c10b7\") " pod="openshift-multus/multus-dsjsv" Nov 03 13:07:47.515201 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.513804 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bcz25\" (UniqueName: \"kubernetes.io/projected/e0300f55-f002-4faa-bf6a-c04bea0a043e-kube-api-access-bcz25\") pod \"network-metrics-daemon-f7hmh\" (UID: \"e0300f55-f002-4faa-bf6a-c04bea0a043e\") " pod="openshift-multus/network-metrics-daemon-f7hmh" Nov 03 13:07:47.515201 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.514368 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/74dc4aee-10b1-463c-a9c3-522b7a6c10b7-system-cni-dir\") pod \"multus-dsjsv\" (UID: \"74dc4aee-10b1-463c-a9c3-522b7a6c10b7\") " pod="openshift-multus/multus-dsjsv" Nov 03 13:07:47.515201 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.514541 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/74dc4aee-10b1-463c-a9c3-522b7a6c10b7-host-run-multus-certs\") pod \"multus-dsjsv\" (UID: \"74dc4aee-10b1-463c-a9c3-522b7a6c10b7\") " pod="openshift-multus/multus-dsjsv" Nov 03 13:07:47.515201 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.514617 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c704eb02-d156-4695-a55d-7dc96654e4cd-etc-modprobe-d\") pod \"tuned-hmbpc\" (UID: \"c704eb02-d156-4695-a55d-7dc96654e4cd\") " pod="openshift-cluster-node-tuning-operator/tuned-hmbpc" Nov 03 13:07:47.515201 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.514834 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9db7d195-220e-4644-8690-9e5ee37fc066-registration-dir\") pod \"aws-ebs-csi-driver-node-c9dvm\" (UID: \"9db7d195-220e-4644-8690-9e5ee37fc066\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c9dvm" Nov 03 13:07:47.515467 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.515434 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/74dc4aee-10b1-463c-a9c3-522b7a6c10b7-multus-daemon-config\") pod \"multus-dsjsv\" (UID: \"74dc4aee-10b1-463c-a9c3-522b7a6c10b7\") " pod="openshift-multus/multus-dsjsv" Nov 03 13:07:47.515933 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.515894 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/fd7e8b58-de31-406a-8cdb-9ccfee140d57-iptables-alerter-script\") pod \"iptables-alerter-g4xgp\" (UID: \"fd7e8b58-de31-406a-8cdb-9ccfee140d57\") " pod="openshift-network-operator/iptables-alerter-g4xgp" Nov 03 13:07:47.516006 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.515989 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/74dc4aee-10b1-463c-a9c3-522b7a6c10b7-cnibin\") pod \"multus-dsjsv\" (UID: \"74dc4aee-10b1-463c-a9c3-522b7a6c10b7\") " pod="openshift-multus/multus-dsjsv" Nov 03 13:07:47.516239 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.516221 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/74dc4aee-10b1-463c-a9c3-522b7a6c10b7-host-var-lib-cni-bin\") pod \"multus-dsjsv\" (UID: \"74dc4aee-10b1-463c-a9c3-522b7a6c10b7\") " pod="openshift-multus/multus-dsjsv" Nov 03 13:07:47.516312 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.516276 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c704eb02-d156-4695-a55d-7dc96654e4cd-etc-sysconfig\") pod \"tuned-hmbpc\" (UID: \"c704eb02-d156-4695-a55d-7dc96654e4cd\") " pod="openshift-cluster-node-tuning-operator/tuned-hmbpc" Nov 03 13:07:47.516361 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.516339 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9db7d195-220e-4644-8690-9e5ee37fc066-sys-fs\") pod \"aws-ebs-csi-driver-node-c9dvm\" (UID: \"9db7d195-220e-4644-8690-9e5ee37fc066\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c9dvm" Nov 03 13:07:47.516399 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.516388 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/39e11989-e598-4cea-87d9-26b3cf78a66b-host\") pod \"node-ca-4tn4j\" (UID: \"39e11989-e598-4cea-87d9-26b3cf78a66b\") " pod="openshift-image-registry/node-ca-4tn4j" Nov 03 13:07:47.516780 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.516763 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/39e11989-e598-4cea-87d9-26b3cf78a66b-serviceca\") pod \"node-ca-4tn4j\" (UID: \"39e11989-e598-4cea-87d9-26b3cf78a66b\") " pod="openshift-image-registry/node-ca-4tn4j" Nov 03 13:07:47.516893 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.516879 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c704eb02-d156-4695-a55d-7dc96654e4cd-etc-sysctl-conf\") pod \"tuned-hmbpc\" (UID: \"c704eb02-d156-4695-a55d-7dc96654e4cd\") " pod="openshift-cluster-node-tuning-operator/tuned-hmbpc" Nov 03 13:07:47.516933 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.516919 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fd7e8b58-de31-406a-8cdb-9ccfee140d57-host-slash\") pod \"iptables-alerter-g4xgp\" (UID: \"fd7e8b58-de31-406a-8cdb-9ccfee140d57\") " pod="openshift-network-operator/iptables-alerter-g4xgp" Nov 03 13:07:47.517059 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.517031 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/35e0fb40-48ce-4223-b7d4-c9290b6861cd-tuning-conf-dir\") pod \"multus-additional-cni-plugins-x2rm4\" (UID: \"35e0fb40-48ce-4223-b7d4-c9290b6861cd\") " pod="openshift-multus/multus-additional-cni-plugins-x2rm4" Nov 03 13:07:47.517155 ip-10-0-131-36 kubenswrapper[2573]: E1103 13:07:47.517141 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 13:07:47.517224 ip-10-0-131-36 kubenswrapper[2573]: E1103 13:07:47.517214 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e0300f55-f002-4faa-bf6a-c04bea0a043e-metrics-certs podName:e0300f55-f002-4faa-bf6a-c04bea0a043e nodeName:}" failed. No retries permitted until 2026-11-03 13:07:48.01718498 +0000 UTC m=+2.991507602 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e0300f55-f002-4faa-bf6a-c04bea0a043e-metrics-certs") pod "network-metrics-daemon-f7hmh" (UID: "e0300f55-f002-4faa-bf6a-c04bea0a043e") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 13:07:47.517705 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.517686 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/35e0fb40-48ce-4223-b7d4-c9290b6861cd-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-x2rm4\" (UID: \"35e0fb40-48ce-4223-b7d4-c9290b6861cd\") " pod="openshift-multus/multus-additional-cni-plugins-x2rm4" Nov 03 13:07:47.517803 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.517789 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9db7d195-220e-4644-8690-9e5ee37fc066-etc-selinux\") pod \"aws-ebs-csi-driver-node-c9dvm\" (UID: \"9db7d195-220e-4644-8690-9e5ee37fc066\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c9dvm" Nov 03 13:07:47.518026 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.517988 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/74dc4aee-10b1-463c-a9c3-522b7a6c10b7-multus-cni-dir\") pod \"multus-dsjsv\" (UID: \"74dc4aee-10b1-463c-a9c3-522b7a6c10b7\") " pod="openshift-multus/multus-dsjsv" Nov 03 13:07:47.518233 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.518207 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c704eb02-d156-4695-a55d-7dc96654e4cd-host\") pod \"tuned-hmbpc\" (UID: \"c704eb02-d156-4695-a55d-7dc96654e4cd\") " pod="openshift-cluster-node-tuning-operator/tuned-hmbpc" Nov 03 13:07:47.518341 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.518325 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9db7d195-220e-4644-8690-9e5ee37fc066-socket-dir\") pod \"aws-ebs-csi-driver-node-c9dvm\" (UID: \"9db7d195-220e-4644-8690-9e5ee37fc066\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c9dvm" Nov 03 13:07:47.518789 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.518768 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/74dc4aee-10b1-463c-a9c3-522b7a6c10b7-cni-binary-copy\") pod \"multus-dsjsv\" (UID: \"74dc4aee-10b1-463c-a9c3-522b7a6c10b7\") " pod="openshift-multus/multus-dsjsv" Nov 03 13:07:47.518873 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.518829 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c704eb02-d156-4695-a55d-7dc96654e4cd-etc-systemd\") pod \"tuned-hmbpc\" (UID: \"c704eb02-d156-4695-a55d-7dc96654e4cd\") " pod="openshift-cluster-node-tuning-operator/tuned-hmbpc" Nov 03 13:07:47.518944 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.518885 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/74dc4aee-10b1-463c-a9c3-522b7a6c10b7-host-var-lib-kubelet\") pod \"multus-dsjsv\" (UID: \"74dc4aee-10b1-463c-a9c3-522b7a6c10b7\") " pod="openshift-multus/multus-dsjsv" Nov 03 13:07:47.519032 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.519018 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c704eb02-d156-4695-a55d-7dc96654e4cd-lib-modules\") pod \"tuned-hmbpc\" (UID: \"c704eb02-d156-4695-a55d-7dc96654e4cd\") " pod="openshift-cluster-node-tuning-operator/tuned-hmbpc" Nov 03 13:07:47.519087 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.519073 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c704eb02-d156-4695-a55d-7dc96654e4cd-var-lib-kubelet\") pod \"tuned-hmbpc\" (UID: \"c704eb02-d156-4695-a55d-7dc96654e4cd\") " pod="openshift-cluster-node-tuning-operator/tuned-hmbpc" Nov 03 13:07:47.519458 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.519422 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c704eb02-d156-4695-a55d-7dc96654e4cd-etc-kubernetes\") pod \"tuned-hmbpc\" (UID: \"c704eb02-d156-4695-a55d-7dc96654e4cd\") " pod="openshift-cluster-node-tuning-operator/tuned-hmbpc" Nov 03 13:07:47.519554 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.519502 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/35e0fb40-48ce-4223-b7d4-c9290b6861cd-cni-binary-copy\") pod \"multus-additional-cni-plugins-x2rm4\" (UID: \"35e0fb40-48ce-4223-b7d4-c9290b6861cd\") " pod="openshift-multus/multus-additional-cni-plugins-x2rm4" Nov 03 13:07:47.519554 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.519510 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c704eb02-d156-4695-a55d-7dc96654e4cd-run\") pod \"tuned-hmbpc\" (UID: \"c704eb02-d156-4695-a55d-7dc96654e4cd\") " pod="openshift-cluster-node-tuning-operator/tuned-hmbpc" Nov 03 13:07:47.519653 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.519557 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/74dc4aee-10b1-463c-a9c3-522b7a6c10b7-host-run-k8s-cni-cncf-io\") pod \"multus-dsjsv\" (UID: \"74dc4aee-10b1-463c-a9c3-522b7a6c10b7\") " pod="openshift-multus/multus-dsjsv" Nov 03 13:07:47.519653 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.519569 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c704eb02-d156-4695-a55d-7dc96654e4cd-etc-sysctl-d\") pod \"tuned-hmbpc\" (UID: \"c704eb02-d156-4695-a55d-7dc96654e4cd\") " pod="openshift-cluster-node-tuning-operator/tuned-hmbpc" Nov 03 13:07:47.519653 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.519620 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/35e0fb40-48ce-4223-b7d4-c9290b6861cd-system-cni-dir\") pod \"multus-additional-cni-plugins-x2rm4\" (UID: \"35e0fb40-48ce-4223-b7d4-c9290b6861cd\") " pod="openshift-multus/multus-additional-cni-plugins-x2rm4" Nov 03 13:07:47.519801 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.519672 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/35e0fb40-48ce-4223-b7d4-c9290b6861cd-cnibin\") pod \"multus-additional-cni-plugins-x2rm4\" (UID: \"35e0fb40-48ce-4223-b7d4-c9290b6861cd\") " pod="openshift-multus/multus-additional-cni-plugins-x2rm4" Nov 03 13:07:47.519801 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.519738 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9db7d195-220e-4644-8690-9e5ee37fc066-kubelet-dir\") pod \"aws-ebs-csi-driver-node-c9dvm\" (UID: \"9db7d195-220e-4644-8690-9e5ee37fc066\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c9dvm" Nov 03 13:07:47.519801 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.519782 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/74dc4aee-10b1-463c-a9c3-522b7a6c10b7-host-run-netns\") pod \"multus-dsjsv\" (UID: \"74dc4aee-10b1-463c-a9c3-522b7a6c10b7\") " pod="openshift-multus/multus-dsjsv" Nov 03 13:07:47.519801 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.519785 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9db7d195-220e-4644-8690-9e5ee37fc066-device-dir\") pod \"aws-ebs-csi-driver-node-c9dvm\" (UID: \"9db7d195-220e-4644-8690-9e5ee37fc066\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c9dvm" Nov 03 13:07:47.520032 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.519817 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/74dc4aee-10b1-463c-a9c3-522b7a6c10b7-hostroot\") pod \"multus-dsjsv\" (UID: \"74dc4aee-10b1-463c-a9c3-522b7a6c10b7\") " pod="openshift-multus/multus-dsjsv" Nov 03 13:07:47.520032 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.519820 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/74dc4aee-10b1-463c-a9c3-522b7a6c10b7-host-var-lib-cni-multus\") pod \"multus-dsjsv\" (UID: \"74dc4aee-10b1-463c-a9c3-522b7a6c10b7\") " pod="openshift-multus/multus-dsjsv" Nov 03 13:07:47.520032 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.519871 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/74dc4aee-10b1-463c-a9c3-522b7a6c10b7-multus-conf-dir\") pod \"multus-dsjsv\" (UID: \"74dc4aee-10b1-463c-a9c3-522b7a6c10b7\") " pod="openshift-multus/multus-dsjsv" Nov 03 13:07:47.520032 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.519908 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/74dc4aee-10b1-463c-a9c3-522b7a6c10b7-etc-kubernetes\") pod \"multus-dsjsv\" (UID: \"74dc4aee-10b1-463c-a9c3-522b7a6c10b7\") " pod="openshift-multus/multus-dsjsv" Nov 03 13:07:47.520226 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.520100 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/35e0fb40-48ce-4223-b7d4-c9290b6861cd-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-x2rm4\" (UID: \"35e0fb40-48ce-4223-b7d4-c9290b6861cd\") " pod="openshift-multus/multus-additional-cni-plugins-x2rm4" Nov 03 13:07:47.520226 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.520098 2573 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 03 13:07:47.520226 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.520151 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c704eb02-d156-4695-a55d-7dc96654e4cd-sys\") pod \"tuned-hmbpc\" (UID: \"c704eb02-d156-4695-a55d-7dc96654e4cd\") " pod="openshift-cluster-node-tuning-operator/tuned-hmbpc" Nov 03 13:07:47.520226 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.519737 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/74dc4aee-10b1-463c-a9c3-522b7a6c10b7-multus-socket-dir-parent\") pod \"multus-dsjsv\" (UID: \"74dc4aee-10b1-463c-a9c3-522b7a6c10b7\") " pod="openshift-multus/multus-dsjsv" Nov 03 13:07:47.524056 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.524038 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c704eb02-d156-4695-a55d-7dc96654e4cd-etc-tuned\") pod \"tuned-hmbpc\" (UID: \"c704eb02-d156-4695-a55d-7dc96654e4cd\") " pod="openshift-cluster-node-tuning-operator/tuned-hmbpc" Nov 03 13:07:47.524149 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.524081 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c704eb02-d156-4695-a55d-7dc96654e4cd-tmp\") pod \"tuned-hmbpc\" (UID: \"c704eb02-d156-4695-a55d-7dc96654e4cd\") " pod="openshift-cluster-node-tuning-operator/tuned-hmbpc" Nov 03 13:07:47.524989 ip-10-0-131-36 kubenswrapper[2573]: E1103 13:07:47.524970 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 13:07:47.525066 ip-10-0-131-36 kubenswrapper[2573]: E1103 13:07:47.524995 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 13:07:47.525066 ip-10-0-131-36 kubenswrapper[2573]: E1103 13:07:47.525008 2573 projected.go:196] Error preparing data for projected volume kube-api-access-hbfgb for pod openshift-network-diagnostics/network-check-target-8rdgq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 13:07:47.525066 ip-10-0-131-36 kubenswrapper[2573]: E1103 13:07:47.525060 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1bfc1afc-5b9f-4447-8d40-fabedbee9c07-kube-api-access-hbfgb podName:1bfc1afc-5b9f-4447-8d40-fabedbee9c07 nodeName:}" failed. No retries permitted until 2026-11-03 13:07:48.025043049 +0000 UTC m=+2.999365673 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-hbfgb" (UniqueName: "kubernetes.io/projected/1bfc1afc-5b9f-4447-8d40-fabedbee9c07-kube-api-access-hbfgb") pod "network-check-target-8rdgq" (UID: "1bfc1afc-5b9f-4447-8d40-fabedbee9c07") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 13:07:47.526956 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.526898 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ftcqc\" (UniqueName: \"kubernetes.io/projected/9db7d195-220e-4644-8690-9e5ee37fc066-kube-api-access-ftcqc\") pod \"aws-ebs-csi-driver-node-c9dvm\" (UID: \"9db7d195-220e-4644-8690-9e5ee37fc066\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c9dvm" Nov 03 13:07:47.528914 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.528826 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zpq8b\" (UniqueName: \"kubernetes.io/projected/39e11989-e598-4cea-87d9-26b3cf78a66b-kube-api-access-zpq8b\") pod \"node-ca-4tn4j\" (UID: \"39e11989-e598-4cea-87d9-26b3cf78a66b\") " pod="openshift-image-registry/node-ca-4tn4j" Nov 03 13:07:47.528914 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.528826 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mss42\" (UniqueName: \"kubernetes.io/projected/35e0fb40-48ce-4223-b7d4-c9290b6861cd-kube-api-access-mss42\") pod \"multus-additional-cni-plugins-x2rm4\" (UID: \"35e0fb40-48ce-4223-b7d4-c9290b6861cd\") " pod="openshift-multus/multus-additional-cni-plugins-x2rm4" Nov 03 13:07:47.528914 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.528848 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-snpf4\" (UniqueName: \"kubernetes.io/projected/fd7e8b58-de31-406a-8cdb-9ccfee140d57-kube-api-access-snpf4\") pod \"iptables-alerter-g4xgp\" (UID: \"fd7e8b58-de31-406a-8cdb-9ccfee140d57\") " pod="openshift-network-operator/iptables-alerter-g4xgp" Nov 03 13:07:47.528914 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.528888 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwrs7\" (UniqueName: \"kubernetes.io/projected/74dc4aee-10b1-463c-a9c3-522b7a6c10b7-kube-api-access-cwrs7\") pod \"multus-dsjsv\" (UID: \"74dc4aee-10b1-463c-a9c3-522b7a6c10b7\") " pod="openshift-multus/multus-dsjsv" Nov 03 13:07:47.529112 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.528939 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4j5g\" (UniqueName: \"kubernetes.io/projected/c704eb02-d156-4695-a55d-7dc96654e4cd-kube-api-access-w4j5g\") pod \"tuned-hmbpc\" (UID: \"c704eb02-d156-4695-a55d-7dc96654e4cd\") " pod="openshift-cluster-node-tuning-operator/tuned-hmbpc" Nov 03 13:07:47.529516 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.529494 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bcz25\" (UniqueName: \"kubernetes.io/projected/e0300f55-f002-4faa-bf6a-c04bea0a043e-kube-api-access-bcz25\") pod \"network-metrics-daemon-f7hmh\" (UID: \"e0300f55-f002-4faa-bf6a-c04bea0a043e\") " pod="openshift-multus/network-metrics-daemon-f7hmh" Nov 03 13:07:47.614316 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.614268 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d926d945-39f4-48f3-b3a4-0b302b15b69b-host-run-netns\") pod \"ovnkube-node-fg6vh\" (UID: \"d926d945-39f4-48f3-b3a4-0b302b15b69b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fg6vh" Nov 03 13:07:47.614488 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.614325 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d926d945-39f4-48f3-b3a4-0b302b15b69b-etc-openvswitch\") pod \"ovnkube-node-fg6vh\" (UID: \"d926d945-39f4-48f3-b3a4-0b302b15b69b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fg6vh" Nov 03 13:07:47.614488 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.614351 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d926d945-39f4-48f3-b3a4-0b302b15b69b-run-ovn\") pod \"ovnkube-node-fg6vh\" (UID: \"d926d945-39f4-48f3-b3a4-0b302b15b69b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fg6vh" Nov 03 13:07:47.614488 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.614381 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d926d945-39f4-48f3-b3a4-0b302b15b69b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-fg6vh\" (UID: \"d926d945-39f4-48f3-b3a4-0b302b15b69b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fg6vh" Nov 03 13:07:47.614488 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.614395 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d926d945-39f4-48f3-b3a4-0b302b15b69b-host-run-netns\") pod \"ovnkube-node-fg6vh\" (UID: \"d926d945-39f4-48f3-b3a4-0b302b15b69b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fg6vh" Nov 03 13:07:47.614488 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.614410 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d926d945-39f4-48f3-b3a4-0b302b15b69b-ovnkube-script-lib\") pod \"ovnkube-node-fg6vh\" (UID: \"d926d945-39f4-48f3-b3a4-0b302b15b69b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fg6vh" Nov 03 13:07:47.614488 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.614434 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d926d945-39f4-48f3-b3a4-0b302b15b69b-run-systemd\") pod \"ovnkube-node-fg6vh\" (UID: \"d926d945-39f4-48f3-b3a4-0b302b15b69b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fg6vh" Nov 03 13:07:47.614488 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.614458 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d926d945-39f4-48f3-b3a4-0b302b15b69b-host-run-ovn-kubernetes\") pod \"ovnkube-node-fg6vh\" (UID: \"d926d945-39f4-48f3-b3a4-0b302b15b69b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fg6vh" Nov 03 13:07:47.614488 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.614463 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d926d945-39f4-48f3-b3a4-0b302b15b69b-etc-openvswitch\") pod \"ovnkube-node-fg6vh\" (UID: \"d926d945-39f4-48f3-b3a4-0b302b15b69b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fg6vh" Nov 03 13:07:47.614488 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.614464 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d926d945-39f4-48f3-b3a4-0b302b15b69b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-fg6vh\" (UID: \"d926d945-39f4-48f3-b3a4-0b302b15b69b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fg6vh" Nov 03 13:07:47.614488 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.614480 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/5ea5cf61-ed1c-4f5e-b1a0-5c46c5c3ee18-hosts-file\") pod \"node-resolver-v8gcq\" (UID: \"5ea5cf61-ed1c-4f5e-b1a0-5c46c5c3ee18\") " pod="openshift-dns/node-resolver-v8gcq" Nov 03 13:07:47.614915 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.614512 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d926d945-39f4-48f3-b3a4-0b302b15b69b-run-ovn\") pod \"ovnkube-node-fg6vh\" (UID: \"d926d945-39f4-48f3-b3a4-0b302b15b69b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fg6vh" Nov 03 13:07:47.614915 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.614525 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d926d945-39f4-48f3-b3a4-0b302b15b69b-systemd-units\") pod \"ovnkube-node-fg6vh\" (UID: \"d926d945-39f4-48f3-b3a4-0b302b15b69b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fg6vh" Nov 03 13:07:47.614915 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.614531 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d926d945-39f4-48f3-b3a4-0b302b15b69b-run-systemd\") pod \"ovnkube-node-fg6vh\" (UID: \"d926d945-39f4-48f3-b3a4-0b302b15b69b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fg6vh" Nov 03 13:07:47.614915 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.614552 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/246971bc-ed35-4fc7-b5f8-941327b90a94-agent-certs\") pod \"konnectivity-agent-bdg96\" (UID: \"246971bc-ed35-4fc7-b5f8-941327b90a94\") " pod="kube-system/konnectivity-agent-bdg96" Nov 03 13:07:47.614915 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.614566 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d926d945-39f4-48f3-b3a4-0b302b15b69b-host-run-ovn-kubernetes\") pod \"ovnkube-node-fg6vh\" (UID: \"d926d945-39f4-48f3-b3a4-0b302b15b69b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fg6vh" Nov 03 13:07:47.614915 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.614580 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d926d945-39f4-48f3-b3a4-0b302b15b69b-host-slash\") pod \"ovnkube-node-fg6vh\" (UID: \"d926d945-39f4-48f3-b3a4-0b302b15b69b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fg6vh" Nov 03 13:07:47.614915 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.614603 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d926d945-39f4-48f3-b3a4-0b302b15b69b-node-log\") pod \"ovnkube-node-fg6vh\" (UID: \"d926d945-39f4-48f3-b3a4-0b302b15b69b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fg6vh" Nov 03 13:07:47.614915 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.614609 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d926d945-39f4-48f3-b3a4-0b302b15b69b-systemd-units\") pod \"ovnkube-node-fg6vh\" (UID: \"d926d945-39f4-48f3-b3a4-0b302b15b69b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fg6vh" Nov 03 13:07:47.614915 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.614646 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d926d945-39f4-48f3-b3a4-0b302b15b69b-env-overrides\") pod \"ovnkube-node-fg6vh\" (UID: \"d926d945-39f4-48f3-b3a4-0b302b15b69b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fg6vh" Nov 03 13:07:47.614915 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.614666 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/5ea5cf61-ed1c-4f5e-b1a0-5c46c5c3ee18-hosts-file\") pod \"node-resolver-v8gcq\" (UID: \"5ea5cf61-ed1c-4f5e-b1a0-5c46c5c3ee18\") " pod="openshift-dns/node-resolver-v8gcq" Nov 03 13:07:47.614915 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.614667 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7e280277-f29f-4a2b-8b85-2491529e5ef2-original-pull-secret\") pod \"global-pull-secret-syncer-5jz7t\" (UID: \"7e280277-f29f-4a2b-8b85-2491529e5ef2\") " pod="kube-system/global-pull-secret-syncer-5jz7t" Nov 03 13:07:47.614915 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.614708 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/246971bc-ed35-4fc7-b5f8-941327b90a94-konnectivity-ca\") pod \"konnectivity-agent-bdg96\" (UID: \"246971bc-ed35-4fc7-b5f8-941327b90a94\") " pod="kube-system/konnectivity-agent-bdg96" Nov 03 13:07:47.614915 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.614731 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5ea5cf61-ed1c-4f5e-b1a0-5c46c5c3ee18-tmp-dir\") pod \"node-resolver-v8gcq\" (UID: \"5ea5cf61-ed1c-4f5e-b1a0-5c46c5c3ee18\") " pod="openshift-dns/node-resolver-v8gcq" Nov 03 13:07:47.614915 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.614756 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6b28l\" (UniqueName: \"kubernetes.io/projected/d926d945-39f4-48f3-b3a4-0b302b15b69b-kube-api-access-6b28l\") pod \"ovnkube-node-fg6vh\" (UID: \"d926d945-39f4-48f3-b3a4-0b302b15b69b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fg6vh" Nov 03 13:07:47.614915 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.614777 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjhjt\" (UniqueName: \"kubernetes.io/projected/5ea5cf61-ed1c-4f5e-b1a0-5c46c5c3ee18-kube-api-access-vjhjt\") pod \"node-resolver-v8gcq\" (UID: \"5ea5cf61-ed1c-4f5e-b1a0-5c46c5c3ee18\") " pod="openshift-dns/node-resolver-v8gcq" Nov 03 13:07:47.614915 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.614793 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d926d945-39f4-48f3-b3a4-0b302b15b69b-log-socket\") pod \"ovnkube-node-fg6vh\" (UID: \"d926d945-39f4-48f3-b3a4-0b302b15b69b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fg6vh" Nov 03 13:07:47.614915 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.614810 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d926d945-39f4-48f3-b3a4-0b302b15b69b-var-lib-openvswitch\") pod \"ovnkube-node-fg6vh\" (UID: \"d926d945-39f4-48f3-b3a4-0b302b15b69b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fg6vh" Nov 03 13:07:47.614915 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.614827 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d926d945-39f4-48f3-b3a4-0b302b15b69b-host-kubelet\") pod \"ovnkube-node-fg6vh\" (UID: \"d926d945-39f4-48f3-b3a4-0b302b15b69b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fg6vh" Nov 03 13:07:47.615445 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.614840 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d926d945-39f4-48f3-b3a4-0b302b15b69b-host-cni-bin\") pod \"ovnkube-node-fg6vh\" (UID: \"d926d945-39f4-48f3-b3a4-0b302b15b69b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fg6vh" Nov 03 13:07:47.615445 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.614853 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d926d945-39f4-48f3-b3a4-0b302b15b69b-host-cni-netd\") pod \"ovnkube-node-fg6vh\" (UID: \"d926d945-39f4-48f3-b3a4-0b302b15b69b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fg6vh" Nov 03 13:07:47.615445 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.614876 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d926d945-39f4-48f3-b3a4-0b302b15b69b-run-openvswitch\") pod \"ovnkube-node-fg6vh\" (UID: \"d926d945-39f4-48f3-b3a4-0b302b15b69b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fg6vh" Nov 03 13:07:47.615445 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.614890 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/7e280277-f29f-4a2b-8b85-2491529e5ef2-dbus\") pod \"global-pull-secret-syncer-5jz7t\" (UID: \"7e280277-f29f-4a2b-8b85-2491529e5ef2\") " pod="kube-system/global-pull-secret-syncer-5jz7t" Nov 03 13:07:47.615445 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.614908 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d926d945-39f4-48f3-b3a4-0b302b15b69b-ovnkube-config\") pod \"ovnkube-node-fg6vh\" (UID: \"d926d945-39f4-48f3-b3a4-0b302b15b69b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fg6vh" Nov 03 13:07:47.615445 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.614924 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/7e280277-f29f-4a2b-8b85-2491529e5ef2-kubelet-config\") pod \"global-pull-secret-syncer-5jz7t\" (UID: \"7e280277-f29f-4a2b-8b85-2491529e5ef2\") " pod="kube-system/global-pull-secret-syncer-5jz7t" Nov 03 13:07:47.615445 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.614943 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d926d945-39f4-48f3-b3a4-0b302b15b69b-ovn-node-metrics-cert\") pod \"ovnkube-node-fg6vh\" (UID: \"d926d945-39f4-48f3-b3a4-0b302b15b69b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fg6vh" Nov 03 13:07:47.615445 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.614982 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d926d945-39f4-48f3-b3a4-0b302b15b69b-var-lib-openvswitch\") pod \"ovnkube-node-fg6vh\" (UID: \"d926d945-39f4-48f3-b3a4-0b302b15b69b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fg6vh" Nov 03 13:07:47.615445 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.615038 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d926d945-39f4-48f3-b3a4-0b302b15b69b-host-kubelet\") pod \"ovnkube-node-fg6vh\" (UID: \"d926d945-39f4-48f3-b3a4-0b302b15b69b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fg6vh" Nov 03 13:07:47.615445 ip-10-0-131-36 kubenswrapper[2573]: E1103 13:07:47.615067 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 13:07:47.615445 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.615089 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d926d945-39f4-48f3-b3a4-0b302b15b69b-host-cni-netd\") pod \"ovnkube-node-fg6vh\" (UID: \"d926d945-39f4-48f3-b3a4-0b302b15b69b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fg6vh" Nov 03 13:07:47.615445 ip-10-0-131-36 kubenswrapper[2573]: E1103 13:07:47.615142 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7e280277-f29f-4a2b-8b85-2491529e5ef2-original-pull-secret podName:7e280277-f29f-4a2b-8b85-2491529e5ef2 nodeName:}" failed. No retries permitted until 2026-11-03 13:07:48.11512338 +0000 UTC m=+3.089446001 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7e280277-f29f-4a2b-8b85-2491529e5ef2-original-pull-secret") pod "global-pull-secret-syncer-5jz7t" (UID: "7e280277-f29f-4a2b-8b85-2491529e5ef2") : object "kube-system"/"original-pull-secret" not registered Nov 03 13:07:47.615445 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.615156 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d926d945-39f4-48f3-b3a4-0b302b15b69b-ovnkube-script-lib\") pod \"ovnkube-node-fg6vh\" (UID: \"d926d945-39f4-48f3-b3a4-0b302b15b69b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fg6vh" Nov 03 13:07:47.615445 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.615438 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d926d945-39f4-48f3-b3a4-0b302b15b69b-log-socket\") pod \"ovnkube-node-fg6vh\" (UID: \"d926d945-39f4-48f3-b3a4-0b302b15b69b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fg6vh" Nov 03 13:07:47.615931 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.615469 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d926d945-39f4-48f3-b3a4-0b302b15b69b-ovnkube-config\") pod \"ovnkube-node-fg6vh\" (UID: \"d926d945-39f4-48f3-b3a4-0b302b15b69b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fg6vh" Nov 03 13:07:47.615931 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.615481 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d926d945-39f4-48f3-b3a4-0b302b15b69b-run-openvswitch\") pod \"ovnkube-node-fg6vh\" (UID: \"d926d945-39f4-48f3-b3a4-0b302b15b69b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fg6vh" Nov 03 13:07:47.615931 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.615071 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d926d945-39f4-48f3-b3a4-0b302b15b69b-host-cni-bin\") pod \"ovnkube-node-fg6vh\" (UID: \"d926d945-39f4-48f3-b3a4-0b302b15b69b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fg6vh" Nov 03 13:07:47.615931 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.615526 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/7e280277-f29f-4a2b-8b85-2491529e5ef2-dbus\") pod \"global-pull-secret-syncer-5jz7t\" (UID: \"7e280277-f29f-4a2b-8b85-2491529e5ef2\") " pod="kube-system/global-pull-secret-syncer-5jz7t" Nov 03 13:07:47.615931 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.615550 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/7e280277-f29f-4a2b-8b85-2491529e5ef2-kubelet-config\") pod \"global-pull-secret-syncer-5jz7t\" (UID: \"7e280277-f29f-4a2b-8b85-2491529e5ef2\") " pod="kube-system/global-pull-secret-syncer-5jz7t" Nov 03 13:07:47.615931 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.615556 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d926d945-39f4-48f3-b3a4-0b302b15b69b-node-log\") pod \"ovnkube-node-fg6vh\" (UID: \"d926d945-39f4-48f3-b3a4-0b302b15b69b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fg6vh" Nov 03 13:07:47.615931 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.615582 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d926d945-39f4-48f3-b3a4-0b302b15b69b-host-slash\") pod \"ovnkube-node-fg6vh\" (UID: \"d926d945-39f4-48f3-b3a4-0b302b15b69b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fg6vh" Nov 03 13:07:47.616421 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.616398 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5ea5cf61-ed1c-4f5e-b1a0-5c46c5c3ee18-tmp-dir\") pod \"node-resolver-v8gcq\" (UID: \"5ea5cf61-ed1c-4f5e-b1a0-5c46c5c3ee18\") " pod="openshift-dns/node-resolver-v8gcq" Nov 03 13:07:47.616662 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.616647 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/246971bc-ed35-4fc7-b5f8-941327b90a94-konnectivity-ca\") pod \"konnectivity-agent-bdg96\" (UID: \"246971bc-ed35-4fc7-b5f8-941327b90a94\") " pod="kube-system/konnectivity-agent-bdg96" Nov 03 13:07:47.616744 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.616727 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d926d945-39f4-48f3-b3a4-0b302b15b69b-env-overrides\") pod \"ovnkube-node-fg6vh\" (UID: \"d926d945-39f4-48f3-b3a4-0b302b15b69b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fg6vh" Nov 03 13:07:47.617563 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.617540 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/246971bc-ed35-4fc7-b5f8-941327b90a94-agent-certs\") pod \"konnectivity-agent-bdg96\" (UID: \"246971bc-ed35-4fc7-b5f8-941327b90a94\") " pod="kube-system/konnectivity-agent-bdg96" Nov 03 13:07:47.619323 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.619299 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d926d945-39f4-48f3-b3a4-0b302b15b69b-ovn-node-metrics-cert\") pod \"ovnkube-node-fg6vh\" (UID: \"d926d945-39f4-48f3-b3a4-0b302b15b69b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fg6vh" Nov 03 13:07:47.623419 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.623393 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjhjt\" (UniqueName: \"kubernetes.io/projected/5ea5cf61-ed1c-4f5e-b1a0-5c46c5c3ee18-kube-api-access-vjhjt\") pod \"node-resolver-v8gcq\" (UID: \"5ea5cf61-ed1c-4f5e-b1a0-5c46c5c3ee18\") " pod="openshift-dns/node-resolver-v8gcq" Nov 03 13:07:47.623926 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.623903 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6b28l\" (UniqueName: \"kubernetes.io/projected/d926d945-39f4-48f3-b3a4-0b302b15b69b-kube-api-access-6b28l\") pod \"ovnkube-node-fg6vh\" (UID: \"d926d945-39f4-48f3-b3a4-0b302b15b69b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fg6vh" Nov 03 13:07:47.634123 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.634031 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-36.us-east-2.compute.internal" event={"ID":"4879f52c5c2e658a43379722bc61381e","Type":"ContainerStarted","Data":"5d20773640ce8e198ac75606401c820525fb0fe1350882ef0b9b2ed7d166824a"} Nov 03 13:07:47.635473 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.635451 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-36.us-east-2.compute.internal" event={"ID":"5ee88558fb38141d796a31757de861ab","Type":"ContainerStarted","Data":"5b44ee172423f97b68d439f1263e56f195b4502d78f9d58d2899e3b7eee3152f"} Nov 03 13:07:47.700370 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.700334 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-hmbpc" Nov 03 13:07:47.709829 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.709801 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-dsjsv" Nov 03 13:07:47.710033 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:47.710010 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc704eb02_d156_4695_a55d_7dc96654e4cd.slice/crio-76516e589de873b36b6de4e04a437e5999788d1915cb8e25b57c8d8ca1a0abf4 WatchSource:0}: Error finding container 76516e589de873b36b6de4e04a437e5999788d1915cb8e25b57c8d8ca1a0abf4: Status 404 returned error can't find the container with id 76516e589de873b36b6de4e04a437e5999788d1915cb8e25b57c8d8ca1a0abf4 Nov 03 13:07:47.718805 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.718785 2573 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 13:07:47.722043 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.722022 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-g4xgp" Nov 03 13:07:47.733995 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:47.733956 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd7e8b58_de31_406a_8cdb_9ccfee140d57.slice/crio-080b34934d5649c436eb755c96cb75220add975b57d1a9023728b6782facbc86 WatchSource:0}: Error finding container 080b34934d5649c436eb755c96cb75220add975b57d1a9023728b6782facbc86: Status 404 returned error can't find the container with id 080b34934d5649c436eb755c96cb75220add975b57d1a9023728b6782facbc86 Nov 03 13:07:47.749672 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.749636 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-fg6vh" Nov 03 13:07:47.753917 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.753362 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-4tn4j" Nov 03 13:07:47.760517 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:47.760488 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd926d945_39f4_48f3_b3a4_0b302b15b69b.slice/crio-39aa5c2c75a1203c58dd67810fc2d1718f5535ce2d67634692f6a034d4a8fe41 WatchSource:0}: Error finding container 39aa5c2c75a1203c58dd67810fc2d1718f5535ce2d67634692f6a034d4a8fe41: Status 404 returned error can't find the container with id 39aa5c2c75a1203c58dd67810fc2d1718f5535ce2d67634692f6a034d4a8fe41 Nov 03 13:07:47.763765 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:47.763736 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39e11989_e598_4cea_87d9_26b3cf78a66b.slice/crio-c29cb1f6582414bd2e24af736205bd2de388faaebcd65096615b22499cfdaf05 WatchSource:0}: Error finding container c29cb1f6582414bd2e24af736205bd2de388faaebcd65096615b22499cfdaf05: Status 404 returned error can't find the container with id c29cb1f6582414bd2e24af736205bd2de388faaebcd65096615b22499cfdaf05 Nov 03 13:07:47.771987 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.771960 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-v8gcq" Nov 03 13:07:47.774047 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.774025 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c9dvm" Nov 03 13:07:47.776925 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.776899 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-x2rm4" Nov 03 13:07:47.778600 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:47.778556 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-bdg96" Nov 03 13:07:47.787317 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:47.787275 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9db7d195_220e_4644_8690_9e5ee37fc066.slice/crio-2714947ff73646c1e92b308d87f7aa0ae91cd24f08a9ed96681ba260c6f6d555 WatchSource:0}: Error finding container 2714947ff73646c1e92b308d87f7aa0ae91cd24f08a9ed96681ba260c6f6d555: Status 404 returned error can't find the container with id 2714947ff73646c1e92b308d87f7aa0ae91cd24f08a9ed96681ba260c6f6d555 Nov 03 13:07:47.792745 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:47.792706 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod246971bc_ed35_4fc7_b5f8_941327b90a94.slice/crio-ecf7ff9bffd580730bcdf842700eb81895af4e3d420ec626d39da9261adba3ee WatchSource:0}: Error finding container ecf7ff9bffd580730bcdf842700eb81895af4e3d420ec626d39da9261adba3ee: Status 404 returned error can't find the container with id ecf7ff9bffd580730bcdf842700eb81895af4e3d420ec626d39da9261adba3ee Nov 03 13:07:47.793929 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:07:47.793908 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35e0fb40_48ce_4223_b7d4_c9290b6861cd.slice/crio-e6c2867616afcbdf7d2a5611abc303a745fe5f9dece80d4859a108a05d8f4438 WatchSource:0}: Error finding container e6c2867616afcbdf7d2a5611abc303a745fe5f9dece80d4859a108a05d8f4438: Status 404 returned error can't find the container with id e6c2867616afcbdf7d2a5611abc303a745fe5f9dece80d4859a108a05d8f4438 Nov 03 13:07:48.118205 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:48.118125 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e0300f55-f002-4faa-bf6a-c04bea0a043e-metrics-certs\") pod \"network-metrics-daemon-f7hmh\" (UID: \"e0300f55-f002-4faa-bf6a-c04bea0a043e\") " pod="openshift-multus/network-metrics-daemon-f7hmh" Nov 03 13:07:48.118383 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:48.118201 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbfgb\" (UniqueName: \"kubernetes.io/projected/1bfc1afc-5b9f-4447-8d40-fabedbee9c07-kube-api-access-hbfgb\") pod \"network-check-target-8rdgq\" (UID: \"1bfc1afc-5b9f-4447-8d40-fabedbee9c07\") " pod="openshift-network-diagnostics/network-check-target-8rdgq" Nov 03 13:07:48.118383 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:48.118240 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7e280277-f29f-4a2b-8b85-2491529e5ef2-original-pull-secret\") pod \"global-pull-secret-syncer-5jz7t\" (UID: \"7e280277-f29f-4a2b-8b85-2491529e5ef2\") " pod="kube-system/global-pull-secret-syncer-5jz7t" Nov 03 13:07:48.118383 ip-10-0-131-36 kubenswrapper[2573]: E1103 13:07:48.118318 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 13:07:48.118383 ip-10-0-131-36 kubenswrapper[2573]: E1103 13:07:48.118359 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 13:07:48.118530 ip-10-0-131-36 kubenswrapper[2573]: E1103 13:07:48.118395 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e0300f55-f002-4faa-bf6a-c04bea0a043e-metrics-certs podName:e0300f55-f002-4faa-bf6a-c04bea0a043e nodeName:}" failed. No retries permitted until 2026-11-03 13:07:49.118375488 +0000 UTC m=+4.092698121 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e0300f55-f002-4faa-bf6a-c04bea0a043e-metrics-certs") pod "network-metrics-daemon-f7hmh" (UID: "e0300f55-f002-4faa-bf6a-c04bea0a043e") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 13:07:48.118530 ip-10-0-131-36 kubenswrapper[2573]: E1103 13:07:48.118426 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7e280277-f29f-4a2b-8b85-2491529e5ef2-original-pull-secret podName:7e280277-f29f-4a2b-8b85-2491529e5ef2 nodeName:}" failed. No retries permitted until 2026-11-03 13:07:49.118405704 +0000 UTC m=+4.092728325 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7e280277-f29f-4a2b-8b85-2491529e5ef2-original-pull-secret") pod "global-pull-secret-syncer-5jz7t" (UID: "7e280277-f29f-4a2b-8b85-2491529e5ef2") : object "kube-system"/"original-pull-secret" not registered Nov 03 13:07:48.118530 ip-10-0-131-36 kubenswrapper[2573]: E1103 13:07:48.118467 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 13:07:48.118530 ip-10-0-131-36 kubenswrapper[2573]: E1103 13:07:48.118481 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 13:07:48.118530 ip-10-0-131-36 kubenswrapper[2573]: E1103 13:07:48.118499 2573 projected.go:196] Error preparing data for projected volume kube-api-access-hbfgb for pod openshift-network-diagnostics/network-check-target-8rdgq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 13:07:48.118530 ip-10-0-131-36 kubenswrapper[2573]: E1103 13:07:48.118529 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1bfc1afc-5b9f-4447-8d40-fabedbee9c07-kube-api-access-hbfgb podName:1bfc1afc-5b9f-4447-8d40-fabedbee9c07 nodeName:}" failed. No retries permitted until 2026-11-03 13:07:49.118518541 +0000 UTC m=+4.092841160 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-hbfgb" (UniqueName: "kubernetes.io/projected/1bfc1afc-5b9f-4447-8d40-fabedbee9c07-kube-api-access-hbfgb") pod "network-check-target-8rdgq" (UID: "1bfc1afc-5b9f-4447-8d40-fabedbee9c07") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 13:07:48.444260 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:48.444150 2573 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 13:02:47 +0000 UTC" deadline="2028-04-05 02:26:35.351987869 +0000 UTC" Nov 03 13:07:48.444260 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:48.444214 2573 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12445h18m46.907778104s" Nov 03 13:07:48.624592 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:48.624052 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8rdgq" Nov 03 13:07:48.624592 ip-10-0-131-36 kubenswrapper[2573]: E1103 13:07:48.624172 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8rdgq" podUID="1bfc1afc-5b9f-4447-8d40-fabedbee9c07" Nov 03 13:07:48.624820 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:48.624645 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f7hmh" Nov 03 13:07:48.624820 ip-10-0-131-36 kubenswrapper[2573]: E1103 13:07:48.624749 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f7hmh" podUID="e0300f55-f002-4faa-bf6a-c04bea0a043e" Nov 03 13:07:48.641875 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:48.641838 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-4tn4j" event={"ID":"39e11989-e598-4cea-87d9-26b3cf78a66b","Type":"ContainerStarted","Data":"c29cb1f6582414bd2e24af736205bd2de388faaebcd65096615b22499cfdaf05"} Nov 03 13:07:48.644425 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:48.644375 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fg6vh" event={"ID":"d926d945-39f4-48f3-b3a4-0b302b15b69b","Type":"ContainerStarted","Data":"39aa5c2c75a1203c58dd67810fc2d1718f5535ce2d67634692f6a034d4a8fe41"} Nov 03 13:07:48.648470 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:48.648439 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-dsjsv" event={"ID":"74dc4aee-10b1-463c-a9c3-522b7a6c10b7","Type":"ContainerStarted","Data":"32b60dc42f674361e6ee2e8b8778e572fd8fc785139362be592e97abee98664a"} Nov 03 13:07:48.653238 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:48.653211 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-x2rm4" event={"ID":"35e0fb40-48ce-4223-b7d4-c9290b6861cd","Type":"ContainerStarted","Data":"e6c2867616afcbdf7d2a5611abc303a745fe5f9dece80d4859a108a05d8f4438"} Nov 03 13:07:48.667576 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:48.667543 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c9dvm" event={"ID":"9db7d195-220e-4644-8690-9e5ee37fc066","Type":"ContainerStarted","Data":"2714947ff73646c1e92b308d87f7aa0ae91cd24f08a9ed96681ba260c6f6d555"} Nov 03 13:07:48.670385 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:48.670341 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-g4xgp" event={"ID":"fd7e8b58-de31-406a-8cdb-9ccfee140d57","Type":"ContainerStarted","Data":"080b34934d5649c436eb755c96cb75220add975b57d1a9023728b6782facbc86"} Nov 03 13:07:48.677010 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:48.676979 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-hmbpc" event={"ID":"c704eb02-d156-4695-a55d-7dc96654e4cd","Type":"ContainerStarted","Data":"76516e589de873b36b6de4e04a437e5999788d1915cb8e25b57c8d8ca1a0abf4"} Nov 03 13:07:48.680201 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:48.680174 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-36.us-east-2.compute.internal" event={"ID":"5ee88558fb38141d796a31757de861ab","Type":"ContainerStarted","Data":"9f04a5587062ce2629d840cb1bf24b80a09c12aabf2775705c79a45bb65216ca"} Nov 03 13:07:48.692254 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:48.692191 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-bdg96" event={"ID":"246971bc-ed35-4fc7-b5f8-941327b90a94","Type":"ContainerStarted","Data":"ecf7ff9bffd580730bcdf842700eb81895af4e3d420ec626d39da9261adba3ee"} Nov 03 13:07:48.703449 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:48.703356 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-v8gcq" event={"ID":"5ea5cf61-ed1c-4f5e-b1a0-5c46c5c3ee18","Type":"ContainerStarted","Data":"6983b50836f972f88268b1ac5ff771dc019eeb5a3416d32647e8046e0e92e2d0"} Nov 03 13:07:49.126412 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:49.126376 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e0300f55-f002-4faa-bf6a-c04bea0a043e-metrics-certs\") pod \"network-metrics-daemon-f7hmh\" (UID: \"e0300f55-f002-4faa-bf6a-c04bea0a043e\") " pod="openshift-multus/network-metrics-daemon-f7hmh" Nov 03 13:07:49.126589 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:49.126455 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbfgb\" (UniqueName: \"kubernetes.io/projected/1bfc1afc-5b9f-4447-8d40-fabedbee9c07-kube-api-access-hbfgb\") pod \"network-check-target-8rdgq\" (UID: \"1bfc1afc-5b9f-4447-8d40-fabedbee9c07\") " pod="openshift-network-diagnostics/network-check-target-8rdgq" Nov 03 13:07:49.126589 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:49.126493 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7e280277-f29f-4a2b-8b85-2491529e5ef2-original-pull-secret\") pod \"global-pull-secret-syncer-5jz7t\" (UID: \"7e280277-f29f-4a2b-8b85-2491529e5ef2\") " pod="kube-system/global-pull-secret-syncer-5jz7t" Nov 03 13:07:49.126814 ip-10-0-131-36 kubenswrapper[2573]: E1103 13:07:49.126624 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 13:07:49.126814 ip-10-0-131-36 kubenswrapper[2573]: E1103 13:07:49.126685 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7e280277-f29f-4a2b-8b85-2491529e5ef2-original-pull-secret podName:7e280277-f29f-4a2b-8b85-2491529e5ef2 nodeName:}" failed. No retries permitted until 2026-11-03 13:07:51.126668443 +0000 UTC m=+6.100991085 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7e280277-f29f-4a2b-8b85-2491529e5ef2-original-pull-secret") pod "global-pull-secret-syncer-5jz7t" (UID: "7e280277-f29f-4a2b-8b85-2491529e5ef2") : object "kube-system"/"original-pull-secret" not registered Nov 03 13:07:49.127113 ip-10-0-131-36 kubenswrapper[2573]: E1103 13:07:49.127091 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 13:07:49.127192 ip-10-0-131-36 kubenswrapper[2573]: E1103 13:07:49.127152 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e0300f55-f002-4faa-bf6a-c04bea0a043e-metrics-certs podName:e0300f55-f002-4faa-bf6a-c04bea0a043e nodeName:}" failed. No retries permitted until 2026-11-03 13:07:51.127136974 +0000 UTC m=+6.101459607 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e0300f55-f002-4faa-bf6a-c04bea0a043e-metrics-certs") pod "network-metrics-daemon-f7hmh" (UID: "e0300f55-f002-4faa-bf6a-c04bea0a043e") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 13:07:49.127250 ip-10-0-131-36 kubenswrapper[2573]: E1103 13:07:49.127234 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 13:07:49.127314 ip-10-0-131-36 kubenswrapper[2573]: E1103 13:07:49.127251 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 13:07:49.127314 ip-10-0-131-36 kubenswrapper[2573]: E1103 13:07:49.127264 2573 projected.go:196] Error preparing data for projected volume kube-api-access-hbfgb for pod openshift-network-diagnostics/network-check-target-8rdgq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 13:07:49.127411 ip-10-0-131-36 kubenswrapper[2573]: E1103 13:07:49.127319 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1bfc1afc-5b9f-4447-8d40-fabedbee9c07-kube-api-access-hbfgb podName:1bfc1afc-5b9f-4447-8d40-fabedbee9c07 nodeName:}" failed. No retries permitted until 2026-11-03 13:07:51.127305543 +0000 UTC m=+6.101628178 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-hbfgb" (UniqueName: "kubernetes.io/projected/1bfc1afc-5b9f-4447-8d40-fabedbee9c07-kube-api-access-hbfgb") pod "network-check-target-8rdgq" (UID: "1bfc1afc-5b9f-4447-8d40-fabedbee9c07") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 13:07:49.445136 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:49.444971 2573 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 13:02:47 +0000 UTC" deadline="2028-05-14 20:12:35.665166869 +0000 UTC" Nov 03 13:07:49.445136 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:49.445036 2573 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13399h4m46.220135246s" Nov 03 13:07:49.628665 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:49.628436 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5jz7t" Nov 03 13:07:49.628908 ip-10-0-131-36 kubenswrapper[2573]: E1103 13:07:49.628880 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5jz7t" podUID="7e280277-f29f-4a2b-8b85-2491529e5ef2" Nov 03 13:07:49.731910 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:49.730211 2573 generic.go:356] "Generic (PLEG): container finished" podID="4879f52c5c2e658a43379722bc61381e" containerID="7938de99cfecbcdb80378eeeb5f8d60e22897f5556f9e4b29d72b35e122fc315" exitCode=0 Nov 03 13:07:49.731910 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:49.731553 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-36.us-east-2.compute.internal" event={"ID":"4879f52c5c2e658a43379722bc61381e","Type":"ContainerDied","Data":"7938de99cfecbcdb80378eeeb5f8d60e22897f5556f9e4b29d72b35e122fc315"} Nov 03 13:07:49.746028 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:49.745974 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-36.us-east-2.compute.internal" podStartSLOduration=3.745957636 podStartE2EDuration="3.745957636s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 13:07:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:07:48.69521609 +0000 UTC m=+3.669538731" watchObservedRunningTime="2026-11-03 13:07:49.745957636 +0000 UTC m=+4.720280268" Nov 03 13:07:50.625501 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:50.624732 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f7hmh" Nov 03 13:07:50.625501 ip-10-0-131-36 kubenswrapper[2573]: E1103 13:07:50.624875 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f7hmh" podUID="e0300f55-f002-4faa-bf6a-c04bea0a043e" Nov 03 13:07:50.625501 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:50.625266 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8rdgq" Nov 03 13:07:50.625501 ip-10-0-131-36 kubenswrapper[2573]: E1103 13:07:50.625379 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8rdgq" podUID="1bfc1afc-5b9f-4447-8d40-fabedbee9c07" Nov 03 13:07:50.750892 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:50.750847 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-36.us-east-2.compute.internal" event={"ID":"4879f52c5c2e658a43379722bc61381e","Type":"ContainerStarted","Data":"6ea16991d8dbeee069c615b16eca7e6ec8aebeaf09d0e7c1bec17e37fdcbfc9b"} Nov 03 13:07:50.766054 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:50.765998 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-36.us-east-2.compute.internal" podStartSLOduration=4.7659779239999995 podStartE2EDuration="4.765977924s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 13:07:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:07:50.765870311 +0000 UTC m=+5.740192953" watchObservedRunningTime="2026-11-03 13:07:50.765977924 +0000 UTC m=+5.740300554" Nov 03 13:07:51.146094 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:51.146054 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e0300f55-f002-4faa-bf6a-c04bea0a043e-metrics-certs\") pod \"network-metrics-daemon-f7hmh\" (UID: \"e0300f55-f002-4faa-bf6a-c04bea0a043e\") " pod="openshift-multus/network-metrics-daemon-f7hmh" Nov 03 13:07:51.146258 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:51.146140 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbfgb\" (UniqueName: \"kubernetes.io/projected/1bfc1afc-5b9f-4447-8d40-fabedbee9c07-kube-api-access-hbfgb\") pod \"network-check-target-8rdgq\" (UID: \"1bfc1afc-5b9f-4447-8d40-fabedbee9c07\") " pod="openshift-network-diagnostics/network-check-target-8rdgq" Nov 03 13:07:51.146349 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:51.146269 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7e280277-f29f-4a2b-8b85-2491529e5ef2-original-pull-secret\") pod \"global-pull-secret-syncer-5jz7t\" (UID: \"7e280277-f29f-4a2b-8b85-2491529e5ef2\") " pod="kube-system/global-pull-secret-syncer-5jz7t" Nov 03 13:07:51.146586 ip-10-0-131-36 kubenswrapper[2573]: E1103 13:07:51.146563 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 13:07:51.146689 ip-10-0-131-36 kubenswrapper[2573]: E1103 13:07:51.146635 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e0300f55-f002-4faa-bf6a-c04bea0a043e-metrics-certs podName:e0300f55-f002-4faa-bf6a-c04bea0a043e nodeName:}" failed. No retries permitted until 2026-11-03 13:07:55.14661611 +0000 UTC m=+10.120938733 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e0300f55-f002-4faa-bf6a-c04bea0a043e-metrics-certs") pod "network-metrics-daemon-f7hmh" (UID: "e0300f55-f002-4faa-bf6a-c04bea0a043e") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 13:07:51.147182 ip-10-0-131-36 kubenswrapper[2573]: E1103 13:07:51.147162 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 13:07:51.147275 ip-10-0-131-36 kubenswrapper[2573]: E1103 13:07:51.147189 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 13:07:51.147275 ip-10-0-131-36 kubenswrapper[2573]: E1103 13:07:51.147202 2573 projected.go:196] Error preparing data for projected volume kube-api-access-hbfgb for pod openshift-network-diagnostics/network-check-target-8rdgq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 13:07:51.147275 ip-10-0-131-36 kubenswrapper[2573]: E1103 13:07:51.147249 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1bfc1afc-5b9f-4447-8d40-fabedbee9c07-kube-api-access-hbfgb podName:1bfc1afc-5b9f-4447-8d40-fabedbee9c07 nodeName:}" failed. No retries permitted until 2026-11-03 13:07:55.147234222 +0000 UTC m=+10.121556862 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-hbfgb" (UniqueName: "kubernetes.io/projected/1bfc1afc-5b9f-4447-8d40-fabedbee9c07-kube-api-access-hbfgb") pod "network-check-target-8rdgq" (UID: "1bfc1afc-5b9f-4447-8d40-fabedbee9c07") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 13:07:51.147449 ip-10-0-131-36 kubenswrapper[2573]: E1103 13:07:51.147327 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 13:07:51.147449 ip-10-0-131-36 kubenswrapper[2573]: E1103 13:07:51.147369 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7e280277-f29f-4a2b-8b85-2491529e5ef2-original-pull-secret podName:7e280277-f29f-4a2b-8b85-2491529e5ef2 nodeName:}" failed. No retries permitted until 2026-11-03 13:07:55.147357231 +0000 UTC m=+10.121679876 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7e280277-f29f-4a2b-8b85-2491529e5ef2-original-pull-secret") pod "global-pull-secret-syncer-5jz7t" (UID: "7e280277-f29f-4a2b-8b85-2491529e5ef2") : object "kube-system"/"original-pull-secret" not registered Nov 03 13:07:51.627664 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:51.627130 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5jz7t" Nov 03 13:07:51.627664 ip-10-0-131-36 kubenswrapper[2573]: E1103 13:07:51.627268 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5jz7t" podUID="7e280277-f29f-4a2b-8b85-2491529e5ef2" Nov 03 13:07:51.760190 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:51.759007 2573 generic.go:356] "Generic (PLEG): container finished" podID="35e0fb40-48ce-4223-b7d4-c9290b6861cd" containerID="37ce76298956d213d38dc00b494aca90f17e26a203fe01350b91451ec3da9483" exitCode=0 Nov 03 13:07:51.760190 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:51.760009 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-x2rm4" event={"ID":"35e0fb40-48ce-4223-b7d4-c9290b6861cd","Type":"ContainerDied","Data":"37ce76298956d213d38dc00b494aca90f17e26a203fe01350b91451ec3da9483"} Nov 03 13:07:52.624406 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:52.624374 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f7hmh" Nov 03 13:07:52.624572 ip-10-0-131-36 kubenswrapper[2573]: E1103 13:07:52.624542 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f7hmh" podUID="e0300f55-f002-4faa-bf6a-c04bea0a043e" Nov 03 13:07:52.624966 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:52.624943 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8rdgq" Nov 03 13:07:52.625088 ip-10-0-131-36 kubenswrapper[2573]: E1103 13:07:52.625031 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8rdgq" podUID="1bfc1afc-5b9f-4447-8d40-fabedbee9c07" Nov 03 13:07:52.772891 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:52.772063 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-bdg96" event={"ID":"246971bc-ed35-4fc7-b5f8-941327b90a94","Type":"ContainerStarted","Data":"e661029aa78bad8cb0e59bbe4261488b9248d9400458411ec294d0c320d7de4c"} Nov 03 13:07:52.788127 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:52.787941 2573 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-bdg96" Nov 03 13:07:52.788550 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:52.788512 2573 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-bdg96" Nov 03 13:07:52.808253 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:52.808153 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-bdg96" podStartSLOduration=3.031911163 podStartE2EDuration="6.808133375s" totalImagesPullingTime="3.776222212s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:07:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:07:52.794858449 +0000 UTC m=+7.769181095" watchObservedRunningTime="2026-11-03 13:07:52.808133375 +0000 UTC m=+7.782456015" Nov 03 13:07:53.626044 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:53.626010 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5jz7t" Nov 03 13:07:53.626195 ip-10-0-131-36 kubenswrapper[2573]: E1103 13:07:53.626150 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5jz7t" podUID="7e280277-f29f-4a2b-8b85-2491529e5ef2" Nov 03 13:07:53.778266 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:53.778191 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-4tn4j" event={"ID":"39e11989-e598-4cea-87d9-26b3cf78a66b","Type":"ContainerStarted","Data":"28f4b003d33e3cacaea8cff9a92cf2c4c82d876054b201787b439b36c93b7eb6"} Nov 03 13:07:53.780925 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:53.780887 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c9dvm" event={"ID":"9db7d195-220e-4644-8690-9e5ee37fc066","Type":"ContainerStarted","Data":"e03fb19793d9259f36afbcf766126b0be7108a000b422a8f3e9bd9692f581a96"} Nov 03 13:07:54.624477 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:54.624417 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f7hmh" Nov 03 13:07:54.624477 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:54.624443 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8rdgq" Nov 03 13:07:54.624714 ip-10-0-131-36 kubenswrapper[2573]: E1103 13:07:54.624561 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f7hmh" podUID="e0300f55-f002-4faa-bf6a-c04bea0a043e" Nov 03 13:07:54.624971 ip-10-0-131-36 kubenswrapper[2573]: E1103 13:07:54.624926 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8rdgq" podUID="1bfc1afc-5b9f-4447-8d40-fabedbee9c07" Nov 03 13:07:54.783392 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:54.782966 2573 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 03 13:07:55.182725 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:55.182689 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e0300f55-f002-4faa-bf6a-c04bea0a043e-metrics-certs\") pod \"network-metrics-daemon-f7hmh\" (UID: \"e0300f55-f002-4faa-bf6a-c04bea0a043e\") " pod="openshift-multus/network-metrics-daemon-f7hmh" Nov 03 13:07:55.182855 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:55.182753 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbfgb\" (UniqueName: \"kubernetes.io/projected/1bfc1afc-5b9f-4447-8d40-fabedbee9c07-kube-api-access-hbfgb\") pod \"network-check-target-8rdgq\" (UID: \"1bfc1afc-5b9f-4447-8d40-fabedbee9c07\") " pod="openshift-network-diagnostics/network-check-target-8rdgq" Nov 03 13:07:55.182855 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:55.182776 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7e280277-f29f-4a2b-8b85-2491529e5ef2-original-pull-secret\") pod \"global-pull-secret-syncer-5jz7t\" (UID: \"7e280277-f29f-4a2b-8b85-2491529e5ef2\") " pod="kube-system/global-pull-secret-syncer-5jz7t" Nov 03 13:07:55.182969 ip-10-0-131-36 kubenswrapper[2573]: E1103 13:07:55.182874 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 13:07:55.182969 ip-10-0-131-36 kubenswrapper[2573]: E1103 13:07:55.182931 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7e280277-f29f-4a2b-8b85-2491529e5ef2-original-pull-secret podName:7e280277-f29f-4a2b-8b85-2491529e5ef2 nodeName:}" failed. No retries permitted until 2026-11-03 13:08:03.182914132 +0000 UTC m=+18.157236765 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7e280277-f29f-4a2b-8b85-2491529e5ef2-original-pull-secret") pod "global-pull-secret-syncer-5jz7t" (UID: "7e280277-f29f-4a2b-8b85-2491529e5ef2") : object "kube-system"/"original-pull-secret" not registered Nov 03 13:07:55.183373 ip-10-0-131-36 kubenswrapper[2573]: E1103 13:07:55.183351 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 13:07:55.183455 ip-10-0-131-36 kubenswrapper[2573]: E1103 13:07:55.183411 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e0300f55-f002-4faa-bf6a-c04bea0a043e-metrics-certs podName:e0300f55-f002-4faa-bf6a-c04bea0a043e nodeName:}" failed. No retries permitted until 2026-11-03 13:08:03.183396427 +0000 UTC m=+18.157719063 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e0300f55-f002-4faa-bf6a-c04bea0a043e-metrics-certs") pod "network-metrics-daemon-f7hmh" (UID: "e0300f55-f002-4faa-bf6a-c04bea0a043e") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 13:07:55.183523 ip-10-0-131-36 kubenswrapper[2573]: E1103 13:07:55.183493 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 13:07:55.183523 ip-10-0-131-36 kubenswrapper[2573]: E1103 13:07:55.183515 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 13:07:55.183616 ip-10-0-131-36 kubenswrapper[2573]: E1103 13:07:55.183527 2573 projected.go:196] Error preparing data for projected volume kube-api-access-hbfgb for pod openshift-network-diagnostics/network-check-target-8rdgq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 13:07:55.183616 ip-10-0-131-36 kubenswrapper[2573]: E1103 13:07:55.183565 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1bfc1afc-5b9f-4447-8d40-fabedbee9c07-kube-api-access-hbfgb podName:1bfc1afc-5b9f-4447-8d40-fabedbee9c07 nodeName:}" failed. No retries permitted until 2026-11-03 13:08:03.183552759 +0000 UTC m=+18.157875402 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-hbfgb" (UniqueName: "kubernetes.io/projected/1bfc1afc-5b9f-4447-8d40-fabedbee9c07-kube-api-access-hbfgb") pod "network-check-target-8rdgq" (UID: "1bfc1afc-5b9f-4447-8d40-fabedbee9c07") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 13:07:55.347857 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:55.347706 2573 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 03 13:07:55.625236 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:55.625203 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5jz7t" Nov 03 13:07:55.625429 ip-10-0-131-36 kubenswrapper[2573]: E1103 13:07:55.625327 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5jz7t" podUID="7e280277-f29f-4a2b-8b85-2491529e5ef2" Nov 03 13:07:55.638297 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:55.638138 2573 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-03T13:07:55.347725106Z","UUID":"c70d3e5b-11ff-4f06-b9c4-c0fcc75aa105","Handler":null,"Name":"","Endpoint":""} Nov 03 13:07:55.642160 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:55.642132 2573 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 03 13:07:55.642324 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:55.642189 2573 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 03 13:07:55.788854 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:55.788817 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-dsjsv" event={"ID":"74dc4aee-10b1-463c-a9c3-522b7a6c10b7","Type":"ContainerStarted","Data":"e9bbd2efd4baf55606563e3922d88f87b42d7aabeb2528026dd23de9bfd03be0"} Nov 03 13:07:55.792886 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:55.792819 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c9dvm" event={"ID":"9db7d195-220e-4644-8690-9e5ee37fc066","Type":"ContainerStarted","Data":"0a335bfdbf1430500276444e08416764582161d1e57adbb420366e483f19ca7c"} Nov 03 13:07:55.804715 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:55.804660 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-4tn4j" podStartSLOduration=5.765326854 podStartE2EDuration="10.804642283s" totalImagesPullingTime="5.039315429s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 13:07:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:07:53.7979518 +0000 UTC m=+8.772274443" watchObservedRunningTime="2026-11-03 13:07:55.804642283 +0000 UTC m=+10.778964925" Nov 03 13:07:55.805083 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:55.805041 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-dsjsv" podStartSLOduration=3.125521929 podStartE2EDuration="10.805033013s" totalImagesPullingTime="7.679511084s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 13:07:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:07:55.803881251 +0000 UTC m=+10.778203897" watchObservedRunningTime="2026-11-03 13:07:55.805033013 +0000 UTC m=+10.779355656" Nov 03 13:07:56.624892 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:56.624381 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f7hmh" Nov 03 13:07:56.624892 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:56.624399 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8rdgq" Nov 03 13:07:56.624892 ip-10-0-131-36 kubenswrapper[2573]: E1103 13:07:56.624527 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f7hmh" podUID="e0300f55-f002-4faa-bf6a-c04bea0a043e" Nov 03 13:07:56.624892 ip-10-0-131-36 kubenswrapper[2573]: E1103 13:07:56.624632 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8rdgq" podUID="1bfc1afc-5b9f-4447-8d40-fabedbee9c07" Nov 03 13:07:56.798220 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:56.798179 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-v8gcq" event={"ID":"5ea5cf61-ed1c-4f5e-b1a0-5c46c5c3ee18","Type":"ContainerStarted","Data":"fed49c8aaccb2d78659509aa13ccad1aa0c9aa0542de470de9e39e3540b43fa8"} Nov 03 13:07:56.800577 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:56.800534 2573 generic.go:356] "Generic (PLEG): container finished" podID="35e0fb40-48ce-4223-b7d4-c9290b6861cd" containerID="e0eb309cf5a77d30354f2d94a35fe246555eda5aa469a2482840a0fc5697a982" exitCode=0 Nov 03 13:07:56.801136 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:56.801112 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-x2rm4" event={"ID":"35e0fb40-48ce-4223-b7d4-c9290b6861cd","Type":"ContainerDied","Data":"e0eb309cf5a77d30354f2d94a35fe246555eda5aa469a2482840a0fc5697a982"} Nov 03 13:07:56.804057 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:56.804033 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c9dvm" event={"ID":"9db7d195-220e-4644-8690-9e5ee37fc066","Type":"ContainerStarted","Data":"d23bc86ba920436128afadd6c04054ec40c8602688036a1b981e07a5421a3f64"} Nov 03 13:07:56.813479 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:56.813440 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-v8gcq" podStartSLOduration=2.921594297 podStartE2EDuration="10.813426904s" totalImagesPullingTime="7.891832607s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 13:07:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:07:56.812803268 +0000 UTC m=+11.787125910" watchObservedRunningTime="2026-11-03 13:07:56.813426904 +0000 UTC m=+11.787749544" Nov 03 13:07:56.846205 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:56.846153 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c9dvm" podStartSLOduration=3.655197487 podStartE2EDuration="11.846136454s" totalImagesPullingTime="8.190938967s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 13:07:45 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:07:56.845689879 +0000 UTC m=+11.820012638" watchObservedRunningTime="2026-11-03 13:07:56.846136454 +0000 UTC m=+11.820459096" Nov 03 13:07:57.537294 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:57.537102 2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-bdg96" Nov 03 13:07:57.537451 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:57.537424 2573 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 03 13:07:57.537838 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:57.537818 2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-bdg96" Nov 03 13:07:57.624509 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:57.624478 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5jz7t" Nov 03 13:07:57.624659 ip-10-0-131-36 kubenswrapper[2573]: E1103 13:07:57.624615 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5jz7t" podUID="7e280277-f29f-4a2b-8b85-2491529e5ef2" Nov 03 13:07:57.810793 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:57.810754 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-g4xgp" event={"ID":"fd7e8b58-de31-406a-8cdb-9ccfee140d57","Type":"ContainerStarted","Data":"3dafaa292ffd61fd5aeea10c2bca563dac38eae87caf42810bd89e0fef3fa6d9"} Nov 03 13:07:57.824780 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:57.824717 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-g4xgp" podStartSLOduration=5.259939312 podStartE2EDuration="12.824697102s" totalImagesPullingTime="7.56475779s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 13:07:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:07:57.82469763 +0000 UTC m=+12.799020273" watchObservedRunningTime="2026-11-03 13:07:57.824697102 +0000 UTC m=+12.799019744" Nov 03 13:07:58.623785 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:58.623746 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8rdgq" Nov 03 13:07:58.624004 ip-10-0-131-36 kubenswrapper[2573]: E1103 13:07:58.623879 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8rdgq" podUID="1bfc1afc-5b9f-4447-8d40-fabedbee9c07" Nov 03 13:07:58.624004 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:58.623938 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f7hmh" Nov 03 13:07:58.624119 ip-10-0-131-36 kubenswrapper[2573]: E1103 13:07:58.624053 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f7hmh" podUID="e0300f55-f002-4faa-bf6a-c04bea0a043e" Nov 03 13:07:58.814303 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:58.814250 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-hmbpc" event={"ID":"c704eb02-d156-4695-a55d-7dc96654e4cd","Type":"ContainerStarted","Data":"713466bb2b1b73f7631eb4139941a51863e3b18268b45ba9e9edfc52fc1afb70"} Nov 03 13:07:58.816977 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:58.816947 2573 generic.go:356] "Generic (PLEG): container finished" podID="35e0fb40-48ce-4223-b7d4-c9290b6861cd" containerID="30ae7c80b1521afa176588c5f54a629ac760877a8bc34593bf1f0c3725b23fb7" exitCode=0 Nov 03 13:07:58.817101 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:58.817033 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-x2rm4" event={"ID":"35e0fb40-48ce-4223-b7d4-c9290b6861cd","Type":"ContainerDied","Data":"30ae7c80b1521afa176588c5f54a629ac760877a8bc34593bf1f0c3725b23fb7"} Nov 03 13:07:58.831797 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:58.831744 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-hmbpc" podStartSLOduration=3.759269321 podStartE2EDuration="13.831730742s" totalImagesPullingTime="10.072461421s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 13:07:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:07:58.831444041 +0000 UTC m=+13.805766681" watchObservedRunningTime="2026-11-03 13:07:58.831730742 +0000 UTC m=+13.806053383" Nov 03 13:07:59.624006 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:07:59.623968 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5jz7t" Nov 03 13:07:59.624167 ip-10-0-131-36 kubenswrapper[2573]: E1103 13:07:59.624092 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5jz7t" podUID="7e280277-f29f-4a2b-8b85-2491529e5ef2" Nov 03 13:08:00.624367 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:00.624334 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f7hmh" Nov 03 13:08:00.624367 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:00.624376 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8rdgq" Nov 03 13:08:00.624982 ip-10-0-131-36 kubenswrapper[2573]: E1103 13:08:00.624536 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8rdgq" podUID="1bfc1afc-5b9f-4447-8d40-fabedbee9c07" Nov 03 13:08:00.624982 ip-10-0-131-36 kubenswrapper[2573]: E1103 13:08:00.624475 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f7hmh" podUID="e0300f55-f002-4faa-bf6a-c04bea0a043e" Nov 03 13:08:00.823914 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:00.823874 2573 generic.go:356] "Generic (PLEG): container finished" podID="35e0fb40-48ce-4223-b7d4-c9290b6861cd" containerID="a5a99121c65a6e1a531c5296b16ae624bf4702c16c21d96601a9bab3a7cf17f7" exitCode=0 Nov 03 13:08:00.824067 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:00.823925 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-x2rm4" event={"ID":"35e0fb40-48ce-4223-b7d4-c9290b6861cd","Type":"ContainerDied","Data":"a5a99121c65a6e1a531c5296b16ae624bf4702c16c21d96601a9bab3a7cf17f7"} Nov 03 13:08:01.624691 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:01.624493 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5jz7t" Nov 03 13:08:01.625083 ip-10-0-131-36 kubenswrapper[2573]: E1103 13:08:01.624786 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5jz7t" podUID="7e280277-f29f-4a2b-8b85-2491529e5ef2" Nov 03 13:08:02.624726 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:02.624685 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8rdgq" Nov 03 13:08:02.625096 ip-10-0-131-36 kubenswrapper[2573]: E1103 13:08:02.624796 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8rdgq" podUID="1bfc1afc-5b9f-4447-8d40-fabedbee9c07" Nov 03 13:08:02.625096 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:02.624848 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f7hmh" Nov 03 13:08:02.625096 ip-10-0-131-36 kubenswrapper[2573]: E1103 13:08:02.624940 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f7hmh" podUID="e0300f55-f002-4faa-bf6a-c04bea0a043e" Nov 03 13:08:02.832138 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:02.832101 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fg6vh" event={"ID":"d926d945-39f4-48f3-b3a4-0b302b15b69b","Type":"ContainerStarted","Data":"09b7678656dedd0a10fa0ebfd812712d6343e3e06e9fb4cc55603db8ac572a94"} Nov 03 13:08:02.832325 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:02.832219 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fg6vh" event={"ID":"d926d945-39f4-48f3-b3a4-0b302b15b69b","Type":"ContainerStarted","Data":"8947f8051ebae788536ac6f82f2df83dcebbf57eeb15d93ab1b421e5d05d0fcd"} Nov 03 13:08:02.832325 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:02.832252 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fg6vh" event={"ID":"d926d945-39f4-48f3-b3a4-0b302b15b69b","Type":"ContainerStarted","Data":"3e11beb31b883401387f0cd951beab90d5727511bf646aad85d41d56523412e0"} Nov 03 13:08:02.832325 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:02.832311 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fg6vh" event={"ID":"d926d945-39f4-48f3-b3a4-0b302b15b69b","Type":"ContainerStarted","Data":"bc7581f95b1702ed5c673391eabab842433c696542da2c271d50cbd5e51759a1"} Nov 03 13:08:02.832486 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:02.832358 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fg6vh" event={"ID":"d926d945-39f4-48f3-b3a4-0b302b15b69b","Type":"ContainerStarted","Data":"fb67d035d4009c6fb379f952b3ad158e3807cc426d924995f935623fe66345c0"} Nov 03 13:08:02.832486 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:02.832405 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fg6vh" event={"ID":"d926d945-39f4-48f3-b3a4-0b302b15b69b","Type":"ContainerStarted","Data":"ec5a93dd4478c59cecb798fa501b40974962fcc652d7ccf802fb5c43be9717fa"} Nov 03 13:08:02.836373 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:02.836342 2573 generic.go:356] "Generic (PLEG): container finished" podID="35e0fb40-48ce-4223-b7d4-c9290b6861cd" containerID="241d8a6be6bd0a09ab0d84fb2151568b70aa0d9799356f04092d09d4c6cc5295" exitCode=0 Nov 03 13:08:02.836490 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:02.836401 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-x2rm4" event={"ID":"35e0fb40-48ce-4223-b7d4-c9290b6861cd","Type":"ContainerDied","Data":"241d8a6be6bd0a09ab0d84fb2151568b70aa0d9799356f04092d09d4c6cc5295"} Nov 03 13:08:03.239385 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:03.239262 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbfgb\" (UniqueName: \"kubernetes.io/projected/1bfc1afc-5b9f-4447-8d40-fabedbee9c07-kube-api-access-hbfgb\") pod \"network-check-target-8rdgq\" (UID: \"1bfc1afc-5b9f-4447-8d40-fabedbee9c07\") " pod="openshift-network-diagnostics/network-check-target-8rdgq" Nov 03 13:08:03.239385 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:03.239328 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7e280277-f29f-4a2b-8b85-2491529e5ef2-original-pull-secret\") pod \"global-pull-secret-syncer-5jz7t\" (UID: \"7e280277-f29f-4a2b-8b85-2491529e5ef2\") " pod="kube-system/global-pull-secret-syncer-5jz7t" Nov 03 13:08:03.239385 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:03.239367 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e0300f55-f002-4faa-bf6a-c04bea0a043e-metrics-certs\") pod \"network-metrics-daemon-f7hmh\" (UID: \"e0300f55-f002-4faa-bf6a-c04bea0a043e\") " pod="openshift-multus/network-metrics-daemon-f7hmh" Nov 03 13:08:03.239598 ip-10-0-131-36 kubenswrapper[2573]: E1103 13:08:03.239413 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 13:08:03.239598 ip-10-0-131-36 kubenswrapper[2573]: E1103 13:08:03.239426 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 13:08:03.239598 ip-10-0-131-36 kubenswrapper[2573]: E1103 13:08:03.239452 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 13:08:03.239598 ip-10-0-131-36 kubenswrapper[2573]: E1103 13:08:03.239456 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 13:08:03.239598 ip-10-0-131-36 kubenswrapper[2573]: E1103 13:08:03.239465 2573 projected.go:196] Error preparing data for projected volume kube-api-access-hbfgb for pod openshift-network-diagnostics/network-check-target-8rdgq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 13:08:03.239598 ip-10-0-131-36 kubenswrapper[2573]: E1103 13:08:03.239478 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7e280277-f29f-4a2b-8b85-2491529e5ef2-original-pull-secret podName:7e280277-f29f-4a2b-8b85-2491529e5ef2 nodeName:}" failed. No retries permitted until 2026-11-03 13:08:19.239462557 +0000 UTC m=+34.213785194 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7e280277-f29f-4a2b-8b85-2491529e5ef2-original-pull-secret") pod "global-pull-secret-syncer-5jz7t" (UID: "7e280277-f29f-4a2b-8b85-2491529e5ef2") : object "kube-system"/"original-pull-secret" not registered Nov 03 13:08:03.239598 ip-10-0-131-36 kubenswrapper[2573]: E1103 13:08:03.239506 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1bfc1afc-5b9f-4447-8d40-fabedbee9c07-kube-api-access-hbfgb podName:1bfc1afc-5b9f-4447-8d40-fabedbee9c07 nodeName:}" failed. No retries permitted until 2026-11-03 13:08:19.239491665 +0000 UTC m=+34.213814285 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-hbfgb" (UniqueName: "kubernetes.io/projected/1bfc1afc-5b9f-4447-8d40-fabedbee9c07-kube-api-access-hbfgb") pod "network-check-target-8rdgq" (UID: "1bfc1afc-5b9f-4447-8d40-fabedbee9c07") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 13:08:03.239598 ip-10-0-131-36 kubenswrapper[2573]: E1103 13:08:03.239522 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e0300f55-f002-4faa-bf6a-c04bea0a043e-metrics-certs podName:e0300f55-f002-4faa-bf6a-c04bea0a043e nodeName:}" failed. No retries permitted until 2026-11-03 13:08:19.239515883 +0000 UTC m=+34.213838502 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e0300f55-f002-4faa-bf6a-c04bea0a043e-metrics-certs") pod "network-metrics-daemon-f7hmh" (UID: "e0300f55-f002-4faa-bf6a-c04bea0a043e") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 13:08:03.624740 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:03.624667 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5jz7t" Nov 03 13:08:03.624864 ip-10-0-131-36 kubenswrapper[2573]: E1103 13:08:03.624767 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5jz7t" podUID="7e280277-f29f-4a2b-8b85-2491529e5ef2" Nov 03 13:08:03.841336 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:03.841296 2573 generic.go:356] "Generic (PLEG): container finished" podID="35e0fb40-48ce-4223-b7d4-c9290b6861cd" containerID="4762aad0f9569e283a300d8804cc2778fe0166f2b826c0cae82766b490647026" exitCode=0 Nov 03 13:08:03.841488 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:03.841349 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-x2rm4" event={"ID":"35e0fb40-48ce-4223-b7d4-c9290b6861cd","Type":"ContainerDied","Data":"4762aad0f9569e283a300d8804cc2778fe0166f2b826c0cae82766b490647026"} Nov 03 13:08:04.624515 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:04.624475 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f7hmh" Nov 03 13:08:04.624515 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:04.624506 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8rdgq" Nov 03 13:08:04.624693 ip-10-0-131-36 kubenswrapper[2573]: E1103 13:08:04.624587 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f7hmh" podUID="e0300f55-f002-4faa-bf6a-c04bea0a043e" Nov 03 13:08:04.624748 ip-10-0-131-36 kubenswrapper[2573]: E1103 13:08:04.624721 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8rdgq" podUID="1bfc1afc-5b9f-4447-8d40-fabedbee9c07" Nov 03 13:08:04.847300 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:04.847254 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-x2rm4" event={"ID":"35e0fb40-48ce-4223-b7d4-c9290b6861cd","Type":"ContainerStarted","Data":"1a4bc26d7bd6ae3404f9c2d0e67ae74c11542dc35b3eab4229073c4b3efb764f"} Nov 03 13:08:04.868807 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:04.868764 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-x2rm4" podStartSLOduration=9.166069093 podStartE2EDuration="19.868750435s" totalImagesPullingTime="10.702681342s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 13:07:45 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:08:04.868699269 +0000 UTC m=+19.843021902" watchObservedRunningTime="2026-11-03 13:08:04.868750435 +0000 UTC m=+19.843073077" Nov 03 13:08:05.625369 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:05.625325 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5jz7t" Nov 03 13:08:05.625544 ip-10-0-131-36 kubenswrapper[2573]: E1103 13:08:05.625415 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5jz7t" podUID="7e280277-f29f-4a2b-8b85-2491529e5ef2" Nov 03 13:08:05.852498 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:05.852456 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fg6vh" event={"ID":"d926d945-39f4-48f3-b3a4-0b302b15b69b","Type":"ContainerStarted","Data":"2abe4eeeaadcc083764be9fb884285647e27c741f4f9ee398c2efd2efccd63e4"} Nov 03 13:08:06.624788 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:06.624735 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8rdgq" Nov 03 13:08:06.624946 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:06.624735 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f7hmh" Nov 03 13:08:06.624946 ip-10-0-131-36 kubenswrapper[2573]: E1103 13:08:06.624865 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8rdgq" podUID="1bfc1afc-5b9f-4447-8d40-fabedbee9c07" Nov 03 13:08:06.625033 ip-10-0-131-36 kubenswrapper[2573]: E1103 13:08:06.624982 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f7hmh" podUID="e0300f55-f002-4faa-bf6a-c04bea0a043e" Nov 03 13:08:07.624436 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:07.624398 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5jz7t" Nov 03 13:08:07.625062 ip-10-0-131-36 kubenswrapper[2573]: E1103 13:08:07.624508 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5jz7t" podUID="7e280277-f29f-4a2b-8b85-2491529e5ef2" Nov 03 13:08:07.859577 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:07.859542 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fg6vh" event={"ID":"d926d945-39f4-48f3-b3a4-0b302b15b69b","Type":"ContainerStarted","Data":"c62ce3a3362edb7fd7ea849cbaff5df513a090218ccb4e0d8d8974aab308fe44"} Nov 03 13:08:07.859860 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:07.859822 2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-fg6vh" Nov 03 13:08:07.859983 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:07.859896 2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-fg6vh" Nov 03 13:08:07.875737 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:07.875525 2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-fg6vh" Nov 03 13:08:07.883481 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:07.883434 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-fg6vh" podStartSLOduration=7.2541492 podStartE2EDuration="21.883420684s" totalImagesPullingTime="14.629271484s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 13:07:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:08:07.882873564 +0000 UTC m=+22.857196203" watchObservedRunningTime="2026-11-03 13:08:07.883420684 +0000 UTC m=+22.857743363" Nov 03 13:08:08.624131 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:08.624094 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f7hmh" Nov 03 13:08:08.624131 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:08.624123 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8rdgq" Nov 03 13:08:08.624357 ip-10-0-131-36 kubenswrapper[2573]: E1103 13:08:08.624244 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f7hmh" podUID="e0300f55-f002-4faa-bf6a-c04bea0a043e" Nov 03 13:08:08.624424 ip-10-0-131-36 kubenswrapper[2573]: E1103 13:08:08.624399 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8rdgq" podUID="1bfc1afc-5b9f-4447-8d40-fabedbee9c07" Nov 03 13:08:08.677590 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:08.677561 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-5jz7t"] Nov 03 13:08:08.677921 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:08.677662 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5jz7t" Nov 03 13:08:08.677921 ip-10-0-131-36 kubenswrapper[2573]: E1103 13:08:08.677754 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5jz7t" podUID="7e280277-f29f-4a2b-8b85-2491529e5ef2" Nov 03 13:08:08.680765 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:08.680738 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-8rdgq"] Nov 03 13:08:08.681449 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:08.681429 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-f7hmh"] Nov 03 13:08:08.861229 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:08.861199 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f7hmh" Nov 03 13:08:08.861229 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:08.861216 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8rdgq" Nov 03 13:08:08.861454 ip-10-0-131-36 kubenswrapper[2573]: E1103 13:08:08.861325 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f7hmh" podUID="e0300f55-f002-4faa-bf6a-c04bea0a043e" Nov 03 13:08:08.861540 ip-10-0-131-36 kubenswrapper[2573]: E1103 13:08:08.861464 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8rdgq" podUID="1bfc1afc-5b9f-4447-8d40-fabedbee9c07" Nov 03 13:08:08.861850 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:08.861836 2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-fg6vh" Nov 03 13:08:08.876860 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:08.876776 2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-fg6vh" Nov 03 13:08:10.624161 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:10.624126 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5jz7t" Nov 03 13:08:10.624737 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:10.624167 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8rdgq" Nov 03 13:08:10.624737 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:10.624222 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f7hmh" Nov 03 13:08:10.624737 ip-10-0-131-36 kubenswrapper[2573]: E1103 13:08:10.624241 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5jz7t" podUID="7e280277-f29f-4a2b-8b85-2491529e5ef2" Nov 03 13:08:10.624737 ip-10-0-131-36 kubenswrapper[2573]: E1103 13:08:10.624313 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f7hmh" podUID="e0300f55-f002-4faa-bf6a-c04bea0a043e" Nov 03 13:08:10.624737 ip-10-0-131-36 kubenswrapper[2573]: E1103 13:08:10.624359 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8rdgq" podUID="1bfc1afc-5b9f-4447-8d40-fabedbee9c07" Nov 03 13:08:12.623990 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:12.623945 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8rdgq" Nov 03 13:08:12.624622 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:12.623945 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f7hmh" Nov 03 13:08:12.624622 ip-10-0-131-36 kubenswrapper[2573]: E1103 13:08:12.624060 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8rdgq" podUID="1bfc1afc-5b9f-4447-8d40-fabedbee9c07" Nov 03 13:08:12.624622 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:12.623955 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5jz7t" Nov 03 13:08:12.624622 ip-10-0-131-36 kubenswrapper[2573]: E1103 13:08:12.624145 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f7hmh" podUID="e0300f55-f002-4faa-bf6a-c04bea0a043e" Nov 03 13:08:12.624622 ip-10-0-131-36 kubenswrapper[2573]: E1103 13:08:12.624219 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5jz7t" podUID="7e280277-f29f-4a2b-8b85-2491529e5ef2" Nov 03 13:08:14.624316 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:14.624244 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5jz7t" Nov 03 13:08:14.624786 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:14.624265 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f7hmh" Nov 03 13:08:14.624786 ip-10-0-131-36 kubenswrapper[2573]: E1103 13:08:14.624381 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5jz7t" podUID="7e280277-f29f-4a2b-8b85-2491529e5ef2" Nov 03 13:08:14.624786 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:14.624393 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8rdgq" Nov 03 13:08:14.624786 ip-10-0-131-36 kubenswrapper[2573]: E1103 13:08:14.624463 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f7hmh" podUID="e0300f55-f002-4faa-bf6a-c04bea0a043e" Nov 03 13:08:14.624786 ip-10-0-131-36 kubenswrapper[2573]: E1103 13:08:14.624535 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8rdgq" podUID="1bfc1afc-5b9f-4447-8d40-fabedbee9c07" Nov 03 13:08:15.290327 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.290227 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-36.us-east-2.compute.internal" event="NodeReady" Nov 03 13:08:15.290494 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.290396 2573 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Nov 03 13:08:15.354958 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.354923 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-p6cbj"] Nov 03 13:08:15.357484 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.356740 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-w5cql"] Nov 03 13:08:15.357717 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.357689 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-p6cbj" Nov 03 13:08:15.358312 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.358270 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-hslrf"] Nov 03 13:08:15.358395 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.358356 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-w5cql" Nov 03 13:08:15.359736 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.359718 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-vl8fr"] Nov 03 13:08:15.359865 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.359845 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-hslrf" Nov 03 13:08:15.361131 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.361110 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-rz4n9"] Nov 03 13:08:15.361224 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.361212 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-vl8fr" Nov 03 13:08:15.362818 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.362795 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-tm4nb"] Nov 03 13:08:15.362928 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.362888 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-rz4n9" Nov 03 13:08:15.364350 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.364328 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-6htht"] Nov 03 13:08:15.364472 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.364425 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-tm4nb" Nov 03 13:08:15.366941 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.366062 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-6nz8d"] Nov 03 13:08:15.366941 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.366201 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-6htht" Nov 03 13:08:15.367963 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.367945 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-555f8976d-mz5ct"] Nov 03 13:08:15.368081 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.368065 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-6nz8d" Nov 03 13:08:15.369910 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.369893 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-7c4986d449-4t8lj"] Nov 03 13:08:15.370011 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.369997 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-555f8976d-mz5ct" Nov 03 13:08:15.377214 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.376377 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-h46gl"] Nov 03 13:08:15.377214 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.376506 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-4t8lj" Nov 03 13:08:15.377898 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.377690 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Nov 03 13:08:15.377898 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.377717 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\"" Nov 03 13:08:15.377898 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.377803 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\"" Nov 03 13:08:15.377898 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.377833 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Nov 03 13:08:15.377898 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.377844 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Nov 03 13:08:15.377898 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.377858 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Nov 03 13:08:15.377898 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.377886 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Nov 03 13:08:15.378558 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.378538 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Nov 03 13:08:15.378642 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.378621 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Nov 03 13:08:15.378698 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.378666 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Nov 03 13:08:15.379329 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.379308 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-7f579888d9-ntt6m"] Nov 03 13:08:15.379467 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.379394 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-h46gl" Nov 03 13:08:15.379559 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.379537 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Nov 03 13:08:15.380069 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.380054 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Nov 03 13:08:15.381393 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.381376 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-vgnv8"] Nov 03 13:08:15.381486 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.381472 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-7f579888d9-ntt6m" Nov 03 13:08:15.383383 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.383366 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Nov 03 13:08:15.383546 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.383533 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Nov 03 13:08:15.383681 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.383663 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7dbb67bd57-74qfj"] Nov 03 13:08:15.383755 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.383685 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Nov 03 13:08:15.383896 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.383881 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Nov 03 13:08:15.383977 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.383901 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Nov 03 13:08:15.383977 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.383914 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-vgnv8" Nov 03 13:08:15.384072 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.383998 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Nov 03 13:08:15.384072 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.384030 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Nov 03 13:08:15.384164 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.384116 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Nov 03 13:08:15.384520 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.384505 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Nov 03 13:08:15.384620 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.384604 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Nov 03 13:08:15.384693 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.384677 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-vd7sh\"" Nov 03 13:08:15.385568 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.385542 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-cqvdv"] Nov 03 13:08:15.385778 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.385763 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7dbb67bd57-74qfj" Nov 03 13:08:15.390309 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.388470 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-jws95"] Nov 03 13:08:15.390309 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.388563 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-cqvdv" Nov 03 13:08:15.391516 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.391493 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-p6cbj"] Nov 03 13:08:15.391614 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.391534 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-hslrf"] Nov 03 13:08:15.391614 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.391557 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-w5cql"] Nov 03 13:08:15.391614 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.391579 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-d8f6l"] Nov 03 13:08:15.391868 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.391683 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-jws95" Nov 03 13:08:15.393464 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.393445 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-6nz8d"] Nov 03 13:08:15.393552 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.393544 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-d8f6l" Nov 03 13:08:15.400446 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.399162 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-6htht"] Nov 03 13:08:15.405682 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.405372 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-4t8lj"] Nov 03 13:08:15.405948 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.405723 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-jws95"] Nov 03 13:08:15.407131 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.407108 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7dbb67bd57-74qfj"] Nov 03 13:08:15.408341 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.408135 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-tm4nb"] Nov 03 13:08:15.409090 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.409071 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-vgnv8"] Nov 03 13:08:15.412837 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.411509 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-7f579888d9-ntt6m"] Nov 03 13:08:15.412837 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.411922 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-rz4n9"] Nov 03 13:08:15.416922 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.416840 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-h46gl"] Nov 03 13:08:15.417791 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.417551 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-cqvdv"] Nov 03 13:08:15.418527 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.418502 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-vl8fr"] Nov 03 13:08:15.421851 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.421824 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-d8f6l"] Nov 03 13:08:15.422957 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.422927 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qcsxn\" (UniqueName: \"kubernetes.io/projected/021e9e00-a973-4726-8f48-02a52c4bae14-kube-api-access-qcsxn\") pod \"volume-data-source-validator-58c9f5f46d-p6cbj\" (UID: \"021e9e00-a973-4726-8f48-02a52c4bae14\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-p6cbj" Nov 03 13:08:15.424194 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.422987 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9fa6d32d-b30b-4b76-9462-5f3f09443d02-serving-cert\") pod \"insights-operator-7c4986d449-4t8lj\" (UID: \"9fa6d32d-b30b-4b76-9462-5f3f09443d02\") " pod="openshift-insights/insights-operator-7c4986d449-4t8lj" Nov 03 13:08:15.424194 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.423047 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7jxlt\" (UniqueName: \"kubernetes.io/projected/07a7e2de-0c09-4819-9670-2be9460a155d-kube-api-access-7jxlt\") pod \"kas-connection-checker-685895c7c9-vl8fr\" (UID: \"07a7e2de-0c09-4819-9670-2be9460a155d\") " pod="kube-system/kas-connection-checker-685895c7c9-vl8fr" Nov 03 13:08:15.424194 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.423077 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9fa6d32d-b30b-4b76-9462-5f3f09443d02-service-ca-bundle\") pod \"insights-operator-7c4986d449-4t8lj\" (UID: \"9fa6d32d-b30b-4b76-9462-5f3f09443d02\") " pod="openshift-insights/insights-operator-7c4986d449-4t8lj" Nov 03 13:08:15.424194 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.423109 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9fa6d32d-b30b-4b76-9462-5f3f09443d02-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-4t8lj\" (UID: \"9fa6d32d-b30b-4b76-9462-5f3f09443d02\") " pod="openshift-insights/insights-operator-7c4986d449-4t8lj" Nov 03 13:08:15.424194 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.423153 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/cc8ef30e-6d4f-4913-8dc8-b09e5aea2f82-trusted-ca\") pod \"image-registry-555f8976d-mz5ct\" (UID: \"cc8ef30e-6d4f-4913-8dc8-b09e5aea2f82\") " pod="openshift-image-registry/image-registry-555f8976d-mz5ct" Nov 03 13:08:15.424194 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.423183 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pt6qk\" (UniqueName: \"kubernetes.io/projected/cd91de08-084e-4667-b13f-2e07b3cff624-kube-api-access-pt6qk\") pod \"console-operator-7cd56b8dbc-6htht\" (UID: \"cd91de08-084e-4667-b13f-2e07b3cff624\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-6htht" Nov 03 13:08:15.424194 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.423373 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n8x9b\" (UniqueName: \"kubernetes.io/projected/9fa6d32d-b30b-4b76-9462-5f3f09443d02-kube-api-access-n8x9b\") pod \"insights-operator-7c4986d449-4t8lj\" (UID: \"9fa6d32d-b30b-4b76-9462-5f3f09443d02\") " pod="openshift-insights/insights-operator-7c4986d449-4t8lj" Nov 03 13:08:15.424194 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.423403 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/cc8ef30e-6d4f-4913-8dc8-b09e5aea2f82-image-registry-private-configuration\") pod \"image-registry-555f8976d-mz5ct\" (UID: \"cc8ef30e-6d4f-4913-8dc8-b09e5aea2f82\") " pod="openshift-image-registry/image-registry-555f8976d-mz5ct" Nov 03 13:08:15.424194 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.423433 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/a3042b60-062f-4f9b-8406-f884c95a8729-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-tm4nb\" (UID: \"a3042b60-062f-4f9b-8406-f884c95a8729\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-tm4nb" Nov 03 13:08:15.424194 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.423594 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cb5g7\" (UniqueName: \"kubernetes.io/projected/c82d9f4b-8c00-4c31-8aa2-99cdc0d87e52-kube-api-access-cb5g7\") pod \"kas-connection-checker-685895c7c9-w5cql\" (UID: \"c82d9f4b-8c00-4c31-8aa2-99cdc0d87e52\") " pod="kube-system/kas-connection-checker-685895c7c9-w5cql" Nov 03 13:08:15.424194 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.423629 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/fa297a30-6da7-4a92-98fb-201323490bb0-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-rz4n9\" (UID: \"fa297a30-6da7-4a92-98fb-201323490bb0\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-rz4n9" Nov 03 13:08:15.424194 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.423672 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a3042b60-062f-4f9b-8406-f884c95a8729-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-tm4nb\" (UID: \"a3042b60-062f-4f9b-8406-f884c95a8729\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-tm4nb" Nov 03 13:08:15.424194 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.423702 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cc8ef30e-6d4f-4913-8dc8-b09e5aea2f82-registry-tls\") pod \"image-registry-555f8976d-mz5ct\" (UID: \"cc8ef30e-6d4f-4913-8dc8-b09e5aea2f82\") " pod="openshift-image-registry/image-registry-555f8976d-mz5ct" Nov 03 13:08:15.424194 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.423848 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6cb478ab-97aa-4881-87b2-b45293cc2349-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-6nz8d\" (UID: \"6cb478ab-97aa-4881-87b2-b45293cc2349\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-6nz8d" Nov 03 13:08:15.424194 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.423879 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h82mn\" (UniqueName: \"kubernetes.io/projected/cc8ef30e-6d4f-4913-8dc8-b09e5aea2f82-kube-api-access-h82mn\") pod \"image-registry-555f8976d-mz5ct\" (UID: \"cc8ef30e-6d4f-4913-8dc8-b09e5aea2f82\") " pod="openshift-image-registry/image-registry-555f8976d-mz5ct" Nov 03 13:08:15.427949 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.423905 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/cd91de08-084e-4667-b13f-2e07b3cff624-serving-cert\") pod \"console-operator-7cd56b8dbc-6htht\" (UID: \"cd91de08-084e-4667-b13f-2e07b3cff624\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-6htht" Nov 03 13:08:15.427949 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.424088 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cc8ef30e-6d4f-4913-8dc8-b09e5aea2f82-bound-sa-token\") pod \"image-registry-555f8976d-mz5ct\" (UID: \"cc8ef30e-6d4f-4913-8dc8-b09e5aea2f82\") " pod="openshift-image-registry/image-registry-555f8976d-mz5ct" Nov 03 13:08:15.427949 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.424117 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/cd91de08-084e-4667-b13f-2e07b3cff624-config\") pod \"console-operator-7cd56b8dbc-6htht\" (UID: \"cd91de08-084e-4667-b13f-2e07b3cff624\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-6htht" Nov 03 13:08:15.427949 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.424154 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x7mw6\" (UniqueName: \"kubernetes.io/projected/6cb478ab-97aa-4881-87b2-b45293cc2349-kube-api-access-x7mw6\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-6nz8d\" (UID: \"6cb478ab-97aa-4881-87b2-b45293cc2349\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-6nz8d" Nov 03 13:08:15.427949 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.424176 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9fa6d32d-b30b-4b76-9462-5f3f09443d02-tmp\") pod \"insights-operator-7c4986d449-4t8lj\" (UID: \"9fa6d32d-b30b-4b76-9462-5f3f09443d02\") " pod="openshift-insights/insights-operator-7c4986d449-4t8lj" Nov 03 13:08:15.427949 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.424389 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4lj56\" (UniqueName: \"kubernetes.io/projected/a3042b60-062f-4f9b-8406-f884c95a8729-kube-api-access-4lj56\") pod \"cluster-monitoring-operator-884576c57-tm4nb\" (UID: \"a3042b60-062f-4f9b-8406-f884c95a8729\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-tm4nb" Nov 03 13:08:15.427949 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.424427 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6cb478ab-97aa-4881-87b2-b45293cc2349-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-6nz8d\" (UID: \"6cb478ab-97aa-4881-87b2-b45293cc2349\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-6nz8d" Nov 03 13:08:15.427949 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.424465 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/fa297a30-6da7-4a92-98fb-201323490bb0-nginx-conf\") pod \"networking-console-plugin-86c478bb88-rz4n9\" (UID: \"fa297a30-6da7-4a92-98fb-201323490bb0\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-rz4n9" Nov 03 13:08:15.427949 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.424496 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/cc8ef30e-6d4f-4913-8dc8-b09e5aea2f82-registry-certificates\") pod \"image-registry-555f8976d-mz5ct\" (UID: \"cc8ef30e-6d4f-4913-8dc8-b09e5aea2f82\") " pod="openshift-image-registry/image-registry-555f8976d-mz5ct" Nov 03 13:08:15.427949 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.424646 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/cd91de08-084e-4667-b13f-2e07b3cff624-trusted-ca\") pod \"console-operator-7cd56b8dbc-6htht\" (UID: \"cd91de08-084e-4667-b13f-2e07b3cff624\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-6htht" Nov 03 13:08:15.427949 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.424684 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/9fa6d32d-b30b-4b76-9462-5f3f09443d02-snapshots\") pod \"insights-operator-7c4986d449-4t8lj\" (UID: \"9fa6d32d-b30b-4b76-9462-5f3f09443d02\") " pod="openshift-insights/insights-operator-7c4986d449-4t8lj" Nov 03 13:08:15.427949 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.424721 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zhwtz\" (UniqueName: \"kubernetes.io/projected/b8fe3220-d285-487f-9fc0-597438982de7-kube-api-access-zhwtz\") pod \"kas-connection-checker-685895c7c9-hslrf\" (UID: \"b8fe3220-d285-487f-9fc0-597438982de7\") " pod="kube-system/kas-connection-checker-685895c7c9-hslrf" Nov 03 13:08:15.427949 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.424748 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/cc8ef30e-6d4f-4913-8dc8-b09e5aea2f82-ca-trust-extracted\") pod \"image-registry-555f8976d-mz5ct\" (UID: \"cc8ef30e-6d4f-4913-8dc8-b09e5aea2f82\") " pod="openshift-image-registry/image-registry-555f8976d-mz5ct" Nov 03 13:08:15.427949 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.424906 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/cc8ef30e-6d4f-4913-8dc8-b09e5aea2f82-installation-pull-secrets\") pod \"image-registry-555f8976d-mz5ct\" (UID: \"cc8ef30e-6d4f-4913-8dc8-b09e5aea2f82\") " pod="openshift-image-registry/image-registry-555f8976d-mz5ct" Nov 03 13:08:15.428655 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.428636 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-555f8976d-mz5ct"] Nov 03 13:08:15.476162 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.476133 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 03 13:08:15.476300 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.476243 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\"" Nov 03 13:08:15.476361 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.476333 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 03 13:08:15.476361 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.476335 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Nov 03 13:08:15.476361 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.476355 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 03 13:08:15.476477 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.476391 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 03 13:08:15.476477 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.476403 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Nov 03 13:08:15.476477 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.476465 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Nov 03 13:08:15.476560 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.476511 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Nov 03 13:08:15.476560 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.476543 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Nov 03 13:08:15.477245 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.477228 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 03 13:08:15.477321 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.477249 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Nov 03 13:08:15.477674 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.477662 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Nov 03 13:08:15.477781 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.477767 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 03 13:08:15.477813 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.477804 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Nov 03 13:08:15.478096 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.478077 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Nov 03 13:08:15.478337 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.478320 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 03 13:08:15.487358 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.487319 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Nov 03 13:08:15.487358 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.487339 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 03 13:08:15.487544 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.487396 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Nov 03 13:08:15.487544 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.487485 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Nov 03 13:08:15.487923 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.487666 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Nov 03 13:08:15.488803 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.488786 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Nov 03 13:08:15.488898 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.488864 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 03 13:08:15.488898 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.488897 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Nov 03 13:08:15.488994 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.488956 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-bbkgd\"" Nov 03 13:08:15.489034 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.488990 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-gkggt\"" Nov 03 13:08:15.489126 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.489115 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-wd8kt\"" Nov 03 13:08:15.497135 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.497112 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Nov 03 13:08:15.504965 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.504943 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Nov 03 13:08:15.525837 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.525806 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4lj56\" (UniqueName: \"kubernetes.io/projected/a3042b60-062f-4f9b-8406-f884c95a8729-kube-api-access-4lj56\") pod \"cluster-monitoring-operator-884576c57-tm4nb\" (UID: \"a3042b60-062f-4f9b-8406-f884c95a8729\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-tm4nb" Nov 03 13:08:15.525837 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.525841 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6cb478ab-97aa-4881-87b2-b45293cc2349-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-6nz8d\" (UID: \"6cb478ab-97aa-4881-87b2-b45293cc2349\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-6nz8d" Nov 03 13:08:15.526053 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.525862 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/898bd0b5-0d3e-4577-b101-2cc7d75283cb-service-ca-bundle\") pod \"router-default-7f579888d9-ntt6m\" (UID: \"898bd0b5-0d3e-4577-b101-2cc7d75283cb\") " pod="openshift-ingress/router-default-7f579888d9-ntt6m" Nov 03 13:08:15.526053 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.525884 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0a892eb3-ecc4-4312-9f4d-73eeba6b6eee-registry-certificates\") pod \"image-registry-7dbb67bd57-74qfj\" (UID: \"0a892eb3-ecc4-4312-9f4d-73eeba6b6eee\") " pod="openshift-image-registry/image-registry-7dbb67bd57-74qfj" Nov 03 13:08:15.526053 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.525931 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/fa297a30-6da7-4a92-98fb-201323490bb0-nginx-conf\") pod \"networking-console-plugin-86c478bb88-rz4n9\" (UID: \"fa297a30-6da7-4a92-98fb-201323490bb0\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-rz4n9" Nov 03 13:08:15.526053 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.525987 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/cc8ef30e-6d4f-4913-8dc8-b09e5aea2f82-registry-certificates\") pod \"image-registry-555f8976d-mz5ct\" (UID: \"cc8ef30e-6d4f-4913-8dc8-b09e5aea2f82\") " pod="openshift-image-registry/image-registry-555f8976d-mz5ct" Nov 03 13:08:15.526053 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.526041 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/cd91de08-084e-4667-b13f-2e07b3cff624-trusted-ca\") pod \"console-operator-7cd56b8dbc-6htht\" (UID: \"cd91de08-084e-4667-b13f-2e07b3cff624\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-6htht" Nov 03 13:08:15.526313 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.526076 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/9fa6d32d-b30b-4b76-9462-5f3f09443d02-snapshots\") pod \"insights-operator-7c4986d449-4t8lj\" (UID: \"9fa6d32d-b30b-4b76-9462-5f3f09443d02\") " pod="openshift-insights/insights-operator-7c4986d449-4t8lj" Nov 03 13:08:15.526313 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.526097 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3fdeb9f3-6c67-4390-8073-17d4d5f802b9-metrics-tls\") pod \"dns-default-vgnv8\" (UID: \"3fdeb9f3-6c67-4390-8073-17d4d5f802b9\") " pod="openshift-dns/dns-default-vgnv8" Nov 03 13:08:15.526313 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.526123 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bddlp\" (UniqueName: \"kubernetes.io/projected/3fdeb9f3-6c67-4390-8073-17d4d5f802b9-kube-api-access-bddlp\") pod \"dns-default-vgnv8\" (UID: \"3fdeb9f3-6c67-4390-8073-17d4d5f802b9\") " pod="openshift-dns/dns-default-vgnv8" Nov 03 13:08:15.526313 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.526153 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9f20ddda-41fb-4aba-b257-7ebc59dbe678-serving-cert\") pod \"service-ca-operator-544d749c8c-cqvdv\" (UID: \"9f20ddda-41fb-4aba-b257-7ebc59dbe678\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-cqvdv" Nov 03 13:08:15.526313 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.526188 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhwtz\" (UniqueName: \"kubernetes.io/projected/b8fe3220-d285-487f-9fc0-597438982de7-kube-api-access-zhwtz\") pod \"kas-connection-checker-685895c7c9-hslrf\" (UID: \"b8fe3220-d285-487f-9fc0-597438982de7\") " pod="kube-system/kas-connection-checker-685895c7c9-hslrf" Nov 03 13:08:15.526313 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.526214 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/cc8ef30e-6d4f-4913-8dc8-b09e5aea2f82-ca-trust-extracted\") pod \"image-registry-555f8976d-mz5ct\" (UID: \"cc8ef30e-6d4f-4913-8dc8-b09e5aea2f82\") " pod="openshift-image-registry/image-registry-555f8976d-mz5ct" Nov 03 13:08:15.526313 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.526276 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/cc8ef30e-6d4f-4913-8dc8-b09e5aea2f82-installation-pull-secrets\") pod \"image-registry-555f8976d-mz5ct\" (UID: \"cc8ef30e-6d4f-4913-8dc8-b09e5aea2f82\") " pod="openshift-image-registry/image-registry-555f8976d-mz5ct" Nov 03 13:08:15.526642 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.526504 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/898bd0b5-0d3e-4577-b101-2cc7d75283cb-metrics-certs\") pod \"router-default-7f579888d9-ntt6m\" (UID: \"898bd0b5-0d3e-4577-b101-2cc7d75283cb\") " pod="openshift-ingress/router-default-7f579888d9-ntt6m" Nov 03 13:08:15.526642 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.526536 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0a892eb3-ecc4-4312-9f4d-73eeba6b6eee-image-registry-private-configuration\") pod \"image-registry-7dbb67bd57-74qfj\" (UID: \"0a892eb3-ecc4-4312-9f4d-73eeba6b6eee\") " pod="openshift-image-registry/image-registry-7dbb67bd57-74qfj" Nov 03 13:08:15.526642 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.526565 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/50d68cb7-b766-4674-b295-8f3229880994-cert\") pod \"ingress-canary-d8f6l\" (UID: \"50d68cb7-b766-4674-b295-8f3229880994\") " pod="openshift-ingress-canary/ingress-canary-d8f6l" Nov 03 13:08:15.526642 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.526621 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qcsxn\" (UniqueName: \"kubernetes.io/projected/021e9e00-a973-4726-8f48-02a52c4bae14-kube-api-access-qcsxn\") pod \"volume-data-source-validator-58c9f5f46d-p6cbj\" (UID: \"021e9e00-a973-4726-8f48-02a52c4bae14\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-p6cbj" Nov 03 13:08:15.526827 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.526647 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/898bd0b5-0d3e-4577-b101-2cc7d75283cb-stats-auth\") pod \"router-default-7f579888d9-ntt6m\" (UID: \"898bd0b5-0d3e-4577-b101-2cc7d75283cb\") " pod="openshift-ingress/router-default-7f579888d9-ntt6m" Nov 03 13:08:15.526827 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.526723 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6cb478ab-97aa-4881-87b2-b45293cc2349-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-6nz8d\" (UID: \"6cb478ab-97aa-4881-87b2-b45293cc2349\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-6nz8d" Nov 03 13:08:15.526827 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.526744 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/cc8ef30e-6d4f-4913-8dc8-b09e5aea2f82-ca-trust-extracted\") pod \"image-registry-555f8976d-mz5ct\" (UID: \"cc8ef30e-6d4f-4913-8dc8-b09e5aea2f82\") " pod="openshift-image-registry/image-registry-555f8976d-mz5ct" Nov 03 13:08:15.526827 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.526788 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/fa297a30-6da7-4a92-98fb-201323490bb0-nginx-conf\") pod \"networking-console-plugin-86c478bb88-rz4n9\" (UID: \"fa297a30-6da7-4a92-98fb-201323490bb0\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-rz4n9" Nov 03 13:08:15.526827 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.526791 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/cc8ef30e-6d4f-4913-8dc8-b09e5aea2f82-registry-certificates\") pod \"image-registry-555f8976d-mz5ct\" (UID: \"cc8ef30e-6d4f-4913-8dc8-b09e5aea2f82\") " pod="openshift-image-registry/image-registry-555f8976d-mz5ct" Nov 03 13:08:15.527013 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.526875 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/9fa6d32d-b30b-4b76-9462-5f3f09443d02-snapshots\") pod \"insights-operator-7c4986d449-4t8lj\" (UID: \"9fa6d32d-b30b-4b76-9462-5f3f09443d02\") " pod="openshift-insights/insights-operator-7c4986d449-4t8lj" Nov 03 13:08:15.527013 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.526889 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0a892eb3-ecc4-4312-9f4d-73eeba6b6eee-trusted-ca\") pod \"image-registry-7dbb67bd57-74qfj\" (UID: \"0a892eb3-ecc4-4312-9f4d-73eeba6b6eee\") " pod="openshift-image-registry/image-registry-7dbb67bd57-74qfj" Nov 03 13:08:15.527013 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.526944 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9fa6d32d-b30b-4b76-9462-5f3f09443d02-serving-cert\") pod \"insights-operator-7c4986d449-4t8lj\" (UID: \"9fa6d32d-b30b-4b76-9462-5f3f09443d02\") " pod="openshift-insights/insights-operator-7c4986d449-4t8lj" Nov 03 13:08:15.527013 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.526968 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bzlml\" (UniqueName: \"kubernetes.io/projected/898bd0b5-0d3e-4577-b101-2cc7d75283cb-kube-api-access-bzlml\") pod \"router-default-7f579888d9-ntt6m\" (UID: \"898bd0b5-0d3e-4577-b101-2cc7d75283cb\") " pod="openshift-ingress/router-default-7f579888d9-ntt6m" Nov 03 13:08:15.527013 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.526983 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/cd91de08-084e-4667-b13f-2e07b3cff624-trusted-ca\") pod \"console-operator-7cd56b8dbc-6htht\" (UID: \"cd91de08-084e-4667-b13f-2e07b3cff624\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-6htht" Nov 03 13:08:15.527013 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.526998 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7jxlt\" (UniqueName: \"kubernetes.io/projected/07a7e2de-0c09-4819-9670-2be9460a155d-kube-api-access-7jxlt\") pod \"kas-connection-checker-685895c7c9-vl8fr\" (UID: \"07a7e2de-0c09-4819-9670-2be9460a155d\") " pod="kube-system/kas-connection-checker-685895c7c9-vl8fr" Nov 03 13:08:15.527307 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.527036 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9fa6d32d-b30b-4b76-9462-5f3f09443d02-service-ca-bundle\") pod \"insights-operator-7c4986d449-4t8lj\" (UID: \"9fa6d32d-b30b-4b76-9462-5f3f09443d02\") " pod="openshift-insights/insights-operator-7c4986d449-4t8lj" Nov 03 13:08:15.527307 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.527059 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vbs5f\" (UniqueName: \"kubernetes.io/projected/21618783-0a4e-4b1f-8041-cabdf69f5bea-kube-api-access-vbs5f\") pod \"cluster-samples-operator-6ddb6c8b94-h46gl\" (UID: \"21618783-0a4e-4b1f-8041-cabdf69f5bea\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-h46gl" Nov 03 13:08:15.527307 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.527079 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9fa6d32d-b30b-4b76-9462-5f3f09443d02-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-4t8lj\" (UID: \"9fa6d32d-b30b-4b76-9462-5f3f09443d02\") " pod="openshift-insights/insights-operator-7c4986d449-4t8lj" Nov 03 13:08:15.527307 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.527132 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/cc8ef30e-6d4f-4913-8dc8-b09e5aea2f82-trusted-ca\") pod \"image-registry-555f8976d-mz5ct\" (UID: \"cc8ef30e-6d4f-4913-8dc8-b09e5aea2f82\") " pod="openshift-image-registry/image-registry-555f8976d-mz5ct" Nov 03 13:08:15.527487 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.527279 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pt6qk\" (UniqueName: \"kubernetes.io/projected/cd91de08-084e-4667-b13f-2e07b3cff624-kube-api-access-pt6qk\") pod \"console-operator-7cd56b8dbc-6htht\" (UID: \"cd91de08-084e-4667-b13f-2e07b3cff624\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-6htht" Nov 03 13:08:15.527487 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.527340 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8x9b\" (UniqueName: \"kubernetes.io/projected/9fa6d32d-b30b-4b76-9462-5f3f09443d02-kube-api-access-n8x9b\") pod \"insights-operator-7c4986d449-4t8lj\" (UID: \"9fa6d32d-b30b-4b76-9462-5f3f09443d02\") " pod="openshift-insights/insights-operator-7c4986d449-4t8lj" Nov 03 13:08:15.527487 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.527368 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/3fdeb9f3-6c67-4390-8073-17d4d5f802b9-config-volume\") pod \"dns-default-vgnv8\" (UID: \"3fdeb9f3-6c67-4390-8073-17d4d5f802b9\") " pod="openshift-dns/dns-default-vgnv8" Nov 03 13:08:15.527487 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.527411 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/cc8ef30e-6d4f-4913-8dc8-b09e5aea2f82-image-registry-private-configuration\") pod \"image-registry-555f8976d-mz5ct\" (UID: \"cc8ef30e-6d4f-4913-8dc8-b09e5aea2f82\") " pod="openshift-image-registry/image-registry-555f8976d-mz5ct" Nov 03 13:08:15.527487 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.527439 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/a3042b60-062f-4f9b-8406-f884c95a8729-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-tm4nb\" (UID: \"a3042b60-062f-4f9b-8406-f884c95a8729\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-tm4nb" Nov 03 13:08:15.527726 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.527486 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0a892eb3-ecc4-4312-9f4d-73eeba6b6eee-registry-tls\") pod \"image-registry-7dbb67bd57-74qfj\" (UID: \"0a892eb3-ecc4-4312-9f4d-73eeba6b6eee\") " pod="openshift-image-registry/image-registry-7dbb67bd57-74qfj" Nov 03 13:08:15.527726 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.527519 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k9fhg\" (UniqueName: \"kubernetes.io/projected/5c17f5bf-5b7d-4b05-a387-21ac6b5186ca-kube-api-access-k9fhg\") pod \"network-check-source-6f78d59f99-jws95\" (UID: \"5c17f5bf-5b7d-4b05-a387-21ac6b5186ca\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-jws95" Nov 03 13:08:15.527726 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.527566 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cb5g7\" (UniqueName: \"kubernetes.io/projected/c82d9f4b-8c00-4c31-8aa2-99cdc0d87e52-kube-api-access-cb5g7\") pod \"kas-connection-checker-685895c7c9-w5cql\" (UID: \"c82d9f4b-8c00-4c31-8aa2-99cdc0d87e52\") " pod="kube-system/kas-connection-checker-685895c7c9-w5cql" Nov 03 13:08:15.527726 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.527596 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0a892eb3-ecc4-4312-9f4d-73eeba6b6eee-bound-sa-token\") pod \"image-registry-7dbb67bd57-74qfj\" (UID: \"0a892eb3-ecc4-4312-9f4d-73eeba6b6eee\") " pod="openshift-image-registry/image-registry-7dbb67bd57-74qfj" Nov 03 13:08:15.527726 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.527641 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/fa297a30-6da7-4a92-98fb-201323490bb0-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-rz4n9\" (UID: \"fa297a30-6da7-4a92-98fb-201323490bb0\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-rz4n9" Nov 03 13:08:15.527726 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.527652 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9fa6d32d-b30b-4b76-9462-5f3f09443d02-service-ca-bundle\") pod \"insights-operator-7c4986d449-4t8lj\" (UID: \"9fa6d32d-b30b-4b76-9462-5f3f09443d02\") " pod="openshift-insights/insights-operator-7c4986d449-4t8lj" Nov 03 13:08:15.527726 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.527668 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/898bd0b5-0d3e-4577-b101-2cc7d75283cb-default-certificate\") pod \"router-default-7f579888d9-ntt6m\" (UID: \"898bd0b5-0d3e-4577-b101-2cc7d75283cb\") " pod="openshift-ingress/router-default-7f579888d9-ntt6m" Nov 03 13:08:15.527726 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.527708 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jknsv\" (UniqueName: \"kubernetes.io/projected/9f20ddda-41fb-4aba-b257-7ebc59dbe678-kube-api-access-jknsv\") pod \"service-ca-operator-544d749c8c-cqvdv\" (UID: \"9f20ddda-41fb-4aba-b257-7ebc59dbe678\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-cqvdv" Nov 03 13:08:15.528111 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.527734 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0a892eb3-ecc4-4312-9f4d-73eeba6b6eee-installation-pull-secrets\") pod \"image-registry-7dbb67bd57-74qfj\" (UID: \"0a892eb3-ecc4-4312-9f4d-73eeba6b6eee\") " pod="openshift-image-registry/image-registry-7dbb67bd57-74qfj" Nov 03 13:08:15.528111 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.527788 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a3042b60-062f-4f9b-8406-f884c95a8729-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-tm4nb\" (UID: \"a3042b60-062f-4f9b-8406-f884c95a8729\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-tm4nb" Nov 03 13:08:15.528111 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.527818 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cc8ef30e-6d4f-4913-8dc8-b09e5aea2f82-registry-tls\") pod \"image-registry-555f8976d-mz5ct\" (UID: \"cc8ef30e-6d4f-4913-8dc8-b09e5aea2f82\") " pod="openshift-image-registry/image-registry-555f8976d-mz5ct" Nov 03 13:08:15.528111 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.527859 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0a892eb3-ecc4-4312-9f4d-73eeba6b6eee-ca-trust-extracted\") pod \"image-registry-7dbb67bd57-74qfj\" (UID: \"0a892eb3-ecc4-4312-9f4d-73eeba6b6eee\") " pod="openshift-image-registry/image-registry-7dbb67bd57-74qfj" Nov 03 13:08:15.528111 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.527885 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6cb478ab-97aa-4881-87b2-b45293cc2349-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-6nz8d\" (UID: \"6cb478ab-97aa-4881-87b2-b45293cc2349\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-6nz8d" Nov 03 13:08:15.528111 ip-10-0-131-36 kubenswrapper[2573]: E1103 13:08:15.527982 2573 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 03 13:08:15.528111 ip-10-0-131-36 kubenswrapper[2573]: E1103 13:08:15.528058 2573 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 13:08:15.528111 ip-10-0-131-36 kubenswrapper[2573]: E1103 13:08:15.528059 2573 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 03 13:08:15.528111 ip-10-0-131-36 kubenswrapper[2573]: E1103 13:08:15.528068 2573 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-555f8976d-mz5ct: secret "image-registry-tls" not found Nov 03 13:08:15.528535 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.528111 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2sxmr\" (UniqueName: \"kubernetes.io/projected/50d68cb7-b766-4674-b295-8f3229880994-kube-api-access-2sxmr\") pod \"ingress-canary-d8f6l\" (UID: \"50d68cb7-b766-4674-b295-8f3229880994\") " pod="openshift-ingress-canary/ingress-canary-d8f6l" Nov 03 13:08:15.528535 ip-10-0-131-36 kubenswrapper[2573]: E1103 13:08:15.528151 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cc8ef30e-6d4f-4913-8dc8-b09e5aea2f82-registry-tls podName:cc8ef30e-6d4f-4913-8dc8-b09e5aea2f82 nodeName:}" failed. No retries permitted until 2026-11-03 13:08:16.028133985 +0000 UTC m=+31.002456608 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/cc8ef30e-6d4f-4913-8dc8-b09e5aea2f82-registry-tls") pod "image-registry-555f8976d-mz5ct" (UID: "cc8ef30e-6d4f-4913-8dc8-b09e5aea2f82") : secret "image-registry-tls" not found Nov 03 13:08:15.528535 ip-10-0-131-36 kubenswrapper[2573]: E1103 13:08:15.528186 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a3042b60-062f-4f9b-8406-f884c95a8729-cluster-monitoring-operator-tls podName:a3042b60-062f-4f9b-8406-f884c95a8729 nodeName:}" failed. No retries permitted until 2026-11-03 13:08:16.028174084 +0000 UTC m=+31.002496719 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/a3042b60-062f-4f9b-8406-f884c95a8729-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-tm4nb" (UID: "a3042b60-062f-4f9b-8406-f884c95a8729") : secret "cluster-monitoring-operator-tls" not found Nov 03 13:08:15.528535 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.528206 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h82mn\" (UniqueName: \"kubernetes.io/projected/cc8ef30e-6d4f-4913-8dc8-b09e5aea2f82-kube-api-access-h82mn\") pod \"image-registry-555f8976d-mz5ct\" (UID: \"cc8ef30e-6d4f-4913-8dc8-b09e5aea2f82\") " pod="openshift-image-registry/image-registry-555f8976d-mz5ct" Nov 03 13:08:15.528535 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.528253 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/cd91de08-084e-4667-b13f-2e07b3cff624-serving-cert\") pod \"console-operator-7cd56b8dbc-6htht\" (UID: \"cd91de08-084e-4667-b13f-2e07b3cff624\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-6htht" Nov 03 13:08:15.528535 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.528301 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9f20ddda-41fb-4aba-b257-7ebc59dbe678-config\") pod \"service-ca-operator-544d749c8c-cqvdv\" (UID: \"9f20ddda-41fb-4aba-b257-7ebc59dbe678\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-cqvdv" Nov 03 13:08:15.528535 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.528305 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/a3042b60-062f-4f9b-8406-f884c95a8729-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-tm4nb\" (UID: \"a3042b60-062f-4f9b-8406-f884c95a8729\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-tm4nb" Nov 03 13:08:15.528535 ip-10-0-131-36 kubenswrapper[2573]: E1103 13:08:15.528390 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fa297a30-6da7-4a92-98fb-201323490bb0-networking-console-plugin-cert podName:fa297a30-6da7-4a92-98fb-201323490bb0 nodeName:}" failed. No retries permitted until 2026-11-03 13:08:16.028364714 +0000 UTC m=+31.002687347 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/fa297a30-6da7-4a92-98fb-201323490bb0-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-rz4n9" (UID: "fa297a30-6da7-4a92-98fb-201323490bb0") : secret "networking-console-plugin-cert" not found Nov 03 13:08:15.528924 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.528630 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/cc8ef30e-6d4f-4913-8dc8-b09e5aea2f82-trusted-ca\") pod \"image-registry-555f8976d-mz5ct\" (UID: \"cc8ef30e-6d4f-4913-8dc8-b09e5aea2f82\") " pod="openshift-image-registry/image-registry-555f8976d-mz5ct" Nov 03 13:08:15.528924 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.528659 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9fa6d32d-b30b-4b76-9462-5f3f09443d02-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-4t8lj\" (UID: \"9fa6d32d-b30b-4b76-9462-5f3f09443d02\") " pod="openshift-insights/insights-operator-7c4986d449-4t8lj" Nov 03 13:08:15.528924 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.528680 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cc8ef30e-6d4f-4913-8dc8-b09e5aea2f82-bound-sa-token\") pod \"image-registry-555f8976d-mz5ct\" (UID: \"cc8ef30e-6d4f-4913-8dc8-b09e5aea2f82\") " pod="openshift-image-registry/image-registry-555f8976d-mz5ct" Nov 03 13:08:15.528924 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.528732 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/cd91de08-084e-4667-b13f-2e07b3cff624-config\") pod \"console-operator-7cd56b8dbc-6htht\" (UID: \"cd91de08-084e-4667-b13f-2e07b3cff624\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-6htht" Nov 03 13:08:15.528924 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.528847 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3fdeb9f3-6c67-4390-8073-17d4d5f802b9-tmp-dir\") pod \"dns-default-vgnv8\" (UID: \"3fdeb9f3-6c67-4390-8073-17d4d5f802b9\") " pod="openshift-dns/dns-default-vgnv8" Nov 03 13:08:15.529152 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.529024 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7mw6\" (UniqueName: \"kubernetes.io/projected/6cb478ab-97aa-4881-87b2-b45293cc2349-kube-api-access-x7mw6\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-6nz8d\" (UID: \"6cb478ab-97aa-4881-87b2-b45293cc2349\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-6nz8d" Nov 03 13:08:15.529152 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.529075 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9fa6d32d-b30b-4b76-9462-5f3f09443d02-tmp\") pod \"insights-operator-7c4986d449-4t8lj\" (UID: \"9fa6d32d-b30b-4b76-9462-5f3f09443d02\") " pod="openshift-insights/insights-operator-7c4986d449-4t8lj" Nov 03 13:08:15.529152 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.529105 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/21618783-0a4e-4b1f-8041-cabdf69f5bea-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-h46gl\" (UID: \"21618783-0a4e-4b1f-8041-cabdf69f5bea\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-h46gl" Nov 03 13:08:15.529511 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.529351 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/21618783-0a4e-4b1f-8041-cabdf69f5bea-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-h46gl\" (UID: \"21618783-0a4e-4b1f-8041-cabdf69f5bea\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-h46gl" Nov 03 13:08:15.529511 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.529390 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9fa6d32d-b30b-4b76-9462-5f3f09443d02-tmp\") pod \"insights-operator-7c4986d449-4t8lj\" (UID: \"9fa6d32d-b30b-4b76-9462-5f3f09443d02\") " pod="openshift-insights/insights-operator-7c4986d449-4t8lj" Nov 03 13:08:15.529511 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.529470 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xmbd7\" (UniqueName: \"kubernetes.io/projected/0a892eb3-ecc4-4312-9f4d-73eeba6b6eee-kube-api-access-xmbd7\") pod \"image-registry-7dbb67bd57-74qfj\" (UID: \"0a892eb3-ecc4-4312-9f4d-73eeba6b6eee\") " pod="openshift-image-registry/image-registry-7dbb67bd57-74qfj" Nov 03 13:08:15.529701 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.529666 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/cd91de08-084e-4667-b13f-2e07b3cff624-config\") pod \"console-operator-7cd56b8dbc-6htht\" (UID: \"cd91de08-084e-4667-b13f-2e07b3cff624\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-6htht" Nov 03 13:08:15.531071 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.531051 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9fa6d32d-b30b-4b76-9462-5f3f09443d02-serving-cert\") pod \"insights-operator-7c4986d449-4t8lj\" (UID: \"9fa6d32d-b30b-4b76-9462-5f3f09443d02\") " pod="openshift-insights/insights-operator-7c4986d449-4t8lj" Nov 03 13:08:15.531155 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.531068 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/cc8ef30e-6d4f-4913-8dc8-b09e5aea2f82-installation-pull-secrets\") pod \"image-registry-555f8976d-mz5ct\" (UID: \"cc8ef30e-6d4f-4913-8dc8-b09e5aea2f82\") " pod="openshift-image-registry/image-registry-555f8976d-mz5ct" Nov 03 13:08:15.531155 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.531081 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/cc8ef30e-6d4f-4913-8dc8-b09e5aea2f82-image-registry-private-configuration\") pod \"image-registry-555f8976d-mz5ct\" (UID: \"cc8ef30e-6d4f-4913-8dc8-b09e5aea2f82\") " pod="openshift-image-registry/image-registry-555f8976d-mz5ct" Nov 03 13:08:15.531265 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.531238 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6cb478ab-97aa-4881-87b2-b45293cc2349-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-6nz8d\" (UID: \"6cb478ab-97aa-4881-87b2-b45293cc2349\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-6nz8d" Nov 03 13:08:15.531497 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.531480 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/cd91de08-084e-4667-b13f-2e07b3cff624-serving-cert\") pod \"console-operator-7cd56b8dbc-6htht\" (UID: \"cd91de08-084e-4667-b13f-2e07b3cff624\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-6htht" Nov 03 13:08:15.541349 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.541259 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zhwtz\" (UniqueName: \"kubernetes.io/projected/b8fe3220-d285-487f-9fc0-597438982de7-kube-api-access-zhwtz\") pod \"kas-connection-checker-685895c7c9-hslrf\" (UID: \"b8fe3220-d285-487f-9fc0-597438982de7\") " pod="kube-system/kas-connection-checker-685895c7c9-hslrf" Nov 03 13:08:15.546135 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.546103 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7jxlt\" (UniqueName: \"kubernetes.io/projected/07a7e2de-0c09-4819-9670-2be9460a155d-kube-api-access-7jxlt\") pod \"kas-connection-checker-685895c7c9-vl8fr\" (UID: \"07a7e2de-0c09-4819-9670-2be9460a155d\") " pod="kube-system/kas-connection-checker-685895c7c9-vl8fr" Nov 03 13:08:15.546466 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.546431 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4lj56\" (UniqueName: \"kubernetes.io/projected/a3042b60-062f-4f9b-8406-f884c95a8729-kube-api-access-4lj56\") pod \"cluster-monitoring-operator-884576c57-tm4nb\" (UID: \"a3042b60-062f-4f9b-8406-f884c95a8729\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-tm4nb" Nov 03 13:08:15.546732 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.546714 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cb5g7\" (UniqueName: \"kubernetes.io/projected/c82d9f4b-8c00-4c31-8aa2-99cdc0d87e52-kube-api-access-cb5g7\") pod \"kas-connection-checker-685895c7c9-w5cql\" (UID: \"c82d9f4b-8c00-4c31-8aa2-99cdc0d87e52\") " pod="kube-system/kas-connection-checker-685895c7c9-w5cql" Nov 03 13:08:15.547556 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.547532 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h82mn\" (UniqueName: \"kubernetes.io/projected/cc8ef30e-6d4f-4913-8dc8-b09e5aea2f82-kube-api-access-h82mn\") pod \"image-registry-555f8976d-mz5ct\" (UID: \"cc8ef30e-6d4f-4913-8dc8-b09e5aea2f82\") " pod="openshift-image-registry/image-registry-555f8976d-mz5ct" Nov 03 13:08:15.548655 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.548631 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8x9b\" (UniqueName: \"kubernetes.io/projected/9fa6d32d-b30b-4b76-9462-5f3f09443d02-kube-api-access-n8x9b\") pod \"insights-operator-7c4986d449-4t8lj\" (UID: \"9fa6d32d-b30b-4b76-9462-5f3f09443d02\") " pod="openshift-insights/insights-operator-7c4986d449-4t8lj" Nov 03 13:08:15.549147 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.549126 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pt6qk\" (UniqueName: \"kubernetes.io/projected/cd91de08-084e-4667-b13f-2e07b3cff624-kube-api-access-pt6qk\") pod \"console-operator-7cd56b8dbc-6htht\" (UID: \"cd91de08-084e-4667-b13f-2e07b3cff624\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-6htht" Nov 03 13:08:15.549987 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.549968 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qcsxn\" (UniqueName: \"kubernetes.io/projected/021e9e00-a973-4726-8f48-02a52c4bae14-kube-api-access-qcsxn\") pod \"volume-data-source-validator-58c9f5f46d-p6cbj\" (UID: \"021e9e00-a973-4726-8f48-02a52c4bae14\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-p6cbj" Nov 03 13:08:15.550386 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.550362 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cc8ef30e-6d4f-4913-8dc8-b09e5aea2f82-bound-sa-token\") pod \"image-registry-555f8976d-mz5ct\" (UID: \"cc8ef30e-6d4f-4913-8dc8-b09e5aea2f82\") " pod="openshift-image-registry/image-registry-555f8976d-mz5ct" Nov 03 13:08:15.551884 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.551865 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x7mw6\" (UniqueName: \"kubernetes.io/projected/6cb478ab-97aa-4881-87b2-b45293cc2349-kube-api-access-x7mw6\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-6nz8d\" (UID: \"6cb478ab-97aa-4881-87b2-b45293cc2349\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-6nz8d" Nov 03 13:08:15.630402 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.630369 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbs5f\" (UniqueName: \"kubernetes.io/projected/21618783-0a4e-4b1f-8041-cabdf69f5bea-kube-api-access-vbs5f\") pod \"cluster-samples-operator-6ddb6c8b94-h46gl\" (UID: \"21618783-0a4e-4b1f-8041-cabdf69f5bea\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-h46gl" Nov 03 13:08:15.630891 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.630416 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/3fdeb9f3-6c67-4390-8073-17d4d5f802b9-config-volume\") pod \"dns-default-vgnv8\" (UID: \"3fdeb9f3-6c67-4390-8073-17d4d5f802b9\") " pod="openshift-dns/dns-default-vgnv8" Nov 03 13:08:15.630891 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.630544 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0a892eb3-ecc4-4312-9f4d-73eeba6b6eee-registry-tls\") pod \"image-registry-7dbb67bd57-74qfj\" (UID: \"0a892eb3-ecc4-4312-9f4d-73eeba6b6eee\") " pod="openshift-image-registry/image-registry-7dbb67bd57-74qfj" Nov 03 13:08:15.630891 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.630594 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9fhg\" (UniqueName: \"kubernetes.io/projected/5c17f5bf-5b7d-4b05-a387-21ac6b5186ca-kube-api-access-k9fhg\") pod \"network-check-source-6f78d59f99-jws95\" (UID: \"5c17f5bf-5b7d-4b05-a387-21ac6b5186ca\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-jws95" Nov 03 13:08:15.630891 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.630625 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0a892eb3-ecc4-4312-9f4d-73eeba6b6eee-bound-sa-token\") pod \"image-registry-7dbb67bd57-74qfj\" (UID: \"0a892eb3-ecc4-4312-9f4d-73eeba6b6eee\") " pod="openshift-image-registry/image-registry-7dbb67bd57-74qfj" Nov 03 13:08:15.630891 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.630665 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/898bd0b5-0d3e-4577-b101-2cc7d75283cb-default-certificate\") pod \"router-default-7f579888d9-ntt6m\" (UID: \"898bd0b5-0d3e-4577-b101-2cc7d75283cb\") " pod="openshift-ingress/router-default-7f579888d9-ntt6m" Nov 03 13:08:15.630891 ip-10-0-131-36 kubenswrapper[2573]: E1103 13:08:15.630680 2573 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 13:08:15.630891 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.630691 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jknsv\" (UniqueName: \"kubernetes.io/projected/9f20ddda-41fb-4aba-b257-7ebc59dbe678-kube-api-access-jknsv\") pod \"service-ca-operator-544d749c8c-cqvdv\" (UID: \"9f20ddda-41fb-4aba-b257-7ebc59dbe678\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-cqvdv" Nov 03 13:08:15.630891 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.630719 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0a892eb3-ecc4-4312-9f4d-73eeba6b6eee-installation-pull-secrets\") pod \"image-registry-7dbb67bd57-74qfj\" (UID: \"0a892eb3-ecc4-4312-9f4d-73eeba6b6eee\") " pod="openshift-image-registry/image-registry-7dbb67bd57-74qfj" Nov 03 13:08:15.630891 ip-10-0-131-36 kubenswrapper[2573]: E1103 13:08:15.630698 2573 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7dbb67bd57-74qfj: secret "image-registry-tls" not found Nov 03 13:08:15.630891 ip-10-0-131-36 kubenswrapper[2573]: E1103 13:08:15.630862 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0a892eb3-ecc4-4312-9f4d-73eeba6b6eee-registry-tls podName:0a892eb3-ecc4-4312-9f4d-73eeba6b6eee nodeName:}" failed. No retries permitted until 2026-11-03 13:08:16.130824288 +0000 UTC m=+31.105146921 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/0a892eb3-ecc4-4312-9f4d-73eeba6b6eee-registry-tls") pod "image-registry-7dbb67bd57-74qfj" (UID: "0a892eb3-ecc4-4312-9f4d-73eeba6b6eee") : secret "image-registry-tls" not found Nov 03 13:08:15.630891 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.630884 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0a892eb3-ecc4-4312-9f4d-73eeba6b6eee-ca-trust-extracted\") pod \"image-registry-7dbb67bd57-74qfj\" (UID: \"0a892eb3-ecc4-4312-9f4d-73eeba6b6eee\") " pod="openshift-image-registry/image-registry-7dbb67bd57-74qfj" Nov 03 13:08:15.631437 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.630938 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2sxmr\" (UniqueName: \"kubernetes.io/projected/50d68cb7-b766-4674-b295-8f3229880994-kube-api-access-2sxmr\") pod \"ingress-canary-d8f6l\" (UID: \"50d68cb7-b766-4674-b295-8f3229880994\") " pod="openshift-ingress-canary/ingress-canary-d8f6l" Nov 03 13:08:15.631437 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.630983 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9f20ddda-41fb-4aba-b257-7ebc59dbe678-config\") pod \"service-ca-operator-544d749c8c-cqvdv\" (UID: \"9f20ddda-41fb-4aba-b257-7ebc59dbe678\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-cqvdv" Nov 03 13:08:15.631437 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.630980 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/3fdeb9f3-6c67-4390-8073-17d4d5f802b9-config-volume\") pod \"dns-default-vgnv8\" (UID: \"3fdeb9f3-6c67-4390-8073-17d4d5f802b9\") " pod="openshift-dns/dns-default-vgnv8" Nov 03 13:08:15.631437 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.631113 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3fdeb9f3-6c67-4390-8073-17d4d5f802b9-tmp-dir\") pod \"dns-default-vgnv8\" (UID: \"3fdeb9f3-6c67-4390-8073-17d4d5f802b9\") " pod="openshift-dns/dns-default-vgnv8" Nov 03 13:08:15.631437 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.631156 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/21618783-0a4e-4b1f-8041-cabdf69f5bea-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-h46gl\" (UID: \"21618783-0a4e-4b1f-8041-cabdf69f5bea\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-h46gl" Nov 03 13:08:15.631437 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.631185 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/21618783-0a4e-4b1f-8041-cabdf69f5bea-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-h46gl\" (UID: \"21618783-0a4e-4b1f-8041-cabdf69f5bea\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-h46gl" Nov 03 13:08:15.631437 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.631212 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xmbd7\" (UniqueName: \"kubernetes.io/projected/0a892eb3-ecc4-4312-9f4d-73eeba6b6eee-kube-api-access-xmbd7\") pod \"image-registry-7dbb67bd57-74qfj\" (UID: \"0a892eb3-ecc4-4312-9f4d-73eeba6b6eee\") " pod="openshift-image-registry/image-registry-7dbb67bd57-74qfj" Nov 03 13:08:15.631437 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.631243 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/898bd0b5-0d3e-4577-b101-2cc7d75283cb-service-ca-bundle\") pod \"router-default-7f579888d9-ntt6m\" (UID: \"898bd0b5-0d3e-4577-b101-2cc7d75283cb\") " pod="openshift-ingress/router-default-7f579888d9-ntt6m" Nov 03 13:08:15.631437 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.631276 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0a892eb3-ecc4-4312-9f4d-73eeba6b6eee-registry-certificates\") pod \"image-registry-7dbb67bd57-74qfj\" (UID: \"0a892eb3-ecc4-4312-9f4d-73eeba6b6eee\") " pod="openshift-image-registry/image-registry-7dbb67bd57-74qfj" Nov 03 13:08:15.631437 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.631274 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0a892eb3-ecc4-4312-9f4d-73eeba6b6eee-ca-trust-extracted\") pod \"image-registry-7dbb67bd57-74qfj\" (UID: \"0a892eb3-ecc4-4312-9f4d-73eeba6b6eee\") " pod="openshift-image-registry/image-registry-7dbb67bd57-74qfj" Nov 03 13:08:15.631437 ip-10-0-131-36 kubenswrapper[2573]: E1103 13:08:15.631297 2573 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 03 13:08:15.631437 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.631336 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3fdeb9f3-6c67-4390-8073-17d4d5f802b9-metrics-tls\") pod \"dns-default-vgnv8\" (UID: \"3fdeb9f3-6c67-4390-8073-17d4d5f802b9\") " pod="openshift-dns/dns-default-vgnv8" Nov 03 13:08:15.631437 ip-10-0-131-36 kubenswrapper[2573]: E1103 13:08:15.631370 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/21618783-0a4e-4b1f-8041-cabdf69f5bea-samples-operator-tls podName:21618783-0a4e-4b1f-8041-cabdf69f5bea nodeName:}" failed. No retries permitted until 2026-11-03 13:08:16.131353576 +0000 UTC m=+31.105676212 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/21618783-0a4e-4b1f-8041-cabdf69f5bea-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-h46gl" (UID: "21618783-0a4e-4b1f-8041-cabdf69f5bea") : secret "samples-operator-tls" not found Nov 03 13:08:15.631437 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.631395 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bddlp\" (UniqueName: \"kubernetes.io/projected/3fdeb9f3-6c67-4390-8073-17d4d5f802b9-kube-api-access-bddlp\") pod \"dns-default-vgnv8\" (UID: \"3fdeb9f3-6c67-4390-8073-17d4d5f802b9\") " pod="openshift-dns/dns-default-vgnv8" Nov 03 13:08:15.631437 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.631407 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3fdeb9f3-6c67-4390-8073-17d4d5f802b9-tmp-dir\") pod \"dns-default-vgnv8\" (UID: \"3fdeb9f3-6c67-4390-8073-17d4d5f802b9\") " pod="openshift-dns/dns-default-vgnv8" Nov 03 13:08:15.631437 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.631429 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9f20ddda-41fb-4aba-b257-7ebc59dbe678-serving-cert\") pod \"service-ca-operator-544d749c8c-cqvdv\" (UID: \"9f20ddda-41fb-4aba-b257-7ebc59dbe678\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-cqvdv" Nov 03 13:08:15.632247 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.631471 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/898bd0b5-0d3e-4577-b101-2cc7d75283cb-metrics-certs\") pod \"router-default-7f579888d9-ntt6m\" (UID: \"898bd0b5-0d3e-4577-b101-2cc7d75283cb\") " pod="openshift-ingress/router-default-7f579888d9-ntt6m" Nov 03 13:08:15.632247 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.631498 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0a892eb3-ecc4-4312-9f4d-73eeba6b6eee-image-registry-private-configuration\") pod \"image-registry-7dbb67bd57-74qfj\" (UID: \"0a892eb3-ecc4-4312-9f4d-73eeba6b6eee\") " pod="openshift-image-registry/image-registry-7dbb67bd57-74qfj" Nov 03 13:08:15.632247 ip-10-0-131-36 kubenswrapper[2573]: E1103 13:08:15.631434 2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 13:08:15.632247 ip-10-0-131-36 kubenswrapper[2573]: E1103 13:08:15.631517 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/898bd0b5-0d3e-4577-b101-2cc7d75283cb-service-ca-bundle podName:898bd0b5-0d3e-4577-b101-2cc7d75283cb nodeName:}" failed. No retries permitted until 2026-11-03 13:08:16.131502696 +0000 UTC m=+31.105825335 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/898bd0b5-0d3e-4577-b101-2cc7d75283cb-service-ca-bundle") pod "router-default-7f579888d9-ntt6m" (UID: "898bd0b5-0d3e-4577-b101-2cc7d75283cb") : configmap references non-existent config key: service-ca.crt Nov 03 13:08:15.632247 ip-10-0-131-36 kubenswrapper[2573]: E1103 13:08:15.631620 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3fdeb9f3-6c67-4390-8073-17d4d5f802b9-metrics-tls podName:3fdeb9f3-6c67-4390-8073-17d4d5f802b9 nodeName:}" failed. No retries permitted until 2026-11-03 13:08:16.131602973 +0000 UTC m=+31.105925593 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3fdeb9f3-6c67-4390-8073-17d4d5f802b9-metrics-tls") pod "dns-default-vgnv8" (UID: "3fdeb9f3-6c67-4390-8073-17d4d5f802b9") : secret "dns-default-metrics-tls" not found Nov 03 13:08:15.632247 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.631640 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/50d68cb7-b766-4674-b295-8f3229880994-cert\") pod \"ingress-canary-d8f6l\" (UID: \"50d68cb7-b766-4674-b295-8f3229880994\") " pod="openshift-ingress-canary/ingress-canary-d8f6l" Nov 03 13:08:15.632247 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.631679 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/898bd0b5-0d3e-4577-b101-2cc7d75283cb-stats-auth\") pod \"router-default-7f579888d9-ntt6m\" (UID: \"898bd0b5-0d3e-4577-b101-2cc7d75283cb\") " pod="openshift-ingress/router-default-7f579888d9-ntt6m" Nov 03 13:08:15.632247 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.631704 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0a892eb3-ecc4-4312-9f4d-73eeba6b6eee-trusted-ca\") pod \"image-registry-7dbb67bd57-74qfj\" (UID: \"0a892eb3-ecc4-4312-9f4d-73eeba6b6eee\") " pod="openshift-image-registry/image-registry-7dbb67bd57-74qfj" Nov 03 13:08:15.632247 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.631741 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9f20ddda-41fb-4aba-b257-7ebc59dbe678-config\") pod \"service-ca-operator-544d749c8c-cqvdv\" (UID: \"9f20ddda-41fb-4aba-b257-7ebc59dbe678\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-cqvdv" Nov 03 13:08:15.632247 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.631743 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzlml\" (UniqueName: \"kubernetes.io/projected/898bd0b5-0d3e-4577-b101-2cc7d75283cb-kube-api-access-bzlml\") pod \"router-default-7f579888d9-ntt6m\" (UID: \"898bd0b5-0d3e-4577-b101-2cc7d75283cb\") " pod="openshift-ingress/router-default-7f579888d9-ntt6m" Nov 03 13:08:15.632247 ip-10-0-131-36 kubenswrapper[2573]: E1103 13:08:15.632013 2573 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 03 13:08:15.632247 ip-10-0-131-36 kubenswrapper[2573]: E1103 13:08:15.632068 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/898bd0b5-0d3e-4577-b101-2cc7d75283cb-metrics-certs podName:898bd0b5-0d3e-4577-b101-2cc7d75283cb nodeName:}" failed. No retries permitted until 2026-11-03 13:08:16.132053517 +0000 UTC m=+31.106376138 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/898bd0b5-0d3e-4577-b101-2cc7d75283cb-metrics-certs") pod "router-default-7f579888d9-ntt6m" (UID: "898bd0b5-0d3e-4577-b101-2cc7d75283cb") : secret "router-metrics-certs-default" not found Nov 03 13:08:15.632247 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.632079 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0a892eb3-ecc4-4312-9f4d-73eeba6b6eee-registry-certificates\") pod \"image-registry-7dbb67bd57-74qfj\" (UID: \"0a892eb3-ecc4-4312-9f4d-73eeba6b6eee\") " pod="openshift-image-registry/image-registry-7dbb67bd57-74qfj" Nov 03 13:08:15.632885 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.632622 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/21618783-0a4e-4b1f-8041-cabdf69f5bea-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-h46gl\" (UID: \"21618783-0a4e-4b1f-8041-cabdf69f5bea\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-h46gl" Nov 03 13:08:15.633459 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.633360 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0a892eb3-ecc4-4312-9f4d-73eeba6b6eee-trusted-ca\") pod \"image-registry-7dbb67bd57-74qfj\" (UID: \"0a892eb3-ecc4-4312-9f4d-73eeba6b6eee\") " pod="openshift-image-registry/image-registry-7dbb67bd57-74qfj" Nov 03 13:08:15.633819 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.633770 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0a892eb3-ecc4-4312-9f4d-73eeba6b6eee-installation-pull-secrets\") pod \"image-registry-7dbb67bd57-74qfj\" (UID: \"0a892eb3-ecc4-4312-9f4d-73eeba6b6eee\") " pod="openshift-image-registry/image-registry-7dbb67bd57-74qfj" Nov 03 13:08:15.633819 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.633802 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/898bd0b5-0d3e-4577-b101-2cc7d75283cb-default-certificate\") pod \"router-default-7f579888d9-ntt6m\" (UID: \"898bd0b5-0d3e-4577-b101-2cc7d75283cb\") " pod="openshift-ingress/router-default-7f579888d9-ntt6m" Nov 03 13:08:15.634174 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.633866 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9f20ddda-41fb-4aba-b257-7ebc59dbe678-serving-cert\") pod \"service-ca-operator-544d749c8c-cqvdv\" (UID: \"9f20ddda-41fb-4aba-b257-7ebc59dbe678\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-cqvdv" Nov 03 13:08:15.634174 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.633964 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0a892eb3-ecc4-4312-9f4d-73eeba6b6eee-image-registry-private-configuration\") pod \"image-registry-7dbb67bd57-74qfj\" (UID: \"0a892eb3-ecc4-4312-9f4d-73eeba6b6eee\") " pod="openshift-image-registry/image-registry-7dbb67bd57-74qfj" Nov 03 13:08:15.634834 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.634770 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/898bd0b5-0d3e-4577-b101-2cc7d75283cb-stats-auth\") pod \"router-default-7f579888d9-ntt6m\" (UID: \"898bd0b5-0d3e-4577-b101-2cc7d75283cb\") " pod="openshift-ingress/router-default-7f579888d9-ntt6m" Nov 03 13:08:15.635544 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.635527 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/50d68cb7-b766-4674-b295-8f3229880994-cert\") pod \"ingress-canary-d8f6l\" (UID: \"50d68cb7-b766-4674-b295-8f3229880994\") " pod="openshift-ingress-canary/ingress-canary-d8f6l" Nov 03 13:08:15.642835 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.642769 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0a892eb3-ecc4-4312-9f4d-73eeba6b6eee-bound-sa-token\") pod \"image-registry-7dbb67bd57-74qfj\" (UID: \"0a892eb3-ecc4-4312-9f4d-73eeba6b6eee\") " pod="openshift-image-registry/image-registry-7dbb67bd57-74qfj" Nov 03 13:08:15.642835 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.642786 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bddlp\" (UniqueName: \"kubernetes.io/projected/3fdeb9f3-6c67-4390-8073-17d4d5f802b9-kube-api-access-bddlp\") pod \"dns-default-vgnv8\" (UID: \"3fdeb9f3-6c67-4390-8073-17d4d5f802b9\") " pod="openshift-dns/dns-default-vgnv8" Nov 03 13:08:15.642835 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.642792 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jknsv\" (UniqueName: \"kubernetes.io/projected/9f20ddda-41fb-4aba-b257-7ebc59dbe678-kube-api-access-jknsv\") pod \"service-ca-operator-544d749c8c-cqvdv\" (UID: \"9f20ddda-41fb-4aba-b257-7ebc59dbe678\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-cqvdv" Nov 03 13:08:15.643061 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.642881 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2sxmr\" (UniqueName: \"kubernetes.io/projected/50d68cb7-b766-4674-b295-8f3229880994-kube-api-access-2sxmr\") pod \"ingress-canary-d8f6l\" (UID: \"50d68cb7-b766-4674-b295-8f3229880994\") " pod="openshift-ingress-canary/ingress-canary-d8f6l" Nov 03 13:08:15.643061 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.642885 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbs5f\" (UniqueName: \"kubernetes.io/projected/21618783-0a4e-4b1f-8041-cabdf69f5bea-kube-api-access-vbs5f\") pod \"cluster-samples-operator-6ddb6c8b94-h46gl\" (UID: \"21618783-0a4e-4b1f-8041-cabdf69f5bea\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-h46gl" Nov 03 13:08:15.643061 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.642976 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9fhg\" (UniqueName: \"kubernetes.io/projected/5c17f5bf-5b7d-4b05-a387-21ac6b5186ca-kube-api-access-k9fhg\") pod \"network-check-source-6f78d59f99-jws95\" (UID: \"5c17f5bf-5b7d-4b05-a387-21ac6b5186ca\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-jws95" Nov 03 13:08:15.643061 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.642999 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xmbd7\" (UniqueName: \"kubernetes.io/projected/0a892eb3-ecc4-4312-9f4d-73eeba6b6eee-kube-api-access-xmbd7\") pod \"image-registry-7dbb67bd57-74qfj\" (UID: \"0a892eb3-ecc4-4312-9f4d-73eeba6b6eee\") " pod="openshift-image-registry/image-registry-7dbb67bd57-74qfj" Nov 03 13:08:15.644132 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.644108 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bzlml\" (UniqueName: \"kubernetes.io/projected/898bd0b5-0d3e-4577-b101-2cc7d75283cb-kube-api-access-bzlml\") pod \"router-default-7f579888d9-ntt6m\" (UID: \"898bd0b5-0d3e-4577-b101-2cc7d75283cb\") " pod="openshift-ingress/router-default-7f579888d9-ntt6m" Nov 03 13:08:15.673674 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.673640 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-p6cbj" Nov 03 13:08:15.681450 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.681415 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-w5cql" Nov 03 13:08:15.688264 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.688211 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-hslrf" Nov 03 13:08:15.695967 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.695939 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-vl8fr" Nov 03 13:08:15.738176 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.738142 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-6htht" Nov 03 13:08:15.742596 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.742564 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-6nz8d" Nov 03 13:08:15.754937 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.754541 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-4t8lj" Nov 03 13:08:15.790118 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.789870 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-cqvdv" Nov 03 13:08:15.792586 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.792470 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-jws95" Nov 03 13:08:15.802917 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.797430 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-d8f6l" Nov 03 13:08:15.981615 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.981584 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-w5cql"] Nov 03 13:08:15.983738 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.983693 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-vl8fr"] Nov 03 13:08:15.985023 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.985002 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-p6cbj"] Nov 03 13:08:15.985237 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:08:15.985180 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc82d9f4b_8c00_4c31_8aa2_99cdc0d87e52.slice/crio-a670effa2a69f2c5a5f3625db93d19af7b21e7237cd4c41a16088e66cb4104a2 WatchSource:0}: Error finding container a670effa2a69f2c5a5f3625db93d19af7b21e7237cd4c41a16088e66cb4104a2: Status 404 returned error can't find the container with id a670effa2a69f2c5a5f3625db93d19af7b21e7237cd4c41a16088e66cb4104a2 Nov 03 13:08:15.986168 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:15.986141 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-hslrf"] Nov 03 13:08:15.989091 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:08:15.989067 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8fe3220_d285_487f_9fc0_597438982de7.slice/crio-68fe3802481c8f7a3638809cd85a097698ee57db88bc75d6ddd15414cd29f9b5 WatchSource:0}: Error finding container 68fe3802481c8f7a3638809cd85a097698ee57db88bc75d6ddd15414cd29f9b5: Status 404 returned error can't find the container with id 68fe3802481c8f7a3638809cd85a097698ee57db88bc75d6ddd15414cd29f9b5 Nov 03 13:08:15.989796 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:08:15.989769 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07a7e2de_0c09_4819_9670_2be9460a155d.slice/crio-12c34fc486ae00af6cdac52a0c622b9549f772c0e690148eea53070e65b1da62 WatchSource:0}: Error finding container 12c34fc486ae00af6cdac52a0c622b9549f772c0e690148eea53070e65b1da62: Status 404 returned error can't find the container with id 12c34fc486ae00af6cdac52a0c622b9549f772c0e690148eea53070e65b1da62 Nov 03 13:08:15.990480 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:08:15.990462 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod021e9e00_a973_4726_8f48_02a52c4bae14.slice/crio-84fabe8a79acb6040874b2b7e92579078c1dff682320fea274fd83a4f027c1ea WatchSource:0}: Error finding container 84fabe8a79acb6040874b2b7e92579078c1dff682320fea274fd83a4f027c1ea: Status 404 returned error can't find the container with id 84fabe8a79acb6040874b2b7e92579078c1dff682320fea274fd83a4f027c1ea Nov 03 13:08:16.036902 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:16.036459 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/fa297a30-6da7-4a92-98fb-201323490bb0-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-rz4n9\" (UID: \"fa297a30-6da7-4a92-98fb-201323490bb0\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-rz4n9" Nov 03 13:08:16.036902 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:16.036519 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a3042b60-062f-4f9b-8406-f884c95a8729-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-tm4nb\" (UID: \"a3042b60-062f-4f9b-8406-f884c95a8729\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-tm4nb" Nov 03 13:08:16.036902 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:16.036551 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cc8ef30e-6d4f-4913-8dc8-b09e5aea2f82-registry-tls\") pod \"image-registry-555f8976d-mz5ct\" (UID: \"cc8ef30e-6d4f-4913-8dc8-b09e5aea2f82\") " pod="openshift-image-registry/image-registry-555f8976d-mz5ct" Nov 03 13:08:16.036902 ip-10-0-131-36 kubenswrapper[2573]: E1103 13:08:16.036577 2573 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 03 13:08:16.036902 ip-10-0-131-36 kubenswrapper[2573]: E1103 13:08:16.036656 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fa297a30-6da7-4a92-98fb-201323490bb0-networking-console-plugin-cert podName:fa297a30-6da7-4a92-98fb-201323490bb0 nodeName:}" failed. No retries permitted until 2026-11-03 13:08:17.036634994 +0000 UTC m=+32.010957630 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/fa297a30-6da7-4a92-98fb-201323490bb0-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-rz4n9" (UID: "fa297a30-6da7-4a92-98fb-201323490bb0") : secret "networking-console-plugin-cert" not found Nov 03 13:08:16.036902 ip-10-0-131-36 kubenswrapper[2573]: E1103 13:08:16.036712 2573 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 13:08:16.036902 ip-10-0-131-36 kubenswrapper[2573]: E1103 13:08:16.036727 2573 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-555f8976d-mz5ct: secret "image-registry-tls" not found Nov 03 13:08:16.036902 ip-10-0-131-36 kubenswrapper[2573]: E1103 13:08:16.036773 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cc8ef30e-6d4f-4913-8dc8-b09e5aea2f82-registry-tls podName:cc8ef30e-6d4f-4913-8dc8-b09e5aea2f82 nodeName:}" failed. No retries permitted until 2026-11-03 13:08:17.036758407 +0000 UTC m=+32.011081043 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/cc8ef30e-6d4f-4913-8dc8-b09e5aea2f82-registry-tls") pod "image-registry-555f8976d-mz5ct" (UID: "cc8ef30e-6d4f-4913-8dc8-b09e5aea2f82") : secret "image-registry-tls" not found Nov 03 13:08:16.036902 ip-10-0-131-36 kubenswrapper[2573]: E1103 13:08:16.036831 2573 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 03 13:08:16.036902 ip-10-0-131-36 kubenswrapper[2573]: E1103 13:08:16.036870 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a3042b60-062f-4f9b-8406-f884c95a8729-cluster-monitoring-operator-tls podName:a3042b60-062f-4f9b-8406-f884c95a8729 nodeName:}" failed. No retries permitted until 2026-11-03 13:08:17.036853928 +0000 UTC m=+32.011176548 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/a3042b60-062f-4f9b-8406-f884c95a8729-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-tm4nb" (UID: "a3042b60-062f-4f9b-8406-f884c95a8729") : secret "cluster-monitoring-operator-tls" not found Nov 03 13:08:16.052692 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:16.052610 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-6nz8d"] Nov 03 13:08:16.057368 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:08:16.057334 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6cb478ab_97aa_4881_87b2_b45293cc2349.slice/crio-1658c9a5bbf6b90cd5b0c2c5ef8bc57b57b5e38c2ebde6e79097b6a75ad221c5 WatchSource:0}: Error finding container 1658c9a5bbf6b90cd5b0c2c5ef8bc57b57b5e38c2ebde6e79097b6a75ad221c5: Status 404 returned error can't find the container with id 1658c9a5bbf6b90cd5b0c2c5ef8bc57b57b5e38c2ebde6e79097b6a75ad221c5 Nov 03 13:08:16.057735 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:16.057713 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-4t8lj"] Nov 03 13:08:16.067306 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:16.067259 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-6htht"] Nov 03 13:08:16.118105 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:16.118071 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-cqvdv"] Nov 03 13:08:16.120656 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:16.120632 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-d8f6l"] Nov 03 13:08:16.125668 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:08:16.125625 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50d68cb7_b766_4674_b295_8f3229880994.slice/crio-3cf1dc4f654fe1eddf341ba01ad4d52e81aac0f2f4a81246a19b59883c62f85a WatchSource:0}: Error finding container 3cf1dc4f654fe1eddf341ba01ad4d52e81aac0f2f4a81246a19b59883c62f85a: Status 404 returned error can't find the container with id 3cf1dc4f654fe1eddf341ba01ad4d52e81aac0f2f4a81246a19b59883c62f85a Nov 03 13:08:16.129606 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:16.129569 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-jws95"] Nov 03 13:08:16.134190 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:08:16.134158 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c17f5bf_5b7d_4b05_a387_21ac6b5186ca.slice/crio-ab29b318c08555358cf838590f8909d568ac39d5a7ad9c2cbd6830c22a1c9f52 WatchSource:0}: Error finding container ab29b318c08555358cf838590f8909d568ac39d5a7ad9c2cbd6830c22a1c9f52: Status 404 returned error can't find the container with id ab29b318c08555358cf838590f8909d568ac39d5a7ad9c2cbd6830c22a1c9f52 Nov 03 13:08:16.137436 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:16.137256 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3fdeb9f3-6c67-4390-8073-17d4d5f802b9-metrics-tls\") pod \"dns-default-vgnv8\" (UID: \"3fdeb9f3-6c67-4390-8073-17d4d5f802b9\") " pod="openshift-dns/dns-default-vgnv8" Nov 03 13:08:16.137436 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:16.137316 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/898bd0b5-0d3e-4577-b101-2cc7d75283cb-metrics-certs\") pod \"router-default-7f579888d9-ntt6m\" (UID: \"898bd0b5-0d3e-4577-b101-2cc7d75283cb\") " pod="openshift-ingress/router-default-7f579888d9-ntt6m" Nov 03 13:08:16.137436 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:16.137367 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0a892eb3-ecc4-4312-9f4d-73eeba6b6eee-registry-tls\") pod \"image-registry-7dbb67bd57-74qfj\" (UID: \"0a892eb3-ecc4-4312-9f4d-73eeba6b6eee\") " pod="openshift-image-registry/image-registry-7dbb67bd57-74qfj" Nov 03 13:08:16.137436 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:16.137425 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/21618783-0a4e-4b1f-8041-cabdf69f5bea-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-h46gl\" (UID: \"21618783-0a4e-4b1f-8041-cabdf69f5bea\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-h46gl" Nov 03 13:08:16.137593 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:16.137446 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/898bd0b5-0d3e-4577-b101-2cc7d75283cb-service-ca-bundle\") pod \"router-default-7f579888d9-ntt6m\" (UID: \"898bd0b5-0d3e-4577-b101-2cc7d75283cb\") " pod="openshift-ingress/router-default-7f579888d9-ntt6m" Nov 03 13:08:16.137593 ip-10-0-131-36 kubenswrapper[2573]: E1103 13:08:16.137542 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/898bd0b5-0d3e-4577-b101-2cc7d75283cb-service-ca-bundle podName:898bd0b5-0d3e-4577-b101-2cc7d75283cb nodeName:}" failed. No retries permitted until 2026-11-03 13:08:17.137530036 +0000 UTC m=+32.111852654 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/898bd0b5-0d3e-4577-b101-2cc7d75283cb-service-ca-bundle") pod "router-default-7f579888d9-ntt6m" (UID: "898bd0b5-0d3e-4577-b101-2cc7d75283cb") : configmap references non-existent config key: service-ca.crt Nov 03 13:08:16.137693 ip-10-0-131-36 kubenswrapper[2573]: E1103 13:08:16.137603 2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 13:08:16.137693 ip-10-0-131-36 kubenswrapper[2573]: E1103 13:08:16.137625 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3fdeb9f3-6c67-4390-8073-17d4d5f802b9-metrics-tls podName:3fdeb9f3-6c67-4390-8073-17d4d5f802b9 nodeName:}" failed. No retries permitted until 2026-11-03 13:08:17.13761844 +0000 UTC m=+32.111941058 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3fdeb9f3-6c67-4390-8073-17d4d5f802b9-metrics-tls") pod "dns-default-vgnv8" (UID: "3fdeb9f3-6c67-4390-8073-17d4d5f802b9") : secret "dns-default-metrics-tls" not found Nov 03 13:08:16.137693 ip-10-0-131-36 kubenswrapper[2573]: E1103 13:08:16.137657 2573 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 03 13:08:16.137693 ip-10-0-131-36 kubenswrapper[2573]: E1103 13:08:16.137674 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/898bd0b5-0d3e-4577-b101-2cc7d75283cb-metrics-certs podName:898bd0b5-0d3e-4577-b101-2cc7d75283cb nodeName:}" failed. No retries permitted until 2026-11-03 13:08:17.137668776 +0000 UTC m=+32.111991395 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/898bd0b5-0d3e-4577-b101-2cc7d75283cb-metrics-certs") pod "router-default-7f579888d9-ntt6m" (UID: "898bd0b5-0d3e-4577-b101-2cc7d75283cb") : secret "router-metrics-certs-default" not found Nov 03 13:08:16.137809 ip-10-0-131-36 kubenswrapper[2573]: E1103 13:08:16.137711 2573 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 13:08:16.137809 ip-10-0-131-36 kubenswrapper[2573]: E1103 13:08:16.137718 2573 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7dbb67bd57-74qfj: secret "image-registry-tls" not found Nov 03 13:08:16.137809 ip-10-0-131-36 kubenswrapper[2573]: E1103 13:08:16.137737 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0a892eb3-ecc4-4312-9f4d-73eeba6b6eee-registry-tls podName:0a892eb3-ecc4-4312-9f4d-73eeba6b6eee nodeName:}" failed. No retries permitted until 2026-11-03 13:08:17.13773179 +0000 UTC m=+32.112054409 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/0a892eb3-ecc4-4312-9f4d-73eeba6b6eee-registry-tls") pod "image-registry-7dbb67bd57-74qfj" (UID: "0a892eb3-ecc4-4312-9f4d-73eeba6b6eee") : secret "image-registry-tls" not found Nov 03 13:08:16.137809 ip-10-0-131-36 kubenswrapper[2573]: E1103 13:08:16.137770 2573 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 03 13:08:16.137809 ip-10-0-131-36 kubenswrapper[2573]: E1103 13:08:16.137787 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/21618783-0a4e-4b1f-8041-cabdf69f5bea-samples-operator-tls podName:21618783-0a4e-4b1f-8041-cabdf69f5bea nodeName:}" failed. No retries permitted until 2026-11-03 13:08:17.137782208 +0000 UTC m=+32.112104827 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/21618783-0a4e-4b1f-8041-cabdf69f5bea-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-h46gl" (UID: "21618783-0a4e-4b1f-8041-cabdf69f5bea") : secret "samples-operator-tls" not found Nov 03 13:08:16.624236 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:16.624203 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8rdgq" Nov 03 13:08:16.624416 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:16.624241 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5jz7t" Nov 03 13:08:16.624416 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:16.624317 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f7hmh" Nov 03 13:08:16.627276 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:16.627253 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 03 13:08:16.627390 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:16.627339 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-d95tp\"" Nov 03 13:08:16.627390 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:16.627354 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-569wf\"" Nov 03 13:08:16.627466 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:16.627455 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-hltl5\"" Nov 03 13:08:16.627725 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:16.627711 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 03 13:08:16.883741 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:16.883650 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-6nz8d" event={"ID":"6cb478ab-97aa-4881-87b2-b45293cc2349","Type":"ContainerStarted","Data":"1658c9a5bbf6b90cd5b0c2c5ef8bc57b57b5e38c2ebde6e79097b6a75ad221c5"} Nov 03 13:08:16.885745 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:16.885707 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-hslrf" event={"ID":"b8fe3220-d285-487f-9fc0-597438982de7","Type":"ContainerStarted","Data":"31a758fec935ad6f712a02f2634ab245ae3792d016c2167c7031cfa4f8f0913f"} Nov 03 13:08:16.885870 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:16.885758 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-hslrf" event={"ID":"b8fe3220-d285-487f-9fc0-597438982de7","Type":"ContainerStarted","Data":"68fe3802481c8f7a3638809cd85a097698ee57db88bc75d6ddd15414cd29f9b5"} Nov 03 13:08:16.887705 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:16.887677 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-vl8fr" event={"ID":"07a7e2de-0c09-4819-9670-2be9460a155d","Type":"ContainerStarted","Data":"28808674f682829164148b6da532a44fde9871b00a6acce19ee222d3330879a6"} Nov 03 13:08:16.887840 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:16.887721 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-vl8fr" event={"ID":"07a7e2de-0c09-4819-9670-2be9460a155d","Type":"ContainerStarted","Data":"12c34fc486ae00af6cdac52a0c622b9549f772c0e690148eea53070e65b1da62"} Nov 03 13:08:16.889024 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:16.889001 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-p6cbj" event={"ID":"021e9e00-a973-4726-8f48-02a52c4bae14","Type":"ContainerStarted","Data":"84fabe8a79acb6040874b2b7e92579078c1dff682320fea274fd83a4f027c1ea"} Nov 03 13:08:16.890624 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:16.890602 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-jws95" event={"ID":"5c17f5bf-5b7d-4b05-a387-21ac6b5186ca","Type":"ContainerStarted","Data":"ab29b318c08555358cf838590f8909d568ac39d5a7ad9c2cbd6830c22a1c9f52"} Nov 03 13:08:16.892070 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:16.892043 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-d8f6l" event={"ID":"50d68cb7-b766-4674-b295-8f3229880994","Type":"ContainerStarted","Data":"3cf1dc4f654fe1eddf341ba01ad4d52e81aac0f2f4a81246a19b59883c62f85a"} Nov 03 13:08:16.893592 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:16.893570 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-4t8lj" event={"ID":"9fa6d32d-b30b-4b76-9462-5f3f09443d02","Type":"ContainerStarted","Data":"5ea2ec30de3f7897100ae5923cb74d11ffc40a71885605a5855258979863426c"} Nov 03 13:08:16.895348 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:16.895323 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-6htht" event={"ID":"cd91de08-084e-4667-b13f-2e07b3cff624","Type":"ContainerStarted","Data":"a8456e348c89166330071974486e35de81374e53475426213db288f247638ed1"} Nov 03 13:08:16.897188 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:16.897161 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-w5cql" event={"ID":"c82d9f4b-8c00-4c31-8aa2-99cdc0d87e52","Type":"ContainerStarted","Data":"b42e779c3f878b31927b9a6ae998406a183144ea97eb87574d067f66fcef6e26"} Nov 03 13:08:16.897300 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:16.897203 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-w5cql" event={"ID":"c82d9f4b-8c00-4c31-8aa2-99cdc0d87e52","Type":"ContainerStarted","Data":"a670effa2a69f2c5a5f3625db93d19af7b21e7237cd4c41a16088e66cb4104a2"} Nov 03 13:08:16.899193 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:16.899170 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-cqvdv" event={"ID":"9f20ddda-41fb-4aba-b257-7ebc59dbe678","Type":"ContainerStarted","Data":"b60ab4a7af01474429ab8f251670ff485fabd1f7d74bcd699a42190bfa711191"} Nov 03 13:08:16.922463 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:16.922394 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-hslrf" podStartSLOduration=427.922375643 podStartE2EDuration="7m7.922375643s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:01:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:08:16.902734006 +0000 UTC m=+31.877056648" watchObservedRunningTime="2026-11-03 13:08:16.922375643 +0000 UTC m=+31.896698288" Nov 03 13:08:16.936439 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:16.936380 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-w5cql" podStartSLOduration=427.936363359 podStartE2EDuration="7m7.936363359s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:01:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:08:16.922565275 +0000 UTC m=+31.896887913" watchObservedRunningTime="2026-11-03 13:08:16.936363359 +0000 UTC m=+31.910686000" Nov 03 13:08:16.936969 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:16.936930 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-vl8fr" podStartSLOduration=427.936920354 podStartE2EDuration="7m7.936920354s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:01:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:08:16.936531473 +0000 UTC m=+31.910854115" watchObservedRunningTime="2026-11-03 13:08:16.936920354 +0000 UTC m=+31.911242994" Nov 03 13:08:17.047023 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:17.046981 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/fa297a30-6da7-4a92-98fb-201323490bb0-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-rz4n9\" (UID: \"fa297a30-6da7-4a92-98fb-201323490bb0\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-rz4n9" Nov 03 13:08:17.047187 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:17.047038 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a3042b60-062f-4f9b-8406-f884c95a8729-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-tm4nb\" (UID: \"a3042b60-062f-4f9b-8406-f884c95a8729\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-tm4nb" Nov 03 13:08:17.047187 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:17.047061 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cc8ef30e-6d4f-4913-8dc8-b09e5aea2f82-registry-tls\") pod \"image-registry-555f8976d-mz5ct\" (UID: \"cc8ef30e-6d4f-4913-8dc8-b09e5aea2f82\") " pod="openshift-image-registry/image-registry-555f8976d-mz5ct" Nov 03 13:08:17.047187 ip-10-0-131-36 kubenswrapper[2573]: E1103 13:08:17.047148 2573 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 03 13:08:17.047356 ip-10-0-131-36 kubenswrapper[2573]: E1103 13:08:17.047209 2573 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 13:08:17.047356 ip-10-0-131-36 kubenswrapper[2573]: E1103 13:08:17.047223 2573 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-555f8976d-mz5ct: secret "image-registry-tls" not found Nov 03 13:08:17.047356 ip-10-0-131-36 kubenswrapper[2573]: E1103 13:08:17.047227 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a3042b60-062f-4f9b-8406-f884c95a8729-cluster-monitoring-operator-tls podName:a3042b60-062f-4f9b-8406-f884c95a8729 nodeName:}" failed. No retries permitted until 2026-11-03 13:08:19.047206883 +0000 UTC m=+34.021529533 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/a3042b60-062f-4f9b-8406-f884c95a8729-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-tm4nb" (UID: "a3042b60-062f-4f9b-8406-f884c95a8729") : secret "cluster-monitoring-operator-tls" not found Nov 03 13:08:17.047356 ip-10-0-131-36 kubenswrapper[2573]: E1103 13:08:17.047148 2573 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 03 13:08:17.047356 ip-10-0-131-36 kubenswrapper[2573]: E1103 13:08:17.047257 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cc8ef30e-6d4f-4913-8dc8-b09e5aea2f82-registry-tls podName:cc8ef30e-6d4f-4913-8dc8-b09e5aea2f82 nodeName:}" failed. No retries permitted until 2026-11-03 13:08:19.047246019 +0000 UTC m=+34.021568657 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/cc8ef30e-6d4f-4913-8dc8-b09e5aea2f82-registry-tls") pod "image-registry-555f8976d-mz5ct" (UID: "cc8ef30e-6d4f-4913-8dc8-b09e5aea2f82") : secret "image-registry-tls" not found Nov 03 13:08:17.047356 ip-10-0-131-36 kubenswrapper[2573]: E1103 13:08:17.047275 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fa297a30-6da7-4a92-98fb-201323490bb0-networking-console-plugin-cert podName:fa297a30-6da7-4a92-98fb-201323490bb0 nodeName:}" failed. No retries permitted until 2026-11-03 13:08:19.047266253 +0000 UTC m=+34.021588875 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/fa297a30-6da7-4a92-98fb-201323490bb0-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-rz4n9" (UID: "fa297a30-6da7-4a92-98fb-201323490bb0") : secret "networking-console-plugin-cert" not found Nov 03 13:08:17.148618 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:17.148527 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/21618783-0a4e-4b1f-8041-cabdf69f5bea-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-h46gl\" (UID: \"21618783-0a4e-4b1f-8041-cabdf69f5bea\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-h46gl" Nov 03 13:08:17.148618 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:17.148579 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/898bd0b5-0d3e-4577-b101-2cc7d75283cb-service-ca-bundle\") pod \"router-default-7f579888d9-ntt6m\" (UID: \"898bd0b5-0d3e-4577-b101-2cc7d75283cb\") " pod="openshift-ingress/router-default-7f579888d9-ntt6m" Nov 03 13:08:17.148922 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:17.148625 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3fdeb9f3-6c67-4390-8073-17d4d5f802b9-metrics-tls\") pod \"dns-default-vgnv8\" (UID: \"3fdeb9f3-6c67-4390-8073-17d4d5f802b9\") " pod="openshift-dns/dns-default-vgnv8" Nov 03 13:08:17.148922 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:17.148660 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/898bd0b5-0d3e-4577-b101-2cc7d75283cb-metrics-certs\") pod \"router-default-7f579888d9-ntt6m\" (UID: \"898bd0b5-0d3e-4577-b101-2cc7d75283cb\") " pod="openshift-ingress/router-default-7f579888d9-ntt6m" Nov 03 13:08:17.148922 ip-10-0-131-36 kubenswrapper[2573]: E1103 13:08:17.148696 2573 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 03 13:08:17.148922 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:17.148720 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0a892eb3-ecc4-4312-9f4d-73eeba6b6eee-registry-tls\") pod \"image-registry-7dbb67bd57-74qfj\" (UID: \"0a892eb3-ecc4-4312-9f4d-73eeba6b6eee\") " pod="openshift-image-registry/image-registry-7dbb67bd57-74qfj" Nov 03 13:08:17.148922 ip-10-0-131-36 kubenswrapper[2573]: E1103 13:08:17.148767 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/21618783-0a4e-4b1f-8041-cabdf69f5bea-samples-operator-tls podName:21618783-0a4e-4b1f-8041-cabdf69f5bea nodeName:}" failed. No retries permitted until 2026-11-03 13:08:19.148747543 +0000 UTC m=+34.123070179 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/21618783-0a4e-4b1f-8041-cabdf69f5bea-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-h46gl" (UID: "21618783-0a4e-4b1f-8041-cabdf69f5bea") : secret "samples-operator-tls" not found Nov 03 13:08:17.148922 ip-10-0-131-36 kubenswrapper[2573]: E1103 13:08:17.148816 2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 13:08:17.148922 ip-10-0-131-36 kubenswrapper[2573]: E1103 13:08:17.148829 2573 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 03 13:08:17.148922 ip-10-0-131-36 kubenswrapper[2573]: E1103 13:08:17.148833 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/898bd0b5-0d3e-4577-b101-2cc7d75283cb-service-ca-bundle podName:898bd0b5-0d3e-4577-b101-2cc7d75283cb nodeName:}" failed. No retries permitted until 2026-11-03 13:08:19.148821637 +0000 UTC m=+34.123144261 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/898bd0b5-0d3e-4577-b101-2cc7d75283cb-service-ca-bundle") pod "router-default-7f579888d9-ntt6m" (UID: "898bd0b5-0d3e-4577-b101-2cc7d75283cb") : configmap references non-existent config key: service-ca.crt Nov 03 13:08:17.148922 ip-10-0-131-36 kubenswrapper[2573]: E1103 13:08:17.148898 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3fdeb9f3-6c67-4390-8073-17d4d5f802b9-metrics-tls podName:3fdeb9f3-6c67-4390-8073-17d4d5f802b9 nodeName:}" failed. No retries permitted until 2026-11-03 13:08:19.14888138 +0000 UTC m=+34.123204007 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3fdeb9f3-6c67-4390-8073-17d4d5f802b9-metrics-tls") pod "dns-default-vgnv8" (UID: "3fdeb9f3-6c67-4390-8073-17d4d5f802b9") : secret "dns-default-metrics-tls" not found Nov 03 13:08:17.148922 ip-10-0-131-36 kubenswrapper[2573]: E1103 13:08:17.148900 2573 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 13:08:17.148922 ip-10-0-131-36 kubenswrapper[2573]: E1103 13:08:17.148912 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/898bd0b5-0d3e-4577-b101-2cc7d75283cb-metrics-certs podName:898bd0b5-0d3e-4577-b101-2cc7d75283cb nodeName:}" failed. No retries permitted until 2026-11-03 13:08:19.148903682 +0000 UTC m=+34.123226308 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/898bd0b5-0d3e-4577-b101-2cc7d75283cb-metrics-certs") pod "router-default-7f579888d9-ntt6m" (UID: "898bd0b5-0d3e-4577-b101-2cc7d75283cb") : secret "router-metrics-certs-default" not found Nov 03 13:08:17.148922 ip-10-0-131-36 kubenswrapper[2573]: E1103 13:08:17.148915 2573 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7dbb67bd57-74qfj: secret "image-registry-tls" not found Nov 03 13:08:17.149503 ip-10-0-131-36 kubenswrapper[2573]: E1103 13:08:17.148956 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0a892eb3-ecc4-4312-9f4d-73eeba6b6eee-registry-tls podName:0a892eb3-ecc4-4312-9f4d-73eeba6b6eee nodeName:}" failed. No retries permitted until 2026-11-03 13:08:19.148944949 +0000 UTC m=+34.123267584 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/0a892eb3-ecc4-4312-9f4d-73eeba6b6eee-registry-tls") pod "image-registry-7dbb67bd57-74qfj" (UID: "0a892eb3-ecc4-4312-9f4d-73eeba6b6eee") : secret "image-registry-tls" not found Nov 03 13:08:18.907498 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:18.907456 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-p6cbj" event={"ID":"021e9e00-a973-4726-8f48-02a52c4bae14","Type":"ContainerStarted","Data":"31fab1bd4f287a97d8b723e1286d96783a85e56c719baa95940a9c993da4d139"} Nov 03 13:08:18.923368 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:18.923304 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-p6cbj" podStartSLOduration=377.136846272 podStartE2EDuration="6m19.923268777s" totalImagesPullingTime="2.786422505s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:01:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:08:18.921615449 +0000 UTC m=+33.895938091" watchObservedRunningTime="2026-11-03 13:08:18.923268777 +0000 UTC m=+33.897591420" Nov 03 13:08:19.071663 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:19.071556 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/fa297a30-6da7-4a92-98fb-201323490bb0-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-rz4n9\" (UID: \"fa297a30-6da7-4a92-98fb-201323490bb0\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-rz4n9" Nov 03 13:08:19.071663 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:19.071635 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a3042b60-062f-4f9b-8406-f884c95a8729-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-tm4nb\" (UID: \"a3042b60-062f-4f9b-8406-f884c95a8729\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-tm4nb" Nov 03 13:08:19.071663 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:19.071660 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cc8ef30e-6d4f-4913-8dc8-b09e5aea2f82-registry-tls\") pod \"image-registry-555f8976d-mz5ct\" (UID: \"cc8ef30e-6d4f-4913-8dc8-b09e5aea2f82\") " pod="openshift-image-registry/image-registry-555f8976d-mz5ct" Nov 03 13:08:19.071858 ip-10-0-131-36 kubenswrapper[2573]: E1103 13:08:19.071739 2573 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 03 13:08:19.071858 ip-10-0-131-36 kubenswrapper[2573]: E1103 13:08:19.071781 2573 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 13:08:19.071858 ip-10-0-131-36 kubenswrapper[2573]: E1103 13:08:19.071793 2573 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-555f8976d-mz5ct: secret "image-registry-tls" not found Nov 03 13:08:19.071858 ip-10-0-131-36 kubenswrapper[2573]: E1103 13:08:19.071805 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fa297a30-6da7-4a92-98fb-201323490bb0-networking-console-plugin-cert podName:fa297a30-6da7-4a92-98fb-201323490bb0 nodeName:}" failed. No retries permitted until 2026-11-03 13:08:23.071787053 +0000 UTC m=+38.046109692 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/fa297a30-6da7-4a92-98fb-201323490bb0-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-rz4n9" (UID: "fa297a30-6da7-4a92-98fb-201323490bb0") : secret "networking-console-plugin-cert" not found Nov 03 13:08:19.071858 ip-10-0-131-36 kubenswrapper[2573]: E1103 13:08:19.071739 2573 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 03 13:08:19.072002 ip-10-0-131-36 kubenswrapper[2573]: E1103 13:08:19.071825 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cc8ef30e-6d4f-4913-8dc8-b09e5aea2f82-registry-tls podName:cc8ef30e-6d4f-4913-8dc8-b09e5aea2f82 nodeName:}" failed. No retries permitted until 2026-11-03 13:08:23.071815019 +0000 UTC m=+38.046137651 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/cc8ef30e-6d4f-4913-8dc8-b09e5aea2f82-registry-tls") pod "image-registry-555f8976d-mz5ct" (UID: "cc8ef30e-6d4f-4913-8dc8-b09e5aea2f82") : secret "image-registry-tls" not found Nov 03 13:08:19.072002 ip-10-0-131-36 kubenswrapper[2573]: E1103 13:08:19.071913 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a3042b60-062f-4f9b-8406-f884c95a8729-cluster-monitoring-operator-tls podName:a3042b60-062f-4f9b-8406-f884c95a8729 nodeName:}" failed. No retries permitted until 2026-11-03 13:08:23.071890425 +0000 UTC m=+38.046213051 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/a3042b60-062f-4f9b-8406-f884c95a8729-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-tm4nb" (UID: "a3042b60-062f-4f9b-8406-f884c95a8729") : secret "cluster-monitoring-operator-tls" not found Nov 03 13:08:19.172891 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:19.172853 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/21618783-0a4e-4b1f-8041-cabdf69f5bea-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-h46gl\" (UID: \"21618783-0a4e-4b1f-8041-cabdf69f5bea\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-h46gl" Nov 03 13:08:19.173087 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:19.172898 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/898bd0b5-0d3e-4577-b101-2cc7d75283cb-service-ca-bundle\") pod \"router-default-7f579888d9-ntt6m\" (UID: \"898bd0b5-0d3e-4577-b101-2cc7d75283cb\") " pod="openshift-ingress/router-default-7f579888d9-ntt6m" Nov 03 13:08:19.173087 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:19.172941 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3fdeb9f3-6c67-4390-8073-17d4d5f802b9-metrics-tls\") pod \"dns-default-vgnv8\" (UID: \"3fdeb9f3-6c67-4390-8073-17d4d5f802b9\") " pod="openshift-dns/dns-default-vgnv8" Nov 03 13:08:19.173087 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:19.172975 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/898bd0b5-0d3e-4577-b101-2cc7d75283cb-metrics-certs\") pod \"router-default-7f579888d9-ntt6m\" (UID: \"898bd0b5-0d3e-4577-b101-2cc7d75283cb\") " pod="openshift-ingress/router-default-7f579888d9-ntt6m" Nov 03 13:08:19.173087 ip-10-0-131-36 kubenswrapper[2573]: E1103 13:08:19.173022 2573 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 03 13:08:19.173087 ip-10-0-131-36 kubenswrapper[2573]: E1103 13:08:19.173053 2573 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 03 13:08:19.173087 ip-10-0-131-36 kubenswrapper[2573]: E1103 13:08:19.173087 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/21618783-0a4e-4b1f-8041-cabdf69f5bea-samples-operator-tls podName:21618783-0a4e-4b1f-8041-cabdf69f5bea nodeName:}" failed. No retries permitted until 2026-11-03 13:08:23.173069245 +0000 UTC m=+38.147391887 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/21618783-0a4e-4b1f-8041-cabdf69f5bea-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-h46gl" (UID: "21618783-0a4e-4b1f-8041-cabdf69f5bea") : secret "samples-operator-tls" not found Nov 03 13:08:19.173338 ip-10-0-131-36 kubenswrapper[2573]: E1103 13:08:19.173096 2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 13:08:19.173338 ip-10-0-131-36 kubenswrapper[2573]: E1103 13:08:19.173104 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/898bd0b5-0d3e-4577-b101-2cc7d75283cb-metrics-certs podName:898bd0b5-0d3e-4577-b101-2cc7d75283cb nodeName:}" failed. No retries permitted until 2026-11-03 13:08:23.173096716 +0000 UTC m=+38.147419335 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/898bd0b5-0d3e-4577-b101-2cc7d75283cb-metrics-certs") pod "router-default-7f579888d9-ntt6m" (UID: "898bd0b5-0d3e-4577-b101-2cc7d75283cb") : secret "router-metrics-certs-default" not found Nov 03 13:08:19.173338 ip-10-0-131-36 kubenswrapper[2573]: E1103 13:08:19.173165 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3fdeb9f3-6c67-4390-8073-17d4d5f802b9-metrics-tls podName:3fdeb9f3-6c67-4390-8073-17d4d5f802b9 nodeName:}" failed. No retries permitted until 2026-11-03 13:08:23.173152732 +0000 UTC m=+38.147475351 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3fdeb9f3-6c67-4390-8073-17d4d5f802b9-metrics-tls") pod "dns-default-vgnv8" (UID: "3fdeb9f3-6c67-4390-8073-17d4d5f802b9") : secret "dns-default-metrics-tls" not found Nov 03 13:08:19.173338 ip-10-0-131-36 kubenswrapper[2573]: E1103 13:08:19.173175 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/898bd0b5-0d3e-4577-b101-2cc7d75283cb-service-ca-bundle podName:898bd0b5-0d3e-4577-b101-2cc7d75283cb nodeName:}" failed. No retries permitted until 2026-11-03 13:08:23.173169983 +0000 UTC m=+38.147492601 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/898bd0b5-0d3e-4577-b101-2cc7d75283cb-service-ca-bundle") pod "router-default-7f579888d9-ntt6m" (UID: "898bd0b5-0d3e-4577-b101-2cc7d75283cb") : configmap references non-existent config key: service-ca.crt Nov 03 13:08:19.173338 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:19.173222 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0a892eb3-ecc4-4312-9f4d-73eeba6b6eee-registry-tls\") pod \"image-registry-7dbb67bd57-74qfj\" (UID: \"0a892eb3-ecc4-4312-9f4d-73eeba6b6eee\") " pod="openshift-image-registry/image-registry-7dbb67bd57-74qfj" Nov 03 13:08:19.173338 ip-10-0-131-36 kubenswrapper[2573]: E1103 13:08:19.173319 2573 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 13:08:19.173338 ip-10-0-131-36 kubenswrapper[2573]: E1103 13:08:19.173328 2573 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7dbb67bd57-74qfj: secret "image-registry-tls" not found Nov 03 13:08:19.173596 ip-10-0-131-36 kubenswrapper[2573]: E1103 13:08:19.173355 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0a892eb3-ecc4-4312-9f4d-73eeba6b6eee-registry-tls podName:0a892eb3-ecc4-4312-9f4d-73eeba6b6eee nodeName:}" failed. No retries permitted until 2026-11-03 13:08:23.173348973 +0000 UTC m=+38.147671592 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/0a892eb3-ecc4-4312-9f4d-73eeba6b6eee-registry-tls") pod "image-registry-7dbb67bd57-74qfj" (UID: "0a892eb3-ecc4-4312-9f4d-73eeba6b6eee") : secret "image-registry-tls" not found Nov 03 13:08:19.274256 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:19.274210 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbfgb\" (UniqueName: \"kubernetes.io/projected/1bfc1afc-5b9f-4447-8d40-fabedbee9c07-kube-api-access-hbfgb\") pod \"network-check-target-8rdgq\" (UID: \"1bfc1afc-5b9f-4447-8d40-fabedbee9c07\") " pod="openshift-network-diagnostics/network-check-target-8rdgq" Nov 03 13:08:19.274441 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:19.274274 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7e280277-f29f-4a2b-8b85-2491529e5ef2-original-pull-secret\") pod \"global-pull-secret-syncer-5jz7t\" (UID: \"7e280277-f29f-4a2b-8b85-2491529e5ef2\") " pod="kube-system/global-pull-secret-syncer-5jz7t" Nov 03 13:08:19.274441 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:19.274351 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e0300f55-f002-4faa-bf6a-c04bea0a043e-metrics-certs\") pod \"network-metrics-daemon-f7hmh\" (UID: \"e0300f55-f002-4faa-bf6a-c04bea0a043e\") " pod="openshift-multus/network-metrics-daemon-f7hmh" Nov 03 13:08:19.274572 ip-10-0-131-36 kubenswrapper[2573]: E1103 13:08:19.274472 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 03 13:08:19.274572 ip-10-0-131-36 kubenswrapper[2573]: E1103 13:08:19.274538 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e0300f55-f002-4faa-bf6a-c04bea0a043e-metrics-certs podName:e0300f55-f002-4faa-bf6a-c04bea0a043e nodeName:}" failed. No retries permitted until 2026-11-03 13:08:51.274523228 +0000 UTC m=+66.248845851 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e0300f55-f002-4faa-bf6a-c04bea0a043e-metrics-certs") pod "network-metrics-daemon-f7hmh" (UID: "e0300f55-f002-4faa-bf6a-c04bea0a043e") : secret "metrics-daemon-secret" not found Nov 03 13:08:19.277212 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:19.277187 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7e280277-f29f-4a2b-8b85-2491529e5ef2-original-pull-secret\") pod \"global-pull-secret-syncer-5jz7t\" (UID: \"7e280277-f29f-4a2b-8b85-2491529e5ef2\") " pod="kube-system/global-pull-secret-syncer-5jz7t" Nov 03 13:08:19.277474 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:19.277456 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hbfgb\" (UniqueName: \"kubernetes.io/projected/1bfc1afc-5b9f-4447-8d40-fabedbee9c07-kube-api-access-hbfgb\") pod \"network-check-target-8rdgq\" (UID: \"1bfc1afc-5b9f-4447-8d40-fabedbee9c07\") " pod="openshift-network-diagnostics/network-check-target-8rdgq" Nov 03 13:08:19.337188 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:19.337113 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8rdgq" Nov 03 13:08:19.347060 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:19.347031 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5jz7t" Nov 03 13:08:19.629623 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:19.629596 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-5jz7t"] Nov 03 13:08:19.632844 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:19.632815 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-8rdgq"] Nov 03 13:08:19.633237 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:08:19.633210 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e280277_f29f_4a2b_8b85_2491529e5ef2.slice/crio-20af2de0975a5d8d3f6cee427d3f37295edf7f21d547d66bb589a540ecc17985 WatchSource:0}: Error finding container 20af2de0975a5d8d3f6cee427d3f37295edf7f21d547d66bb589a540ecc17985: Status 404 returned error can't find the container with id 20af2de0975a5d8d3f6cee427d3f37295edf7f21d547d66bb589a540ecc17985 Nov 03 13:08:19.636031 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:08:19.636005 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1bfc1afc_5b9f_4447_8d40_fabedbee9c07.slice/crio-48ba5bc216484088efd6cef6f8687c4da1e5abc695dc4f3df755ca6977ace6b0 WatchSource:0}: Error finding container 48ba5bc216484088efd6cef6f8687c4da1e5abc695dc4f3df755ca6977ace6b0: Status 404 returned error can't find the container with id 48ba5bc216484088efd6cef6f8687c4da1e5abc695dc4f3df755ca6977ace6b0 Nov 03 13:08:19.911157 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:19.911078 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-5jz7t" event={"ID":"7e280277-f29f-4a2b-8b85-2491529e5ef2","Type":"ContainerStarted","Data":"20af2de0975a5d8d3f6cee427d3f37295edf7f21d547d66bb589a540ecc17985"} Nov 03 13:08:19.912295 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:19.912259 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-8rdgq" event={"ID":"1bfc1afc-5b9f-4447-8d40-fabedbee9c07","Type":"ContainerStarted","Data":"48ba5bc216484088efd6cef6f8687c4da1e5abc695dc4f3df755ca6977ace6b0"} Nov 03 13:08:22.923142 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:22.923102 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-d8f6l" event={"ID":"50d68cb7-b766-4674-b295-8f3229880994","Type":"ContainerStarted","Data":"8a4680818899e686abed7f6b5e55f995bae6dbd2450ce3cc27dea636e13ef1b2"} Nov 03 13:08:22.939917 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:22.939866 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-d8f6l" podStartSLOduration=1.466181335 podStartE2EDuration="7.939851646s" totalImagesPullingTime="6.473670311s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:08:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:08:22.939377665 +0000 UTC m=+37.913700322" watchObservedRunningTime="2026-11-03 13:08:22.939851646 +0000 UTC m=+37.914174295" Nov 03 13:08:23.111250 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:23.111213 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/fa297a30-6da7-4a92-98fb-201323490bb0-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-rz4n9\" (UID: \"fa297a30-6da7-4a92-98fb-201323490bb0\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-rz4n9" Nov 03 13:08:23.111431 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:23.111259 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a3042b60-062f-4f9b-8406-f884c95a8729-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-tm4nb\" (UID: \"a3042b60-062f-4f9b-8406-f884c95a8729\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-tm4nb" Nov 03 13:08:23.111431 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:23.111278 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cc8ef30e-6d4f-4913-8dc8-b09e5aea2f82-registry-tls\") pod \"image-registry-555f8976d-mz5ct\" (UID: \"cc8ef30e-6d4f-4913-8dc8-b09e5aea2f82\") " pod="openshift-image-registry/image-registry-555f8976d-mz5ct" Nov 03 13:08:23.111431 ip-10-0-131-36 kubenswrapper[2573]: E1103 13:08:23.111377 2573 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 13:08:23.111431 ip-10-0-131-36 kubenswrapper[2573]: E1103 13:08:23.111378 2573 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 03 13:08:23.111604 ip-10-0-131-36 kubenswrapper[2573]: E1103 13:08:23.111456 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fa297a30-6da7-4a92-98fb-201323490bb0-networking-console-plugin-cert podName:fa297a30-6da7-4a92-98fb-201323490bb0 nodeName:}" failed. No retries permitted until 2026-11-03 13:08:31.111438187 +0000 UTC m=+46.085760819 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/fa297a30-6da7-4a92-98fb-201323490bb0-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-rz4n9" (UID: "fa297a30-6da7-4a92-98fb-201323490bb0") : secret "networking-console-plugin-cert" not found Nov 03 13:08:23.111604 ip-10-0-131-36 kubenswrapper[2573]: E1103 13:08:23.111377 2573 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 03 13:08:23.111604 ip-10-0-131-36 kubenswrapper[2573]: E1103 13:08:23.111495 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a3042b60-062f-4f9b-8406-f884c95a8729-cluster-monitoring-operator-tls podName:a3042b60-062f-4f9b-8406-f884c95a8729 nodeName:}" failed. No retries permitted until 2026-11-03 13:08:31.111486827 +0000 UTC m=+46.085809449 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/a3042b60-062f-4f9b-8406-f884c95a8729-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-tm4nb" (UID: "a3042b60-062f-4f9b-8406-f884c95a8729") : secret "cluster-monitoring-operator-tls" not found Nov 03 13:08:23.111604 ip-10-0-131-36 kubenswrapper[2573]: E1103 13:08:23.111388 2573 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-555f8976d-mz5ct: secret "image-registry-tls" not found Nov 03 13:08:23.111604 ip-10-0-131-36 kubenswrapper[2573]: E1103 13:08:23.111532 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cc8ef30e-6d4f-4913-8dc8-b09e5aea2f82-registry-tls podName:cc8ef30e-6d4f-4913-8dc8-b09e5aea2f82 nodeName:}" failed. No retries permitted until 2026-11-03 13:08:31.111523634 +0000 UTC m=+46.085846268 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/cc8ef30e-6d4f-4913-8dc8-b09e5aea2f82-registry-tls") pod "image-registry-555f8976d-mz5ct" (UID: "cc8ef30e-6d4f-4913-8dc8-b09e5aea2f82") : secret "image-registry-tls" not found Nov 03 13:08:23.212119 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:23.212026 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3fdeb9f3-6c67-4390-8073-17d4d5f802b9-metrics-tls\") pod \"dns-default-vgnv8\" (UID: \"3fdeb9f3-6c67-4390-8073-17d4d5f802b9\") " pod="openshift-dns/dns-default-vgnv8" Nov 03 13:08:23.212119 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:23.212096 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/898bd0b5-0d3e-4577-b101-2cc7d75283cb-metrics-certs\") pod \"router-default-7f579888d9-ntt6m\" (UID: \"898bd0b5-0d3e-4577-b101-2cc7d75283cb\") " pod="openshift-ingress/router-default-7f579888d9-ntt6m" Nov 03 13:08:23.212367 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:23.212158 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0a892eb3-ecc4-4312-9f4d-73eeba6b6eee-registry-tls\") pod \"image-registry-7dbb67bd57-74qfj\" (UID: \"0a892eb3-ecc4-4312-9f4d-73eeba6b6eee\") " pod="openshift-image-registry/image-registry-7dbb67bd57-74qfj" Nov 03 13:08:23.212367 ip-10-0-131-36 kubenswrapper[2573]: E1103 13:08:23.212210 2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 13:08:23.212367 ip-10-0-131-36 kubenswrapper[2573]: E1103 13:08:23.212249 2573 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 03 13:08:23.212367 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:23.212256 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/21618783-0a4e-4b1f-8041-cabdf69f5bea-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-h46gl\" (UID: \"21618783-0a4e-4b1f-8041-cabdf69f5bea\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-h46gl" Nov 03 13:08:23.212367 ip-10-0-131-36 kubenswrapper[2573]: E1103 13:08:23.212301 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3fdeb9f3-6c67-4390-8073-17d4d5f802b9-metrics-tls podName:3fdeb9f3-6c67-4390-8073-17d4d5f802b9 nodeName:}" failed. No retries permitted until 2026-11-03 13:08:31.212264269 +0000 UTC m=+46.186586899 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3fdeb9f3-6c67-4390-8073-17d4d5f802b9-metrics-tls") pod "dns-default-vgnv8" (UID: "3fdeb9f3-6c67-4390-8073-17d4d5f802b9") : secret "dns-default-metrics-tls" not found Nov 03 13:08:23.212367 ip-10-0-131-36 kubenswrapper[2573]: E1103 13:08:23.212354 2573 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 03 13:08:23.212367 ip-10-0-131-36 kubenswrapper[2573]: E1103 13:08:23.212366 2573 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 13:08:23.212621 ip-10-0-131-36 kubenswrapper[2573]: E1103 13:08:23.212380 2573 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7dbb67bd57-74qfj: secret "image-registry-tls" not found Nov 03 13:08:23.212621 ip-10-0-131-36 kubenswrapper[2573]: E1103 13:08:23.212405 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/898bd0b5-0d3e-4577-b101-2cc7d75283cb-metrics-certs podName:898bd0b5-0d3e-4577-b101-2cc7d75283cb nodeName:}" failed. No retries permitted until 2026-11-03 13:08:31.212389412 +0000 UTC m=+46.186712046 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/898bd0b5-0d3e-4577-b101-2cc7d75283cb-metrics-certs") pod "router-default-7f579888d9-ntt6m" (UID: "898bd0b5-0d3e-4577-b101-2cc7d75283cb") : secret "router-metrics-certs-default" not found Nov 03 13:08:23.212621 ip-10-0-131-36 kubenswrapper[2573]: E1103 13:08:23.212422 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/21618783-0a4e-4b1f-8041-cabdf69f5bea-samples-operator-tls podName:21618783-0a4e-4b1f-8041-cabdf69f5bea nodeName:}" failed. No retries permitted until 2026-11-03 13:08:31.212414409 +0000 UTC m=+46.186737028 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/21618783-0a4e-4b1f-8041-cabdf69f5bea-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-h46gl" (UID: "21618783-0a4e-4b1f-8041-cabdf69f5bea") : secret "samples-operator-tls" not found Nov 03 13:08:23.212621 ip-10-0-131-36 kubenswrapper[2573]: E1103 13:08:23.212448 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0a892eb3-ecc4-4312-9f4d-73eeba6b6eee-registry-tls podName:0a892eb3-ecc4-4312-9f4d-73eeba6b6eee nodeName:}" failed. No retries permitted until 2026-11-03 13:08:31.212439442 +0000 UTC m=+46.186762067 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/0a892eb3-ecc4-4312-9f4d-73eeba6b6eee-registry-tls") pod "image-registry-7dbb67bd57-74qfj" (UID: "0a892eb3-ecc4-4312-9f4d-73eeba6b6eee") : secret "image-registry-tls" not found Nov 03 13:08:23.212621 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:23.212487 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/898bd0b5-0d3e-4577-b101-2cc7d75283cb-service-ca-bundle\") pod \"router-default-7f579888d9-ntt6m\" (UID: \"898bd0b5-0d3e-4577-b101-2cc7d75283cb\") " pod="openshift-ingress/router-default-7f579888d9-ntt6m" Nov 03 13:08:23.212621 ip-10-0-131-36 kubenswrapper[2573]: E1103 13:08:23.212621 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/898bd0b5-0d3e-4577-b101-2cc7d75283cb-service-ca-bundle podName:898bd0b5-0d3e-4577-b101-2cc7d75283cb nodeName:}" failed. No retries permitted until 2026-11-03 13:08:31.212595427 +0000 UTC m=+46.186918048 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/898bd0b5-0d3e-4577-b101-2cc7d75283cb-service-ca-bundle") pod "router-default-7f579888d9-ntt6m" (UID: "898bd0b5-0d3e-4577-b101-2cc7d75283cb") : configmap references non-existent config key: service-ca.crt Nov 03 13:08:24.931698 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:24.931657 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-4t8lj" event={"ID":"9fa6d32d-b30b-4b76-9462-5f3f09443d02","Type":"ContainerStarted","Data":"61609a0eadc2e90859796d714a04559f7442b35166cf465ad0da1508ed8e60ec"} Nov 03 13:08:24.949888 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:24.949828 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-operator-7c4986d449-4t8lj" podStartSLOduration=373.445564908 podStartE2EDuration="6m21.949811481s" totalImagesPullingTime="8.504246573s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:02:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:08:24.949562567 +0000 UTC m=+39.923885210" watchObservedRunningTime="2026-11-03 13:08:24.949811481 +0000 UTC m=+39.924134123" Nov 03 13:08:25.282696 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:25.282628 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-d7mwz"] Nov 03 13:08:25.295629 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:25.295602 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-d7mwz"] Nov 03 13:08:25.295764 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:25.295741 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-d7mwz" Nov 03 13:08:25.302915 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:25.302886 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 03 13:08:25.303050 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:25.302956 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 03 13:08:25.303050 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:25.303008 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-s2gms\"" Nov 03 13:08:25.435566 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:25.435535 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fff35d1b-e7b2-4dd3-9824-8b64fad02506-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-d7mwz\" (UID: \"fff35d1b-e7b2-4dd3-9824-8b64fad02506\") " pod="openshift-insights/insights-runtime-extractor-d7mwz" Nov 03 13:08:25.435566 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:25.435569 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jssfx\" (UniqueName: \"kubernetes.io/projected/fff35d1b-e7b2-4dd3-9824-8b64fad02506-kube-api-access-jssfx\") pod \"insights-runtime-extractor-d7mwz\" (UID: \"fff35d1b-e7b2-4dd3-9824-8b64fad02506\") " pod="openshift-insights/insights-runtime-extractor-d7mwz" Nov 03 13:08:25.435750 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:25.435683 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fff35d1b-e7b2-4dd3-9824-8b64fad02506-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-d7mwz\" (UID: \"fff35d1b-e7b2-4dd3-9824-8b64fad02506\") " pod="openshift-insights/insights-runtime-extractor-d7mwz" Nov 03 13:08:25.435856 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:25.435832 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/fff35d1b-e7b2-4dd3-9824-8b64fad02506-data-volume\") pod \"insights-runtime-extractor-d7mwz\" (UID: \"fff35d1b-e7b2-4dd3-9824-8b64fad02506\") " pod="openshift-insights/insights-runtime-extractor-d7mwz" Nov 03 13:08:25.435890 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:25.435875 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/fff35d1b-e7b2-4dd3-9824-8b64fad02506-crio-socket\") pod \"insights-runtime-extractor-d7mwz\" (UID: \"fff35d1b-e7b2-4dd3-9824-8b64fad02506\") " pod="openshift-insights/insights-runtime-extractor-d7mwz" Nov 03 13:08:25.536547 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:25.536466 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fff35d1b-e7b2-4dd3-9824-8b64fad02506-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-d7mwz\" (UID: \"fff35d1b-e7b2-4dd3-9824-8b64fad02506\") " pod="openshift-insights/insights-runtime-extractor-d7mwz" Nov 03 13:08:25.536707 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:25.536600 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/fff35d1b-e7b2-4dd3-9824-8b64fad02506-data-volume\") pod \"insights-runtime-extractor-d7mwz\" (UID: \"fff35d1b-e7b2-4dd3-9824-8b64fad02506\") " pod="openshift-insights/insights-runtime-extractor-d7mwz" Nov 03 13:08:25.536707 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:25.536640 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/fff35d1b-e7b2-4dd3-9824-8b64fad02506-crio-socket\") pod \"insights-runtime-extractor-d7mwz\" (UID: \"fff35d1b-e7b2-4dd3-9824-8b64fad02506\") " pod="openshift-insights/insights-runtime-extractor-d7mwz" Nov 03 13:08:25.536707 ip-10-0-131-36 kubenswrapper[2573]: E1103 13:08:25.536646 2573 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 03 13:08:25.536707 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:25.536698 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fff35d1b-e7b2-4dd3-9824-8b64fad02506-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-d7mwz\" (UID: \"fff35d1b-e7b2-4dd3-9824-8b64fad02506\") " pod="openshift-insights/insights-runtime-extractor-d7mwz" Nov 03 13:08:25.536898 ip-10-0-131-36 kubenswrapper[2573]: E1103 13:08:25.536724 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fff35d1b-e7b2-4dd3-9824-8b64fad02506-insights-runtime-extractor-tls podName:fff35d1b-e7b2-4dd3-9824-8b64fad02506 nodeName:}" failed. No retries permitted until 2026-11-03 13:08:26.036703864 +0000 UTC m=+41.011026503 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/fff35d1b-e7b2-4dd3-9824-8b64fad02506-insights-runtime-extractor-tls") pod "insights-runtime-extractor-d7mwz" (UID: "fff35d1b-e7b2-4dd3-9824-8b64fad02506") : secret "insights-runtime-extractor-tls" not found Nov 03 13:08:25.536898 ip-10-0-131-36 kubenswrapper[2573]: E1103 13:08:25.536755 2573 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 03 13:08:25.536898 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:25.536759 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jssfx\" (UniqueName: \"kubernetes.io/projected/fff35d1b-e7b2-4dd3-9824-8b64fad02506-kube-api-access-jssfx\") pod \"insights-runtime-extractor-d7mwz\" (UID: \"fff35d1b-e7b2-4dd3-9824-8b64fad02506\") " pod="openshift-insights/insights-runtime-extractor-d7mwz" Nov 03 13:08:25.536898 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:25.536788 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/fff35d1b-e7b2-4dd3-9824-8b64fad02506-crio-socket\") pod \"insights-runtime-extractor-d7mwz\" (UID: \"fff35d1b-e7b2-4dd3-9824-8b64fad02506\") " pod="openshift-insights/insights-runtime-extractor-d7mwz" Nov 03 13:08:25.536898 ip-10-0-131-36 kubenswrapper[2573]: E1103 13:08:25.536806 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/fff35d1b-e7b2-4dd3-9824-8b64fad02506-kube-rbac-proxy-cm podName:fff35d1b-e7b2-4dd3-9824-8b64fad02506 nodeName:}" failed. No retries permitted until 2026-11-03 13:08:26.036790344 +0000 UTC m=+41.011112985 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/fff35d1b-e7b2-4dd3-9824-8b64fad02506-kube-rbac-proxy-cm") pod "insights-runtime-extractor-d7mwz" (UID: "fff35d1b-e7b2-4dd3-9824-8b64fad02506") : configmap "kube-rbac-proxy" not found Nov 03 13:08:25.537053 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:25.536933 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/fff35d1b-e7b2-4dd3-9824-8b64fad02506-data-volume\") pod \"insights-runtime-extractor-d7mwz\" (UID: \"fff35d1b-e7b2-4dd3-9824-8b64fad02506\") " pod="openshift-insights/insights-runtime-extractor-d7mwz" Nov 03 13:08:25.546336 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:25.546314 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jssfx\" (UniqueName: \"kubernetes.io/projected/fff35d1b-e7b2-4dd3-9824-8b64fad02506-kube-api-access-jssfx\") pod \"insights-runtime-extractor-d7mwz\" (UID: \"fff35d1b-e7b2-4dd3-9824-8b64fad02506\") " pod="openshift-insights/insights-runtime-extractor-d7mwz" Nov 03 13:08:25.938773 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:25.938728 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-jws95" event={"ID":"5c17f5bf-5b7d-4b05-a387-21ac6b5186ca","Type":"ContainerStarted","Data":"8586dde866eeb98675ee230e4c933255958792134ff19c143a6509ba43a622b8"} Nov 03 13:08:25.954043 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:25.953994 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-jws95" podStartSLOduration=372.550896155 podStartE2EDuration="6m21.953979712s" totalImagesPullingTime="9.403083557s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:02:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:08:25.953209069 +0000 UTC m=+40.927531712" watchObservedRunningTime="2026-11-03 13:08:25.953979712 +0000 UTC m=+40.928302349" Nov 03 13:08:26.043335 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:26.043270 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fff35d1b-e7b2-4dd3-9824-8b64fad02506-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-d7mwz\" (UID: \"fff35d1b-e7b2-4dd3-9824-8b64fad02506\") " pod="openshift-insights/insights-runtime-extractor-d7mwz" Nov 03 13:08:26.043512 ip-10-0-131-36 kubenswrapper[2573]: E1103 13:08:26.043393 2573 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 03 13:08:26.043512 ip-10-0-131-36 kubenswrapper[2573]: E1103 13:08:26.043475 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/fff35d1b-e7b2-4dd3-9824-8b64fad02506-kube-rbac-proxy-cm podName:fff35d1b-e7b2-4dd3-9824-8b64fad02506 nodeName:}" failed. No retries permitted until 2026-11-03 13:08:27.043453938 +0000 UTC m=+42.017776572 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/fff35d1b-e7b2-4dd3-9824-8b64fad02506-kube-rbac-proxy-cm") pod "insights-runtime-extractor-d7mwz" (UID: "fff35d1b-e7b2-4dd3-9824-8b64fad02506") : configmap "kube-rbac-proxy" not found Nov 03 13:08:26.043512 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:26.043503 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fff35d1b-e7b2-4dd3-9824-8b64fad02506-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-d7mwz\" (UID: \"fff35d1b-e7b2-4dd3-9824-8b64fad02506\") " pod="openshift-insights/insights-runtime-extractor-d7mwz" Nov 03 13:08:26.043727 ip-10-0-131-36 kubenswrapper[2573]: E1103 13:08:26.043714 2573 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 03 13:08:26.043779 ip-10-0-131-36 kubenswrapper[2573]: E1103 13:08:26.043754 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fff35d1b-e7b2-4dd3-9824-8b64fad02506-insights-runtime-extractor-tls podName:fff35d1b-e7b2-4dd3-9824-8b64fad02506 nodeName:}" failed. No retries permitted until 2026-11-03 13:08:27.043742875 +0000 UTC m=+42.018065508 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/fff35d1b-e7b2-4dd3-9824-8b64fad02506-insights-runtime-extractor-tls") pod "insights-runtime-extractor-d7mwz" (UID: "fff35d1b-e7b2-4dd3-9824-8b64fad02506") : secret "insights-runtime-extractor-tls" not found Nov 03 13:08:27.052776 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:27.052747 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fff35d1b-e7b2-4dd3-9824-8b64fad02506-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-d7mwz\" (UID: \"fff35d1b-e7b2-4dd3-9824-8b64fad02506\") " pod="openshift-insights/insights-runtime-extractor-d7mwz" Nov 03 13:08:27.053229 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:27.052811 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fff35d1b-e7b2-4dd3-9824-8b64fad02506-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-d7mwz\" (UID: \"fff35d1b-e7b2-4dd3-9824-8b64fad02506\") " pod="openshift-insights/insights-runtime-extractor-d7mwz" Nov 03 13:08:27.053229 ip-10-0-131-36 kubenswrapper[2573]: E1103 13:08:27.052903 2573 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 03 13:08:27.053229 ip-10-0-131-36 kubenswrapper[2573]: E1103 13:08:27.053002 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fff35d1b-e7b2-4dd3-9824-8b64fad02506-insights-runtime-extractor-tls podName:fff35d1b-e7b2-4dd3-9824-8b64fad02506 nodeName:}" failed. No retries permitted until 2026-11-03 13:08:29.052979503 +0000 UTC m=+44.027302140 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/fff35d1b-e7b2-4dd3-9824-8b64fad02506-insights-runtime-extractor-tls") pod "insights-runtime-extractor-d7mwz" (UID: "fff35d1b-e7b2-4dd3-9824-8b64fad02506") : secret "insights-runtime-extractor-tls" not found Nov 03 13:08:27.053444 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:27.053423 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fff35d1b-e7b2-4dd3-9824-8b64fad02506-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-d7mwz\" (UID: \"fff35d1b-e7b2-4dd3-9824-8b64fad02506\") " pod="openshift-insights/insights-runtime-extractor-d7mwz" Nov 03 13:08:28.952775 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:28.952732 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-8rdgq" event={"ID":"1bfc1afc-5b9f-4447-8d40-fabedbee9c07","Type":"ContainerStarted","Data":"36dd83b87d089f5b54f5c6b67084d478941deef6dd82384b21689d9db5dbcd2e"} Nov 03 13:08:28.953222 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:28.952832 2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-8rdgq" Nov 03 13:08:28.968837 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:28.968785 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-8rdgq" podStartSLOduration=34.853840665 podStartE2EDuration="43.968766818s" totalImagesPullingTime="9.114926153s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:07:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:08:28.966864361 +0000 UTC m=+43.941186993" watchObservedRunningTime="2026-11-03 13:08:28.968766818 +0000 UTC m=+43.943089505" Nov 03 13:08:29.072921 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:29.072837 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fff35d1b-e7b2-4dd3-9824-8b64fad02506-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-d7mwz\" (UID: \"fff35d1b-e7b2-4dd3-9824-8b64fad02506\") " pod="openshift-insights/insights-runtime-extractor-d7mwz" Nov 03 13:08:29.073059 ip-10-0-131-36 kubenswrapper[2573]: E1103 13:08:29.072999 2573 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 03 13:08:29.073105 ip-10-0-131-36 kubenswrapper[2573]: E1103 13:08:29.073062 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fff35d1b-e7b2-4dd3-9824-8b64fad02506-insights-runtime-extractor-tls podName:fff35d1b-e7b2-4dd3-9824-8b64fad02506 nodeName:}" failed. No retries permitted until 2026-11-03 13:08:33.073047509 +0000 UTC m=+48.047370129 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/fff35d1b-e7b2-4dd3-9824-8b64fad02506-insights-runtime-extractor-tls") pod "insights-runtime-extractor-d7mwz" (UID: "fff35d1b-e7b2-4dd3-9824-8b64fad02506") : secret "insights-runtime-extractor-tls" not found Nov 03 13:08:29.960469 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:29.959566 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6htht_cd91de08-084e-4667-b13f-2e07b3cff624/console-operator/0.log" Nov 03 13:08:29.960469 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:29.959615 2573 generic.go:356] "Generic (PLEG): container finished" podID="cd91de08-084e-4667-b13f-2e07b3cff624" containerID="2c9953b166a72c36f8e6c090bb6d7b837cebcc425ad0f43bdfc37926af143a99" exitCode=255 Nov 03 13:08:29.960469 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:29.959991 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-6htht" event={"ID":"cd91de08-084e-4667-b13f-2e07b3cff624","Type":"ContainerDied","Data":"2c9953b166a72c36f8e6c090bb6d7b837cebcc425ad0f43bdfc37926af143a99"} Nov 03 13:08:29.960867 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:29.960612 2573 scope.go:122] "RemoveContainer" containerID="2c9953b166a72c36f8e6c090bb6d7b837cebcc425ad0f43bdfc37926af143a99" Nov 03 13:08:30.785201 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:30.785136 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-v8gcq_5ea5cf61-ed1c-4f5e-b1a0-5c46c5c3ee18/dns-node-resolver/0.log" Nov 03 13:08:30.965565 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:30.965522 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-6nz8d" event={"ID":"6cb478ab-97aa-4881-87b2-b45293cc2349","Type":"ContainerStarted","Data":"402758d12c8d4eaafa8121774c00f0b07ba63f8606dc2f3933548434390c58a4"} Nov 03 13:08:30.967156 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:30.967137 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6htht_cd91de08-084e-4667-b13f-2e07b3cff624/console-operator/1.log" Nov 03 13:08:30.967615 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:30.967596 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6htht_cd91de08-084e-4667-b13f-2e07b3cff624/console-operator/0.log" Nov 03 13:08:30.967778 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:30.967634 2573 generic.go:356] "Generic (PLEG): container finished" podID="cd91de08-084e-4667-b13f-2e07b3cff624" containerID="b811aececd178145fbcac8b298ed317468857bb459f9492b4b679c31b566f969" exitCode=255 Nov 03 13:08:30.967778 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:30.967675 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-6htht" event={"ID":"cd91de08-084e-4667-b13f-2e07b3cff624","Type":"ContainerDied","Data":"b811aececd178145fbcac8b298ed317468857bb459f9492b4b679c31b566f969"} Nov 03 13:08:30.967778 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:30.967719 2573 scope.go:122] "RemoveContainer" containerID="2c9953b166a72c36f8e6c090bb6d7b837cebcc425ad0f43bdfc37926af143a99" Nov 03 13:08:30.968009 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:30.967990 2573 scope.go:122] "RemoveContainer" containerID="b811aececd178145fbcac8b298ed317468857bb459f9492b4b679c31b566f969" Nov 03 13:08:30.968277 ip-10-0-131-36 kubenswrapper[2573]: E1103 13:08:30.968255 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-6htht_openshift-console-operator(cd91de08-084e-4667-b13f-2e07b3cff624)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-6htht" podUID="cd91de08-084e-4667-b13f-2e07b3cff624" Nov 03 13:08:30.981513 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:30.981459 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-6nz8d" podStartSLOduration=375.44069135 podStartE2EDuration="6m29.981439052s" totalImagesPullingTime="14.540747702s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:02:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:08:30.979971583 +0000 UTC m=+45.954294224" watchObservedRunningTime="2026-11-03 13:08:30.981439052 +0000 UTC m=+45.955761694" Nov 03 13:08:31.192219 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:31.192183 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/fa297a30-6da7-4a92-98fb-201323490bb0-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-rz4n9\" (UID: \"fa297a30-6da7-4a92-98fb-201323490bb0\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-rz4n9" Nov 03 13:08:31.192402 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:31.192228 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a3042b60-062f-4f9b-8406-f884c95a8729-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-tm4nb\" (UID: \"a3042b60-062f-4f9b-8406-f884c95a8729\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-tm4nb" Nov 03 13:08:31.192402 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:31.192244 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cc8ef30e-6d4f-4913-8dc8-b09e5aea2f82-registry-tls\") pod \"image-registry-555f8976d-mz5ct\" (UID: \"cc8ef30e-6d4f-4913-8dc8-b09e5aea2f82\") " pod="openshift-image-registry/image-registry-555f8976d-mz5ct" Nov 03 13:08:31.192402 ip-10-0-131-36 kubenswrapper[2573]: E1103 13:08:31.192369 2573 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 03 13:08:31.192582 ip-10-0-131-36 kubenswrapper[2573]: E1103 13:08:31.192418 2573 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 13:08:31.192582 ip-10-0-131-36 kubenswrapper[2573]: E1103 13:08:31.192430 2573 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-555f8976d-mz5ct: secret "image-registry-tls" not found Nov 03 13:08:31.192582 ip-10-0-131-36 kubenswrapper[2573]: E1103 13:08:31.192434 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a3042b60-062f-4f9b-8406-f884c95a8729-cluster-monitoring-operator-tls podName:a3042b60-062f-4f9b-8406-f884c95a8729 nodeName:}" failed. No retries permitted until 2026-11-03 13:08:47.192420422 +0000 UTC m=+62.166743078 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/a3042b60-062f-4f9b-8406-f884c95a8729-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-tm4nb" (UID: "a3042b60-062f-4f9b-8406-f884c95a8729") : secret "cluster-monitoring-operator-tls" not found Nov 03 13:08:31.192582 ip-10-0-131-36 kubenswrapper[2573]: E1103 13:08:31.192371 2573 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 03 13:08:31.192582 ip-10-0-131-36 kubenswrapper[2573]: E1103 13:08:31.192468 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cc8ef30e-6d4f-4913-8dc8-b09e5aea2f82-registry-tls podName:cc8ef30e-6d4f-4913-8dc8-b09e5aea2f82 nodeName:}" failed. No retries permitted until 2026-11-03 13:08:47.192454931 +0000 UTC m=+62.166777550 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/cc8ef30e-6d4f-4913-8dc8-b09e5aea2f82-registry-tls") pod "image-registry-555f8976d-mz5ct" (UID: "cc8ef30e-6d4f-4913-8dc8-b09e5aea2f82") : secret "image-registry-tls" not found Nov 03 13:08:31.192582 ip-10-0-131-36 kubenswrapper[2573]: E1103 13:08:31.192557 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fa297a30-6da7-4a92-98fb-201323490bb0-networking-console-plugin-cert podName:fa297a30-6da7-4a92-98fb-201323490bb0 nodeName:}" failed. No retries permitted until 2026-11-03 13:08:47.192532751 +0000 UTC m=+62.166855385 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/fa297a30-6da7-4a92-98fb-201323490bb0-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-rz4n9" (UID: "fa297a30-6da7-4a92-98fb-201323490bb0") : secret "networking-console-plugin-cert" not found Nov 03 13:08:31.292802 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:31.292722 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/898bd0b5-0d3e-4577-b101-2cc7d75283cb-service-ca-bundle\") pod \"router-default-7f579888d9-ntt6m\" (UID: \"898bd0b5-0d3e-4577-b101-2cc7d75283cb\") " pod="openshift-ingress/router-default-7f579888d9-ntt6m" Nov 03 13:08:31.292802 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:31.292778 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3fdeb9f3-6c67-4390-8073-17d4d5f802b9-metrics-tls\") pod \"dns-default-vgnv8\" (UID: \"3fdeb9f3-6c67-4390-8073-17d4d5f802b9\") " pod="openshift-dns/dns-default-vgnv8" Nov 03 13:08:31.292980 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:31.292806 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/898bd0b5-0d3e-4577-b101-2cc7d75283cb-metrics-certs\") pod \"router-default-7f579888d9-ntt6m\" (UID: \"898bd0b5-0d3e-4577-b101-2cc7d75283cb\") " pod="openshift-ingress/router-default-7f579888d9-ntt6m" Nov 03 13:08:31.292980 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:31.292849 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0a892eb3-ecc4-4312-9f4d-73eeba6b6eee-registry-tls\") pod \"image-registry-7dbb67bd57-74qfj\" (UID: \"0a892eb3-ecc4-4312-9f4d-73eeba6b6eee\") " pod="openshift-image-registry/image-registry-7dbb67bd57-74qfj" Nov 03 13:08:31.292980 ip-10-0-131-36 kubenswrapper[2573]: E1103 13:08:31.292878 2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 13:08:31.292980 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:31.292921 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/21618783-0a4e-4b1f-8041-cabdf69f5bea-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-h46gl\" (UID: \"21618783-0a4e-4b1f-8041-cabdf69f5bea\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-h46gl" Nov 03 13:08:31.292980 ip-10-0-131-36 kubenswrapper[2573]: E1103 13:08:31.292941 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/898bd0b5-0d3e-4577-b101-2cc7d75283cb-service-ca-bundle podName:898bd0b5-0d3e-4577-b101-2cc7d75283cb nodeName:}" failed. No retries permitted until 2026-11-03 13:08:47.292921879 +0000 UTC m=+62.267244500 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/898bd0b5-0d3e-4577-b101-2cc7d75283cb-service-ca-bundle") pod "router-default-7f579888d9-ntt6m" (UID: "898bd0b5-0d3e-4577-b101-2cc7d75283cb") : configmap references non-existent config key: service-ca.crt Nov 03 13:08:31.292980 ip-10-0-131-36 kubenswrapper[2573]: E1103 13:08:31.292945 2573 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 03 13:08:31.292980 ip-10-0-131-36 kubenswrapper[2573]: E1103 13:08:31.292966 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3fdeb9f3-6c67-4390-8073-17d4d5f802b9-metrics-tls podName:3fdeb9f3-6c67-4390-8073-17d4d5f802b9 nodeName:}" failed. No retries permitted until 2026-11-03 13:08:47.292954437 +0000 UTC m=+62.267277060 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3fdeb9f3-6c67-4390-8073-17d4d5f802b9-metrics-tls") pod "dns-default-vgnv8" (UID: "3fdeb9f3-6c67-4390-8073-17d4d5f802b9") : secret "dns-default-metrics-tls" not found Nov 03 13:08:31.293226 ip-10-0-131-36 kubenswrapper[2573]: E1103 13:08:31.292983 2573 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 03 13:08:31.293226 ip-10-0-131-36 kubenswrapper[2573]: E1103 13:08:31.293003 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/898bd0b5-0d3e-4577-b101-2cc7d75283cb-metrics-certs podName:898bd0b5-0d3e-4577-b101-2cc7d75283cb nodeName:}" failed. No retries permitted until 2026-11-03 13:08:47.292986787 +0000 UTC m=+62.267309420 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/898bd0b5-0d3e-4577-b101-2cc7d75283cb-metrics-certs") pod "router-default-7f579888d9-ntt6m" (UID: "898bd0b5-0d3e-4577-b101-2cc7d75283cb") : secret "router-metrics-certs-default" not found Nov 03 13:08:31.293226 ip-10-0-131-36 kubenswrapper[2573]: E1103 13:08:31.293023 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/21618783-0a4e-4b1f-8041-cabdf69f5bea-samples-operator-tls podName:21618783-0a4e-4b1f-8041-cabdf69f5bea nodeName:}" failed. No retries permitted until 2026-11-03 13:08:47.293012235 +0000 UTC m=+62.267334876 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/21618783-0a4e-4b1f-8041-cabdf69f5bea-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-h46gl" (UID: "21618783-0a4e-4b1f-8041-cabdf69f5bea") : secret "samples-operator-tls" not found Nov 03 13:08:31.293226 ip-10-0-131-36 kubenswrapper[2573]: E1103 13:08:31.293026 2573 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 13:08:31.293226 ip-10-0-131-36 kubenswrapper[2573]: E1103 13:08:31.293040 2573 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7dbb67bd57-74qfj: secret "image-registry-tls" not found Nov 03 13:08:31.293226 ip-10-0-131-36 kubenswrapper[2573]: E1103 13:08:31.293075 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0a892eb3-ecc4-4312-9f4d-73eeba6b6eee-registry-tls podName:0a892eb3-ecc4-4312-9f4d-73eeba6b6eee nodeName:}" failed. No retries permitted until 2026-11-03 13:08:47.293065079 +0000 UTC m=+62.267387697 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/0a892eb3-ecc4-4312-9f4d-73eeba6b6eee-registry-tls") pod "image-registry-7dbb67bd57-74qfj" (UID: "0a892eb3-ecc4-4312-9f4d-73eeba6b6eee") : secret "image-registry-tls" not found Nov 03 13:08:31.388827 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:31.388803 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-4tn4j_39e11989-e598-4cea-87d9-26b3cf78a66b/node-ca/0.log" Nov 03 13:08:31.392758 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:31.392735 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-gkblf"] Nov 03 13:08:31.406407 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:31.406372 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-gkblf" Nov 03 13:08:31.407847 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:31.407826 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-gkblf"] Nov 03 13:08:31.409753 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:31.409730 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Nov 03 13:08:31.409850 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:31.409734 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Nov 03 13:08:31.409850 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:31.409782 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-zrpxj\"" Nov 03 13:08:31.494254 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:31.494218 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mtd8p\" (UniqueName: \"kubernetes.io/projected/bd523616-71b2-4b6c-b82c-f1efb585c271-kube-api-access-mtd8p\") pod \"migrator-66df5d496b-gkblf\" (UID: \"bd523616-71b2-4b6c-b82c-f1efb585c271\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-gkblf" Nov 03 13:08:31.594846 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:31.594771 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtd8p\" (UniqueName: \"kubernetes.io/projected/bd523616-71b2-4b6c-b82c-f1efb585c271-kube-api-access-mtd8p\") pod \"migrator-66df5d496b-gkblf\" (UID: \"bd523616-71b2-4b6c-b82c-f1efb585c271\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-gkblf" Nov 03 13:08:31.602348 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:31.602320 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mtd8p\" (UniqueName: \"kubernetes.io/projected/bd523616-71b2-4b6c-b82c-f1efb585c271-kube-api-access-mtd8p\") pod \"migrator-66df5d496b-gkblf\" (UID: \"bd523616-71b2-4b6c-b82c-f1efb585c271\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-gkblf" Nov 03 13:08:31.741525 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:31.741490 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-gkblf" Nov 03 13:08:31.973194 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:31.973167 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6htht_cd91de08-084e-4667-b13f-2e07b3cff624/console-operator/1.log" Nov 03 13:08:31.973692 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:31.973585 2573 scope.go:122] "RemoveContainer" containerID="b811aececd178145fbcac8b298ed317468857bb459f9492b4b679c31b566f969" Nov 03 13:08:31.973871 ip-10-0-131-36 kubenswrapper[2573]: E1103 13:08:31.973827 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-6htht_openshift-console-operator(cd91de08-084e-4667-b13f-2e07b3cff624)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-6htht" podUID="cd91de08-084e-4667-b13f-2e07b3cff624" Nov 03 13:08:31.975479 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:31.975452 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-cqvdv" event={"ID":"9f20ddda-41fb-4aba-b257-7ebc59dbe678","Type":"ContainerStarted","Data":"c9f958c9f711ed0fab0f4057527197d9dfabc75f606d1cebfdabe78993cd43ca"} Nov 03 13:08:32.001231 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:32.001185 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-cqvdv" podStartSLOduration=369.524992185 podStartE2EDuration="6m25.001171992s" totalImagesPullingTime="15.476179807s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:02:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:08:32.000024864 +0000 UTC m=+46.974347526" watchObservedRunningTime="2026-11-03 13:08:32.001171992 +0000 UTC m=+46.975494627" Nov 03 13:08:32.031428 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:32.031393 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-gkblf"] Nov 03 13:08:32.036043 ip-10-0-131-36 kubenswrapper[2573]: W1103 13:08:32.036011 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd523616_71b2_4b6c_b82c_f1efb585c271.slice/crio-a15ed7253699ac60f5504c1ca9afee7dc425b76d3113908f1e8d7116db36f547 WatchSource:0}: Error finding container a15ed7253699ac60f5504c1ca9afee7dc425b76d3113908f1e8d7116db36f547: Status 404 returned error can't find the container with id a15ed7253699ac60f5504c1ca9afee7dc425b76d3113908f1e8d7116db36f547 Nov 03 13:08:32.186072 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:32.186040 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-d8f6l_50d68cb7-b766-4674-b295-8f3229880994/serve-healthcheck-canary/0.log" Nov 03 13:08:32.980514 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:32.980480 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-gkblf" event={"ID":"bd523616-71b2-4b6c-b82c-f1efb585c271","Type":"ContainerStarted","Data":"a15ed7253699ac60f5504c1ca9afee7dc425b76d3113908f1e8d7116db36f547"} Nov 03 13:08:33.108412 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:33.108374 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fff35d1b-e7b2-4dd3-9824-8b64fad02506-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-d7mwz\" (UID: \"fff35d1b-e7b2-4dd3-9824-8b64fad02506\") " pod="openshift-insights/insights-runtime-extractor-d7mwz" Nov 03 13:08:33.108570 ip-10-0-131-36 kubenswrapper[2573]: E1103 13:08:33.108519 2573 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 03 13:08:33.108613 ip-10-0-131-36 kubenswrapper[2573]: E1103 13:08:33.108583 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fff35d1b-e7b2-4dd3-9824-8b64fad02506-insights-runtime-extractor-tls podName:fff35d1b-e7b2-4dd3-9824-8b64fad02506 nodeName:}" failed. No retries permitted until 2026-11-03 13:08:41.108568937 +0000 UTC m=+56.082891571 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/fff35d1b-e7b2-4dd3-9824-8b64fad02506-insights-runtime-extractor-tls") pod "insights-runtime-extractor-d7mwz" (UID: "fff35d1b-e7b2-4dd3-9824-8b64fad02506") : secret "insights-runtime-extractor-tls" not found Nov 03 13:08:35.738596 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:35.738560 2573 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-6htht" Nov 03 13:08:35.738976 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:35.738619 2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-6htht" Nov 03 13:08:35.738976 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:35.738918 2573 scope.go:122] "RemoveContainer" containerID="b811aececd178145fbcac8b298ed317468857bb459f9492b4b679c31b566f969" Nov 03 13:08:35.739095 ip-10-0-131-36 kubenswrapper[2573]: E1103 13:08:35.739075 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-6htht_openshift-console-operator(cd91de08-084e-4667-b13f-2e07b3cff624)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-6htht" podUID="cd91de08-084e-4667-b13f-2e07b3cff624" Nov 03 13:08:39.936465 ip-10-0-131-36 kubenswrapper[2573]: I1103 13:08:39.936349 2573 dynamic_cafile_content.go:175] "Shutting down controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 13:08:39.936513 ip-10-0-131-36 systemd[1]: Stopping Kubernetes Kubelet... Nov 03 13:08:39.949710 ip-10-0-131-36 systemd[1]: kubelet.service: Deactivated successfully. Nov 03 13:08:39.949879 ip-10-0-131-36 systemd[1]: Stopped Kubernetes Kubelet. Nov 03 13:08:39.949929 ip-10-0-131-36 systemd[1]: kubelet.service: Consumed 3.416s CPU time. Nov 03 13:08:39.962006 ip-10-0-131-36 systemd[1]: Starting Kubernetes Kubelet... Nov 03 13:08:40.021996 ip-10-0-131-36 kubenswrapper[5024]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 13:08:40.021996 ip-10-0-131-36 kubenswrapper[5024]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 03 13:08:40.021996 ip-10-0-131-36 kubenswrapper[5024]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 13:08:40.021996 ip-10-0-131-36 kubenswrapper[5024]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 13:08:40.025773 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.025662 5024 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 13:08:40.025773 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.025772 5024 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 13:08:40.025773 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.025778 5024 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 13:08:40.025976 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.025783 5024 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 13:08:40.025976 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.025788 5024 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 13:08:40.025976 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.025792 5024 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 13:08:40.025976 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.025797 5024 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 13:08:40.025976 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.025801 5024 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 13:08:40.025976 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.025805 5024 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 13:08:40.025976 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.025811 5024 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 13:08:40.025976 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.025816 5024 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 13:08:40.025976 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.025826 5024 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 13:08:40.025976 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.025830 5024 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 13:08:40.025976 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.025834 5024 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 13:08:40.025976 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.025838 5024 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 13:08:40.025976 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.025841 5024 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 13:08:40.025976 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.025845 5024 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 13:08:40.025976 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.025850 5024 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 13:08:40.025976 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.025854 5024 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 13:08:40.025976 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.025860 5024 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 13:08:40.025976 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.025864 5024 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 13:08:40.025976 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.025869 5024 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 13:08:40.025976 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.025873 5024 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 13:08:40.026518 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.025877 5024 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 13:08:40.026518 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.025882 5024 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 13:08:40.026518 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.025886 5024 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 13:08:40.026518 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.025891 5024 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 13:08:40.026518 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.025895 5024 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 13:08:40.026518 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.025898 5024 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 13:08:40.026518 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.025901 5024 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 13:08:40.026518 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.025904 5024 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 13:08:40.026518 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.025907 5024 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 13:08:40.026518 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.025911 5024 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 13:08:40.026518 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.025914 5024 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 13:08:40.026518 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.025918 5024 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 13:08:40.026518 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.025921 5024 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 13:08:40.026518 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.025923 5024 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 13:08:40.026518 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.025926 5024 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 13:08:40.026518 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.025929 5024 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 13:08:40.026518 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.025933 5024 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 13:08:40.026518 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.025937 5024 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 13:08:40.026518 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.025939 5024 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 13:08:40.027001 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.025942 5024 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 13:08:40.027001 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.025945 5024 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 13:08:40.027001 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.025949 5024 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 13:08:40.027001 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.025952 5024 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 13:08:40.027001 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.025955 5024 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 13:08:40.027001 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.025958 5024 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 13:08:40.027001 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.025960 5024 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 13:08:40.027001 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.025963 5024 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 13:08:40.027001 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.025967 5024 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 13:08:40.027001 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.025969 5024 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 13:08:40.027001 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.025972 5024 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 13:08:40.027001 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.025975 5024 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 13:08:40.027001 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.025978 5024 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 13:08:40.027001 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.025980 5024 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 13:08:40.027001 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.025983 5024 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 13:08:40.027001 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.025986 5024 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 13:08:40.027001 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.025989 5024 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 13:08:40.027001 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.025992 5024 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 13:08:40.027001 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.025995 5024 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 13:08:40.027001 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.025997 5024 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 13:08:40.027540 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.026000 5024 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 13:08:40.027540 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.026003 5024 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 13:08:40.027540 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.026005 5024 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 13:08:40.027540 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.026008 5024 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 13:08:40.027540 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.026010 5024 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 13:08:40.027540 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.026013 5024 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 13:08:40.027540 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.026016 5024 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 13:08:40.027540 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.026019 5024 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 13:08:40.027540 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.026021 5024 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 13:08:40.027540 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.026025 5024 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 13:08:40.027540 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.026027 5024 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 13:08:40.027540 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.026030 5024 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 13:08:40.027540 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.026033 5024 feature_gate.go:386] unrecognized feature gate: Example Nov 03 13:08:40.027540 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.026035 5024 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 13:08:40.027540 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.026038 5024 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 13:08:40.027540 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.026041 5024 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 13:08:40.027540 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.026043 5024 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 13:08:40.027540 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.026048 5024 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 13:08:40.027540 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.026053 5024 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 13:08:40.027540 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.026056 5024 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 13:08:40.028074 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.026059 5024 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 13:08:40.028074 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.026063 5024 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 13:08:40.028074 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.026066 5024 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 13:08:40.028074 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.026068 5024 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 13:08:40.028074 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.026072 5024 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 13:08:40.028074 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.026077 5024 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 13:08:40.028074 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.026081 5024 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 13:08:40.028074 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.026084 5024 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 13:08:40.028074 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.026087 5024 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 13:08:40.028074 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.026090 5024 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 13:08:40.028074 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.026092 5024 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 13:08:40.028074 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.026097 5024 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 13:08:40.028074 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.026099 5024 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 13:08:40.028074 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.026102 5024 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 13:08:40.028074 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.026104 5024 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 13:08:40.028074 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.026107 5024 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 13:08:40.028074 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.026110 5024 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 13:08:40.028074 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.026113 5024 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 13:08:40.028074 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.026115 5024 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 13:08:40.028620 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.026118 5024 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 13:08:40.028620 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.026122 5024 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 13:08:40.028620 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.026124 5024 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 13:08:40.028620 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.026127 5024 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 13:08:40.028620 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.026130 5024 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 13:08:40.028620 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.026954 5024 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 13:08:40.028620 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.026963 5024 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 13:08:40.028620 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.026966 5024 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 13:08:40.028620 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.026969 5024 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 13:08:40.028620 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.026971 5024 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 13:08:40.028620 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.026976 5024 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 13:08:40.028620 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.026979 5024 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 13:08:40.028620 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.026982 5024 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 13:08:40.028620 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.026984 5024 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 13:08:40.028620 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.026987 5024 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 13:08:40.028620 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.026990 5024 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 13:08:40.028620 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.026993 5024 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 13:08:40.028620 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.026996 5024 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 13:08:40.028620 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.026998 5024 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 13:08:40.028620 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.027001 5024 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 13:08:40.029122 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.027004 5024 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 13:08:40.029122 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.027008 5024 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 13:08:40.029122 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.027011 5024 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 13:08:40.029122 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.027013 5024 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 13:08:40.029122 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.027016 5024 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 13:08:40.029122 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.027019 5024 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 13:08:40.029122 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.027022 5024 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 13:08:40.029122 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.027025 5024 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 13:08:40.029122 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.027027 5024 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 13:08:40.029122 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.027030 5024 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 13:08:40.029122 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.027033 5024 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 13:08:40.029122 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.027036 5024 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 13:08:40.029122 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.027039 5024 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 13:08:40.029122 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.027041 5024 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 13:08:40.029122 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.027044 5024 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 13:08:40.029122 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.027047 5024 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 13:08:40.029122 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.027049 5024 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 13:08:40.029122 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.027052 5024 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 13:08:40.029122 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.027055 5024 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 13:08:40.029626 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.027059 5024 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 13:08:40.029626 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.027063 5024 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 13:08:40.029626 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.027066 5024 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 13:08:40.029626 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.027069 5024 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 13:08:40.029626 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.027073 5024 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 13:08:40.029626 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.027076 5024 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 13:08:40.029626 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.027079 5024 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 13:08:40.029626 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.027081 5024 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 13:08:40.029626 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.027084 5024 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 13:08:40.029626 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.027087 5024 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 13:08:40.029626 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.027090 5024 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 13:08:40.029626 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.027093 5024 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 13:08:40.029626 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.027096 5024 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 13:08:40.029626 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.027098 5024 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 13:08:40.029626 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.027101 5024 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 13:08:40.029626 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.027103 5024 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 13:08:40.029626 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.027106 5024 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 13:08:40.029626 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.027109 5024 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 13:08:40.029626 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.027116 5024 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 13:08:40.030104 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.027118 5024 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 13:08:40.030104 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.027121 5024 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 13:08:40.030104 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.027124 5024 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 13:08:40.030104 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.027127 5024 feature_gate.go:386] unrecognized feature gate: Example Nov 03 13:08:40.030104 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.027129 5024 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 13:08:40.030104 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.027132 5024 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 13:08:40.030104 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.027135 5024 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 13:08:40.030104 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.027137 5024 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 13:08:40.030104 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.027140 5024 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 13:08:40.030104 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.027142 5024 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 13:08:40.030104 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.027145 5024 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 13:08:40.030104 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.027148 5024 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 13:08:40.030104 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.027150 5024 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 13:08:40.030104 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.027153 5024 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 13:08:40.030104 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.027155 5024 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 13:08:40.030104 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.027158 5024 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 13:08:40.030104 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.027161 5024 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 13:08:40.030104 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.027163 5024 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 13:08:40.030104 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.027166 5024 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 13:08:40.030104 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.027168 5024 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 13:08:40.030644 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.027171 5024 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 13:08:40.030644 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.027174 5024 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 13:08:40.030644 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.027176 5024 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 13:08:40.030644 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.027180 5024 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 13:08:40.030644 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.027183 5024 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 13:08:40.030644 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.027185 5024 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 13:08:40.030644 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.027188 5024 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 13:08:40.030644 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.027190 5024 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 13:08:40.030644 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.027193 5024 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 13:08:40.030644 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.027197 5024 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 13:08:40.030644 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.027201 5024 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 13:08:40.030644 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.027205 5024 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 13:08:40.030644 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.027209 5024 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 13:08:40.030644 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.027212 5024 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 13:08:40.030644 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.027214 5024 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 13:08:40.030644 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.027217 5024 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 13:08:40.030644 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.027220 5024 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 13:08:40.030644 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.027222 5024 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 13:08:40.030644 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.027225 5024 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 13:08:40.031137 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.027228 5024 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 13:08:40.031137 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.027230 5024 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 13:08:40.031137 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.027233 5024 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 13:08:40.031137 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.027237 5024 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 13:08:40.031137 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.027239 5024 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 13:08:40.031137 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.027242 5024 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 13:08:40.031137 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.027244 5024 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 13:08:40.031137 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.027247 5024 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 13:08:40.031137 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.027249 5024 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 13:08:40.031137 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.027252 5024 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 13:08:40.031137 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.027254 5024 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 13:08:40.031137 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.027257 5024 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 13:08:40.031137 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.027259 5024 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 13:08:40.031137 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.027262 5024 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 13:08:40.031137 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.027771 5024 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 13:08:40.031137 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.027777 5024 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 13:08:40.031137 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.027780 5024 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 13:08:40.031137 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.027784 5024 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 13:08:40.031137 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.027787 5024 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 13:08:40.031659 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.027790 5024 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 13:08:40.031659 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.027793 5024 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 13:08:40.031659 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.027795 5024 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 13:08:40.031659 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.027798 5024 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 13:08:40.031659 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.027801 5024 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 13:08:40.031659 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.027806 5024 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 13:08:40.031659 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.027809 5024 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 13:08:40.031659 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.027812 5024 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 13:08:40.031659 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.027814 5024 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 13:08:40.031659 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.027817 5024 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 13:08:40.031659 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.027820 5024 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 13:08:40.031659 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.027822 5024 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 13:08:40.031659 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.027825 5024 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 13:08:40.031659 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.027828 5024 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 13:08:40.031659 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.027830 5024 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 13:08:40.031659 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.027833 5024 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 13:08:40.031659 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.027835 5024 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 13:08:40.031659 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.027838 5024 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 13:08:40.031659 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.027840 5024 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 13:08:40.031659 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.027843 5024 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 13:08:40.032155 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.027846 5024 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 13:08:40.032155 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.027848 5024 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 13:08:40.032155 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.027852 5024 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 13:08:40.032155 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.027854 5024 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 13:08:40.032155 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.027857 5024 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 13:08:40.032155 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.027859 5024 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 13:08:40.032155 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.027862 5024 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 13:08:40.032155 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.027866 5024 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 13:08:40.032155 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.027869 5024 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 13:08:40.032155 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.027873 5024 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 13:08:40.032155 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.027876 5024 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 13:08:40.032155 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.027879 5024 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 13:08:40.032155 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.027882 5024 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 13:08:40.032155 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.027885 5024 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 13:08:40.032155 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.027887 5024 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 13:08:40.032155 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.027890 5024 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 13:08:40.032155 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.027893 5024 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 13:08:40.032155 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.027897 5024 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 13:08:40.032155 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.027899 5024 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 13:08:40.032673 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.027902 5024 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 13:08:40.032673 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.027905 5024 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 13:08:40.032673 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.027907 5024 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 13:08:40.032673 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.027910 5024 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 13:08:40.032673 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.027914 5024 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 13:08:40.032673 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.027917 5024 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 13:08:40.032673 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.027919 5024 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 13:08:40.032673 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.027922 5024 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 13:08:40.032673 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.027924 5024 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 13:08:40.032673 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.027927 5024 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 13:08:40.032673 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.027930 5024 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 13:08:40.032673 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.027932 5024 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 13:08:40.032673 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.027935 5024 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 13:08:40.032673 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.027937 5024 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 13:08:40.032673 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.027940 5024 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 13:08:40.032673 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.027943 5024 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 13:08:40.032673 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.027945 5024 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 13:08:40.032673 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.027948 5024 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 13:08:40.032673 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.027950 5024 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 13:08:40.032673 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.027953 5024 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 13:08:40.033142 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.027955 5024 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 13:08:40.033142 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.027958 5024 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 13:08:40.033142 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.027961 5024 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 13:08:40.033142 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.027964 5024 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 13:08:40.033142 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.027967 5024 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 13:08:40.033142 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.027970 5024 feature_gate.go:386] unrecognized feature gate: Example Nov 03 13:08:40.033142 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.027972 5024 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 13:08:40.033142 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.027975 5024 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 13:08:40.033142 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.027977 5024 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 13:08:40.033142 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.027980 5024 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 13:08:40.033142 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.027984 5024 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 13:08:40.033142 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.027987 5024 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 13:08:40.033142 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.027990 5024 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 13:08:40.033142 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.027992 5024 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 13:08:40.033142 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.027995 5024 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 13:08:40.033142 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.027997 5024 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 13:08:40.033142 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.028000 5024 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 13:08:40.033142 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.028003 5024 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 13:08:40.033142 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.028005 5024 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 13:08:40.033142 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.028008 5024 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 13:08:40.033659 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.028010 5024 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 13:08:40.033659 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.028013 5024 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 13:08:40.033659 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.028017 5024 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 13:08:40.033659 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.028021 5024 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 13:08:40.033659 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.028024 5024 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 13:08:40.033659 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.028026 5024 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 13:08:40.033659 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.028029 5024 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 13:08:40.033659 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.028032 5024 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 13:08:40.033659 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.028035 5024 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 13:08:40.033659 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.028037 5024 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 13:08:40.033659 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.028040 5024 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 13:08:40.033659 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.028042 5024 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 13:08:40.033659 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.028045 5024 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 13:08:40.033659 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.028048 5024 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 13:08:40.033659 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.028050 5024 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 13:08:40.033659 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.028053 5024 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 13:08:40.033659 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.028056 5024 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 13:08:40.033659 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.028059 5024 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 13:08:40.033659 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.028062 5024 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 13:08:40.033659 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.028064 5024 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 13:08:40.034179 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.028067 5024 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 13:08:40.034179 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.028070 5024 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 13:08:40.034179 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.028171 5024 flags.go:64] FLAG: --address="0.0.0.0" Nov 03 13:08:40.034179 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.028190 5024 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 03 13:08:40.034179 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.028199 5024 flags.go:64] FLAG: --anonymous-auth="true" Nov 03 13:08:40.034179 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.028206 5024 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 03 13:08:40.034179 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.028211 5024 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 03 13:08:40.034179 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.028214 5024 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 03 13:08:40.034179 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.028219 5024 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 03 13:08:40.034179 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.028223 5024 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 03 13:08:40.034179 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.028227 5024 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 03 13:08:40.034179 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.028230 5024 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 03 13:08:40.034179 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.028234 5024 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 03 13:08:40.034179 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.028237 5024 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 03 13:08:40.034179 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.028241 5024 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 03 13:08:40.034179 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.028244 5024 flags.go:64] FLAG: --cgroup-root="" Nov 03 13:08:40.034179 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.028247 5024 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 03 13:08:40.034179 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.028250 5024 flags.go:64] FLAG: --client-ca-file="" Nov 03 13:08:40.034179 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.028253 5024 flags.go:64] FLAG: --cloud-provider="external" Nov 03 13:08:40.034179 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.028256 5024 flags.go:64] FLAG: --cluster-dns="[]" Nov 03 13:08:40.034179 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.028261 5024 flags.go:64] FLAG: --cluster-domain="" Nov 03 13:08:40.034179 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.028264 5024 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 03 13:08:40.034179 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.028267 5024 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 03 13:08:40.034763 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.028271 5024 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 03 13:08:40.034763 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.028274 5024 flags.go:64] FLAG: --container-log-max-files="5" Nov 03 13:08:40.034763 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.028278 5024 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 03 13:08:40.034763 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.028299 5024 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 03 13:08:40.034763 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.028303 5024 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 03 13:08:40.034763 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.028307 5024 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 03 13:08:40.034763 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.028310 5024 flags.go:64] FLAG: --contention-profiling="false" Nov 03 13:08:40.034763 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.028314 5024 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 03 13:08:40.034763 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.028317 5024 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 03 13:08:40.034763 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.028320 5024 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 03 13:08:40.034763 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.028324 5024 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 03 13:08:40.034763 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.028329 5024 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 03 13:08:40.034763 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.028332 5024 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 03 13:08:40.034763 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.028335 5024 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 03 13:08:40.034763 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.028339 5024 flags.go:64] FLAG: --enable-load-reader="false" Nov 03 13:08:40.034763 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.028342 5024 flags.go:64] FLAG: --enable-server="true" Nov 03 13:08:40.034763 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.028345 5024 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 03 13:08:40.034763 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.028349 5024 flags.go:64] FLAG: --event-burst="100" Nov 03 13:08:40.034763 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.028352 5024 flags.go:64] FLAG: --event-qps="50" Nov 03 13:08:40.034763 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.028355 5024 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 03 13:08:40.034763 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.028359 5024 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 03 13:08:40.034763 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.028362 5024 flags.go:64] FLAG: --eviction-hard="" Nov 03 13:08:40.034763 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.028366 5024 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 03 13:08:40.034763 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.028369 5024 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 03 13:08:40.035419 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.028373 5024 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 03 13:08:40.035419 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.028376 5024 flags.go:64] FLAG: --eviction-soft="" Nov 03 13:08:40.035419 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.028379 5024 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 03 13:08:40.035419 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.028382 5024 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 03 13:08:40.035419 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.028385 5024 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 03 13:08:40.035419 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.028388 5024 flags.go:64] FLAG: --experimental-mounter-path="" Nov 03 13:08:40.035419 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.028391 5024 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 03 13:08:40.035419 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.028394 5024 flags.go:64] FLAG: --fail-swap-on="true" Nov 03 13:08:40.035419 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.028397 5024 flags.go:64] FLAG: --feature-gates="" Nov 03 13:08:40.035419 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.028401 5024 flags.go:64] FLAG: --file-check-frequency="20s" Nov 03 13:08:40.035419 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.028404 5024 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 03 13:08:40.035419 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.028409 5024 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 03 13:08:40.035419 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.028412 5024 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 03 13:08:40.035419 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.028416 5024 flags.go:64] FLAG: --healthz-port="10248" Nov 03 13:08:40.035419 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.028419 5024 flags.go:64] FLAG: --help="false" Nov 03 13:08:40.035419 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.028422 5024 flags.go:64] FLAG: --hostname-override="ip-10-0-131-36.us-east-2.compute.internal" Nov 03 13:08:40.035419 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.028427 5024 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 03 13:08:40.035419 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.028430 5024 flags.go:64] FLAG: --http-check-frequency="20s" Nov 03 13:08:40.035419 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.028434 5024 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 03 13:08:40.035419 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.028437 5024 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 03 13:08:40.035419 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.028441 5024 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 03 13:08:40.035419 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.028444 5024 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 03 13:08:40.035419 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.028447 5024 flags.go:64] FLAG: --image-service-endpoint="" Nov 03 13:08:40.035985 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.028450 5024 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 03 13:08:40.035985 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.028453 5024 flags.go:64] FLAG: --kube-api-burst="100" Nov 03 13:08:40.035985 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.028456 5024 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 03 13:08:40.035985 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.028459 5024 flags.go:64] FLAG: --kube-api-qps="50" Nov 03 13:08:40.035985 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.028462 5024 flags.go:64] FLAG: --kube-reserved="" Nov 03 13:08:40.035985 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.028465 5024 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 03 13:08:40.035985 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.028468 5024 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 03 13:08:40.035985 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.028471 5024 flags.go:64] FLAG: --kubelet-cgroups="" Nov 03 13:08:40.035985 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.028474 5024 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 03 13:08:40.035985 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.028477 5024 flags.go:64] FLAG: --lock-file="" Nov 03 13:08:40.035985 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.028480 5024 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 03 13:08:40.035985 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.028483 5024 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 03 13:08:40.035985 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.028486 5024 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 03 13:08:40.035985 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.028492 5024 flags.go:64] FLAG: --log-json-split-stream="false" Nov 03 13:08:40.035985 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.028495 5024 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 03 13:08:40.035985 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.028498 5024 flags.go:64] FLAG: --log-text-split-stream="false" Nov 03 13:08:40.035985 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.028501 5024 flags.go:64] FLAG: --logging-format="text" Nov 03 13:08:40.035985 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.028504 5024 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 03 13:08:40.035985 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.028508 5024 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 03 13:08:40.035985 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.028511 5024 flags.go:64] FLAG: --manifest-url="" Nov 03 13:08:40.035985 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.028515 5024 flags.go:64] FLAG: --manifest-url-header="" Nov 03 13:08:40.035985 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.028519 5024 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 03 13:08:40.035985 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.028523 5024 flags.go:64] FLAG: --max-open-files="1000000" Nov 03 13:08:40.035985 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.028527 5024 flags.go:64] FLAG: --max-pods="110" Nov 03 13:08:40.035985 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.028530 5024 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 03 13:08:40.036631 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.028533 5024 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 03 13:08:40.036631 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.028539 5024 flags.go:64] FLAG: --memory-manager-policy="None" Nov 03 13:08:40.036631 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.028542 5024 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 03 13:08:40.036631 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.028545 5024 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 03 13:08:40.036631 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.028549 5024 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 03 13:08:40.036631 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.028552 5024 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 03 13:08:40.036631 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.028559 5024 flags.go:64] FLAG: --node-status-max-images="50" Nov 03 13:08:40.036631 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.028562 5024 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 03 13:08:40.036631 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.028565 5024 flags.go:64] FLAG: --oom-score-adj="-999" Nov 03 13:08:40.036631 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.028569 5024 flags.go:64] FLAG: --pod-cidr="" Nov 03 13:08:40.036631 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.028572 5024 flags.go:64] FLAG: --pod-manifest-path="" Nov 03 13:08:40.036631 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.028575 5024 flags.go:64] FLAG: --pod-max-pids="-1" Nov 03 13:08:40.036631 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.028578 5024 flags.go:64] FLAG: --pods-per-core="0" Nov 03 13:08:40.036631 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.028581 5024 flags.go:64] FLAG: --port="10250" Nov 03 13:08:40.036631 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.028584 5024 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 03 13:08:40.036631 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.028587 5024 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-013559e5d3bed1ebf" Nov 03 13:08:40.036631 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.028591 5024 flags.go:64] FLAG: --qos-reserved="" Nov 03 13:08:40.036631 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.028593 5024 flags.go:64] FLAG: --read-only-port="10255" Nov 03 13:08:40.036631 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.028597 5024 flags.go:64] FLAG: --register-node="true" Nov 03 13:08:40.036631 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.028600 5024 flags.go:64] FLAG: --register-with-taints="" Nov 03 13:08:40.036631 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.028604 5024 flags.go:64] FLAG: --registry-burst="10" Nov 03 13:08:40.036631 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.028607 5024 flags.go:64] FLAG: --registry-qps="5" Nov 03 13:08:40.036631 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.028610 5024 flags.go:64] FLAG: --reserved-cpus="" Nov 03 13:08:40.036631 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.028613 5024 flags.go:64] FLAG: --reserved-memory="" Nov 03 13:08:40.036631 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.028617 5024 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 03 13:08:40.036631 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.028621 5024 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 03 13:08:40.037277 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.028624 5024 flags.go:64] FLAG: --rotate-certificates="false" Nov 03 13:08:40.037277 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.028629 5024 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 03 13:08:40.037277 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.028632 5024 flags.go:64] FLAG: --runonce="false" Nov 03 13:08:40.037277 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.028635 5024 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 03 13:08:40.037277 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.028639 5024 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 03 13:08:40.037277 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.028642 5024 flags.go:64] FLAG: --seccomp-default="false" Nov 03 13:08:40.037277 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.028645 5024 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 03 13:08:40.037277 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.028648 5024 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 03 13:08:40.037277 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.028652 5024 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 03 13:08:40.037277 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.028655 5024 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 03 13:08:40.037277 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.028658 5024 flags.go:64] FLAG: --storage-driver-password="root" Nov 03 13:08:40.037277 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.028661 5024 flags.go:64] FLAG: --storage-driver-secure="false" Nov 03 13:08:40.037277 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.028664 5024 flags.go:64] FLAG: --storage-driver-table="stats" Nov 03 13:08:40.037277 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.028667 5024 flags.go:64] FLAG: --storage-driver-user="root" Nov 03 13:08:40.037277 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.028670 5024 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 03 13:08:40.037277 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.028674 5024 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 03 13:08:40.037277 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.028677 5024 flags.go:64] FLAG: --system-cgroups="" Nov 03 13:08:40.037277 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.028679 5024 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 03 13:08:40.037277 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.028685 5024 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 03 13:08:40.037277 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.028688 5024 flags.go:64] FLAG: --tls-cert-file="" Nov 03 13:08:40.037277 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.028691 5024 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 03 13:08:40.037277 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.028695 5024 flags.go:64] FLAG: --tls-min-version="" Nov 03 13:08:40.037277 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.028698 5024 flags.go:64] FLAG: --tls-private-key-file="" Nov 03 13:08:40.037277 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.028701 5024 flags.go:64] FLAG: --topology-manager-policy="none" Nov 03 13:08:40.037277 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.028706 5024 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 03 13:08:40.037955 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.028709 5024 flags.go:64] FLAG: --topology-manager-scope="container" Nov 03 13:08:40.037955 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.028712 5024 flags.go:64] FLAG: --v="2" Nov 03 13:08:40.037955 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.028717 5024 flags.go:64] FLAG: --version="false" Nov 03 13:08:40.037955 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.028721 5024 flags.go:64] FLAG: --vmodule="" Nov 03 13:08:40.037955 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.028726 5024 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 03 13:08:40.037955 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.028729 5024 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 03 13:08:40.037955 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.028868 5024 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 13:08:40.037955 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.028872 5024 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 13:08:40.037955 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.028877 5024 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 13:08:40.037955 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.028879 5024 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 13:08:40.037955 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.028883 5024 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 13:08:40.037955 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.028885 5024 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 13:08:40.037955 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.028888 5024 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 13:08:40.037955 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.028891 5024 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 13:08:40.037955 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.028893 5024 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 13:08:40.037955 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.028898 5024 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 13:08:40.037955 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.028903 5024 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 13:08:40.037955 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.028906 5024 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 13:08:40.037955 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.028909 5024 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 13:08:40.037955 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.028912 5024 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 13:08:40.037955 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.028914 5024 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 13:08:40.038758 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.028917 5024 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 13:08:40.038758 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.028920 5024 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 13:08:40.038758 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.028923 5024 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 13:08:40.038758 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.028925 5024 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 13:08:40.038758 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.028928 5024 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 13:08:40.038758 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.028931 5024 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 13:08:40.038758 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.028933 5024 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 13:08:40.038758 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.028936 5024 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 13:08:40.038758 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.028938 5024 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 13:08:40.038758 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.028941 5024 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 13:08:40.038758 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.028945 5024 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 13:08:40.038758 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.028948 5024 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 13:08:40.038758 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.028950 5024 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 13:08:40.038758 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.028953 5024 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 13:08:40.038758 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.028955 5024 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 13:08:40.038758 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.028958 5024 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 13:08:40.038758 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.028961 5024 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 13:08:40.038758 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.028963 5024 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 13:08:40.038758 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.028966 5024 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 13:08:40.038758 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.028970 5024 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 13:08:40.039581 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.028973 5024 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 13:08:40.039581 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.028975 5024 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 13:08:40.039581 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.028978 5024 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 13:08:40.039581 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.028981 5024 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 13:08:40.039581 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.028983 5024 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 13:08:40.039581 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.028986 5024 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 13:08:40.039581 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.028989 5024 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 13:08:40.039581 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.028991 5024 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 13:08:40.039581 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.028994 5024 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 13:08:40.039581 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.028997 5024 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 13:08:40.039581 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.029000 5024 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 13:08:40.039581 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.029002 5024 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 13:08:40.039581 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.029005 5024 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 13:08:40.039581 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.029007 5024 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 13:08:40.039581 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.029010 5024 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 13:08:40.039581 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.029012 5024 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 13:08:40.039581 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.029015 5024 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 13:08:40.039581 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.029018 5024 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 13:08:40.039581 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.029021 5024 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 13:08:40.039581 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.029023 5024 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 13:08:40.040229 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.029027 5024 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 13:08:40.040229 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.029031 5024 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 13:08:40.040229 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.029035 5024 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 13:08:40.040229 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.029038 5024 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 13:08:40.040229 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.029041 5024 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 13:08:40.040229 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.029044 5024 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 13:08:40.040229 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.029046 5024 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 13:08:40.040229 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.029049 5024 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 13:08:40.040229 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.029052 5024 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 13:08:40.040229 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.029054 5024 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 13:08:40.040229 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.029057 5024 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 13:08:40.040229 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.029061 5024 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 13:08:40.040229 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.029064 5024 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 13:08:40.040229 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.029067 5024 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 13:08:40.040229 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.029070 5024 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 13:08:40.040229 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.029073 5024 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 13:08:40.040229 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.029075 5024 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 13:08:40.040229 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.029078 5024 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 13:08:40.040229 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.029081 5024 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 13:08:40.041070 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.029083 5024 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 13:08:40.041070 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.029086 5024 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 13:08:40.041070 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.029090 5024 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 13:08:40.041070 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.029092 5024 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 13:08:40.041070 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.029095 5024 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 13:08:40.041070 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.029098 5024 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 13:08:40.041070 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.029100 5024 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 13:08:40.041070 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.029103 5024 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 13:08:40.041070 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.029106 5024 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 13:08:40.041070 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.029108 5024 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 13:08:40.041070 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.029111 5024 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 13:08:40.041070 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.029114 5024 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 13:08:40.041070 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.029116 5024 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 13:08:40.041070 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.029119 5024 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 13:08:40.041070 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.029121 5024 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 13:08:40.041070 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.029126 5024 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 13:08:40.041070 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.029129 5024 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 13:08:40.041070 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.029132 5024 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 13:08:40.041070 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.029135 5024 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 13:08:40.041070 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.029137 5024 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 13:08:40.041712 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.029140 5024 feature_gate.go:386] unrecognized feature gate: Example Nov 03 13:08:40.041712 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.029143 5024 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 13:08:40.041712 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.029145 5024 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 13:08:40.041712 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.029148 5024 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 13:08:40.041712 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.029151 5024 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 13:08:40.041712 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.029154 5024 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 13:08:40.041712 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.029156 5024 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 13:08:40.041712 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.029159 5024 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 13:08:40.041712 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.029161 5024 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 13:08:40.041712 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.029164 5024 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 13:08:40.041712 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.029167 5024 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 13:08:40.041712 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.029170 5024 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 13:08:40.041712 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.029196 5024 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 13:08:40.041712 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.029235 5024 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 13:08:40.041712 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.029240 5024 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 13:08:40.041712 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.029244 5024 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 13:08:40.041712 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.029247 5024 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 13:08:40.042142 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.029250 5024 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 13:08:40.042142 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.029253 5024 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 13:08:40.042142 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.029255 5024 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 13:08:40.042142 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.029258 5024 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 13:08:40.042142 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.029261 5024 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 13:08:40.042142 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.029264 5024 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 13:08:40.042142 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.029266 5024 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 13:08:40.042142 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.029269 5024 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 13:08:40.042142 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.029271 5024 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 13:08:40.042142 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.029274 5024 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 13:08:40.042142 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.029277 5024 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 13:08:40.042142 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.029295 5024 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 13:08:40.042142 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.029298 5024 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 13:08:40.042142 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.029301 5024 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 13:08:40.042142 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.029303 5024 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 13:08:40.042142 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.029306 5024 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 13:08:40.042142 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.029309 5024 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 13:08:40.042142 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.029312 5024 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 13:08:40.042142 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.029314 5024 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 13:08:40.042731 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.029317 5024 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 13:08:40.042731 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.029320 5024 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 13:08:40.042731 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.029322 5024 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 13:08:40.042731 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.029325 5024 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 13:08:40.042731 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.029327 5024 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 13:08:40.042731 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.029330 5024 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 13:08:40.042731 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.029333 5024 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 13:08:40.042731 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.029335 5024 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 13:08:40.042731 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.029339 5024 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 13:08:40.042731 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.029341 5024 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 13:08:40.042731 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.029344 5024 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 13:08:40.042731 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.029348 5024 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 13:08:40.042731 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.029352 5024 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 13:08:40.042731 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.029355 5024 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 13:08:40.042731 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.029359 5024 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 13:08:40.042731 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.029362 5024 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 13:08:40.042731 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.029364 5024 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 13:08:40.042731 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.029367 5024 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 13:08:40.042731 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.029370 5024 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 13:08:40.043274 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.029372 5024 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 13:08:40.043274 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.029375 5024 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 13:08:40.043274 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.029378 5024 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 13:08:40.043274 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.029380 5024 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 13:08:40.043274 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.029383 5024 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 13:08:40.043274 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.029387 5024 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 13:08:40.043274 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.029390 5024 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 13:08:40.043274 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.029393 5024 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 13:08:40.043274 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.029396 5024 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 13:08:40.043274 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.029398 5024 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 13:08:40.043274 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.029401 5024 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 13:08:40.043274 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.029404 5024 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 13:08:40.043274 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.029407 5024 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 13:08:40.043274 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.029410 5024 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 13:08:40.043274 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.029413 5024 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 13:08:40.043274 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.029416 5024 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 13:08:40.043274 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.029418 5024 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 13:08:40.043274 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.029421 5024 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 13:08:40.043274 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.029424 5024 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 13:08:40.043274 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.029426 5024 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 13:08:40.043948 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.029429 5024 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 13:08:40.043948 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.029431 5024 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 13:08:40.043948 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.029434 5024 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 13:08:40.043948 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.029436 5024 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 13:08:40.043948 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.029439 5024 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 13:08:40.043948 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.029441 5024 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 13:08:40.043948 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.029445 5024 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 13:08:40.043948 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.029448 5024 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 13:08:40.043948 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.029450 5024 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 13:08:40.043948 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.029453 5024 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 13:08:40.043948 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.029455 5024 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 13:08:40.043948 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.029458 5024 feature_gate.go:386] unrecognized feature gate: Example Nov 03 13:08:40.043948 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.029460 5024 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 13:08:40.043948 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.029463 5024 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 13:08:40.043948 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.029466 5024 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 13:08:40.043948 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.029468 5024 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 13:08:40.043948 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.029471 5024 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 13:08:40.043948 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.029475 5024 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 13:08:40.043948 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.029478 5024 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 13:08:40.043948 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.029480 5024 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 13:08:40.044546 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.029483 5024 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 13:08:40.044546 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.029485 5024 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 13:08:40.044546 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.029488 5024 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 13:08:40.044546 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.029491 5024 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 13:08:40.044546 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.029494 5024 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 13:08:40.044546 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.029498 5024 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 13:08:40.044546 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.029500 5024 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 13:08:40.044546 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.029503 5024 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 13:08:40.044546 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.029506 5024 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 13:08:40.044546 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.029508 5024 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 13:08:40.044546 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.029511 5024 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 13:08:40.044546 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.029513 5024 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 13:08:40.044546 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.029517 5024 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 13:08:40.044546 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.029520 5024 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 13:08:40.044546 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.029524 5024 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 13:08:40.044546 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.029527 5024 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 13:08:40.044546 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.029529 5024 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 13:08:40.044546 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.029533 5024 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 13:08:40.044546 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.029536 5024 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 13:08:40.045046 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.029543 5024 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 13:08:40.045046 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.029546 5024 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 13:08:40.045046 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.029549 5024 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 13:08:40.045046 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.029551 5024 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 13:08:40.045046 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.029554 5024 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 13:08:40.045046 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.033490 5024 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 03 13:08:40.045046 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.033508 5024 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 03 13:08:40.045046 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.033525 5024 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 03 13:08:40.045046 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.033529 5024 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 03 13:08:40.045046 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.033587 5024 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 13:08:40.045046 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.033594 5024 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 13:08:40.045046 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.033598 5024 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 13:08:40.045046 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.033601 5024 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 13:08:40.045046 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.033604 5024 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 13:08:40.045046 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.033606 5024 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 13:08:40.045046 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.033609 5024 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 13:08:40.045046 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.033612 5024 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 13:08:40.045046 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.033615 5024 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 13:08:40.045046 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.033618 5024 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 13:08:40.045610 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.033621 5024 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 13:08:40.045610 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.033624 5024 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 13:08:40.045610 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.033626 5024 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 13:08:40.045610 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.033629 5024 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 13:08:40.045610 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.033632 5024 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 13:08:40.045610 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.033635 5024 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 13:08:40.045610 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.033638 5024 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 13:08:40.045610 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.033642 5024 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 13:08:40.045610 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.033646 5024 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 13:08:40.045610 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.033649 5024 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 13:08:40.045610 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.033652 5024 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 13:08:40.045610 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.033655 5024 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 13:08:40.045610 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.033657 5024 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 13:08:40.045610 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.033660 5024 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 13:08:40.045610 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.033663 5024 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 13:08:40.045610 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.033666 5024 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 13:08:40.045610 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.033669 5024 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 13:08:40.045610 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.033672 5024 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 13:08:40.045610 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.033675 5024 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 13:08:40.046066 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.033684 5024 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 13:08:40.046066 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.033687 5024 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 13:08:40.046066 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.033690 5024 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 13:08:40.046066 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.033693 5024 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 13:08:40.046066 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.033696 5024 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 13:08:40.046066 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.033698 5024 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 13:08:40.046066 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.033701 5024 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 13:08:40.046066 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.033704 5024 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 13:08:40.046066 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.033707 5024 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 13:08:40.046066 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.033709 5024 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 13:08:40.046066 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.033712 5024 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 13:08:40.046066 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.033715 5024 feature_gate.go:386] unrecognized feature gate: Example Nov 03 13:08:40.046066 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.033718 5024 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 13:08:40.046066 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.033721 5024 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 13:08:40.046066 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.033724 5024 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 13:08:40.046066 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.033726 5024 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 13:08:40.046066 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.033729 5024 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 13:08:40.046066 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.033732 5024 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 13:08:40.046066 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.033734 5024 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 13:08:40.046066 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.033737 5024 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 13:08:40.046667 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.033740 5024 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 13:08:40.046667 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.033742 5024 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 13:08:40.046667 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.033745 5024 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 13:08:40.046667 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.033747 5024 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 13:08:40.046667 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.033750 5024 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 13:08:40.046667 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.033753 5024 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 13:08:40.046667 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.033755 5024 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 13:08:40.046667 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.033758 5024 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 13:08:40.046667 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.033761 5024 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 13:08:40.046667 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.033764 5024 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 13:08:40.046667 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.033766 5024 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 13:08:40.046667 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.033769 5024 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 13:08:40.046667 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.033771 5024 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 13:08:40.046667 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.033779 5024 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 13:08:40.046667 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.033782 5024 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 13:08:40.046667 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.033785 5024 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 13:08:40.046667 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.033788 5024 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 13:08:40.046667 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.033790 5024 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 13:08:40.046667 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.033793 5024 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 13:08:40.047250 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.033795 5024 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 13:08:40.047250 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.033798 5024 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 13:08:40.047250 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.033801 5024 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 13:08:40.047250 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.033803 5024 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 13:08:40.047250 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.033806 5024 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 13:08:40.047250 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.033809 5024 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 13:08:40.047250 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.033811 5024 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 13:08:40.047250 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.033814 5024 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 13:08:40.047250 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.033817 5024 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 13:08:40.047250 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.033819 5024 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 13:08:40.047250 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.033822 5024 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 13:08:40.047250 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.033825 5024 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 13:08:40.047250 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.033827 5024 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 13:08:40.047250 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.033830 5024 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 13:08:40.047250 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.033833 5024 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 13:08:40.047250 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.033835 5024 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 13:08:40.047250 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.033838 5024 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 13:08:40.047250 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.033841 5024 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 13:08:40.047250 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.033843 5024 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 13:08:40.047250 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.033846 5024 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 13:08:40.047965 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.033848 5024 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 13:08:40.047965 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.033851 5024 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 13:08:40.047965 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.033853 5024 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 13:08:40.047965 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.033856 5024 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 13:08:40.047965 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.033859 5024 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 13:08:40.047965 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.033861 5024 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 13:08:40.047965 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.033864 5024 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 13:08:40.047965 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.033873 5024 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 13:08:40.047965 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.033876 5024 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 13:08:40.047965 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.033879 5024 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 13:08:40.047965 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.033881 5024 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 13:08:40.047965 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.033884 5024 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 13:08:40.047965 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.033886 5024 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 13:08:40.047965 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.033889 5024 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 13:08:40.047965 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.033892 5024 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 13:08:40.047965 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.033894 5024 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 13:08:40.047965 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.033897 5024 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 13:08:40.047965 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.033899 5024 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 13:08:40.048589 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.033926 5024 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 13:08:40.048589 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.034109 5024 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 13:08:40.048589 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.034113 5024 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 13:08:40.048589 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.034116 5024 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 13:08:40.048589 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.034119 5024 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 13:08:40.048589 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.034122 5024 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 13:08:40.048589 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.034124 5024 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 13:08:40.048589 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.034127 5024 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 13:08:40.048589 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.034130 5024 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 13:08:40.048589 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.034132 5024 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 13:08:40.048589 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.034135 5024 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 13:08:40.048589 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.034137 5024 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 13:08:40.048589 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.034140 5024 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 13:08:40.048589 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.034143 5024 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 13:08:40.048589 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.034145 5024 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 13:08:40.048589 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.034148 5024 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 13:08:40.049105 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.034150 5024 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 13:08:40.049105 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.034153 5024 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 13:08:40.049105 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.034155 5024 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 13:08:40.049105 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.034158 5024 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 13:08:40.049105 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.034161 5024 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 13:08:40.049105 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.034163 5024 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 13:08:40.049105 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.034173 5024 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 13:08:40.049105 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.034176 5024 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 13:08:40.049105 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.034178 5024 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 13:08:40.049105 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.034181 5024 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 13:08:40.049105 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.034183 5024 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 13:08:40.049105 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.034186 5024 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 13:08:40.049105 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.034189 5024 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 13:08:40.049105 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.034192 5024 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 13:08:40.049105 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.034194 5024 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 13:08:40.049105 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.034197 5024 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 13:08:40.049105 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.034200 5024 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 13:08:40.049105 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.034202 5024 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 13:08:40.049105 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.034205 5024 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 13:08:40.049105 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.034207 5024 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 13:08:40.049602 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.034210 5024 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 13:08:40.049602 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.034212 5024 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 13:08:40.049602 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.034216 5024 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 13:08:40.049602 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.034218 5024 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 13:08:40.049602 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.034221 5024 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 13:08:40.049602 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.034223 5024 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 13:08:40.049602 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.034226 5024 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 13:08:40.049602 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.034229 5024 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 13:08:40.049602 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.034231 5024 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 13:08:40.049602 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.034234 5024 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 13:08:40.049602 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.034236 5024 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 13:08:40.049602 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.034239 5024 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 13:08:40.049602 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.034241 5024 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 13:08:40.049602 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.034244 5024 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 13:08:40.049602 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.034247 5024 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 13:08:40.049602 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.034249 5024 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 13:08:40.049602 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.034252 5024 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 13:08:40.049602 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.034254 5024 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 13:08:40.049602 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.034257 5024 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 13:08:40.049602 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.034265 5024 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 13:08:40.050088 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.034268 5024 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 13:08:40.050088 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.034272 5024 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 13:08:40.050088 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.034276 5024 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 13:08:40.050088 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.034279 5024 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 13:08:40.050088 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.034296 5024 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 13:08:40.050088 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.034300 5024 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 13:08:40.050088 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.034302 5024 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 13:08:40.050088 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.034305 5024 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 13:08:40.050088 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.034308 5024 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 13:08:40.050088 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.034310 5024 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 13:08:40.050088 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.034313 5024 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 13:08:40.050088 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.034316 5024 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 13:08:40.050088 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.034318 5024 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 13:08:40.050088 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.034321 5024 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 13:08:40.050088 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.034324 5024 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 13:08:40.050088 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.034326 5024 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 13:08:40.050088 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.034331 5024 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 13:08:40.050088 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.034334 5024 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 13:08:40.050088 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.034337 5024 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 13:08:40.050637 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.034339 5024 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 13:08:40.050637 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.034348 5024 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 13:08:40.050637 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.034351 5024 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 13:08:40.050637 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.034354 5024 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 13:08:40.050637 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.034357 5024 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 13:08:40.050637 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.034359 5024 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 13:08:40.050637 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.034362 5024 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 13:08:40.050637 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.034364 5024 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 13:08:40.050637 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.034368 5024 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 13:08:40.050637 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.034371 5024 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 13:08:40.050637 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.034373 5024 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 13:08:40.050637 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.034377 5024 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 13:08:40.050637 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.034381 5024 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 13:08:40.050637 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.034390 5024 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 13:08:40.050637 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.034393 5024 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 13:08:40.050637 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.034396 5024 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 13:08:40.050637 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.034399 5024 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 13:08:40.050637 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.034401 5024 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 13:08:40.050637 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.034404 5024 feature_gate.go:386] unrecognized feature gate: Example Nov 03 13:08:40.051118 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.034407 5024 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 13:08:40.051118 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.034409 5024 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 13:08:40.051118 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.034412 5024 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 13:08:40.051118 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.034414 5024 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 13:08:40.051118 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.034417 5024 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 13:08:40.051118 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.034420 5024 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 13:08:40.051118 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.034422 5024 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 13:08:40.051118 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.034425 5024 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 13:08:40.051118 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.034427 5024 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 13:08:40.051118 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.034430 5024 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 13:08:40.051118 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.034434 5024 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 13:08:40.051118 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.034437 5024 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 13:08:40.051118 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.034440 5024 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 13:08:40.051118 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.034469 5024 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 13:08:40.051118 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.034509 5024 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 13:08:40.051118 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.034513 5024 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 13:08:40.051534 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.034517 5024 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 13:08:40.051534 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.034520 5024 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 13:08:40.051534 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.034523 5024 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 13:08:40.051534 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.034526 5024 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 13:08:40.051534 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.034528 5024 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 13:08:40.051534 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.034531 5024 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 13:08:40.051534 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.034534 5024 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 13:08:40.051534 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.034536 5024 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 13:08:40.051534 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.034539 5024 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 13:08:40.051534 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.034542 5024 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 13:08:40.051534 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.034544 5024 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 13:08:40.051534 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.034554 5024 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 13:08:40.051534 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.034557 5024 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 13:08:40.051534 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.034560 5024 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 13:08:40.051534 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.034562 5024 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 13:08:40.051534 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.034565 5024 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 13:08:40.051534 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.034567 5024 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 13:08:40.051534 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.034570 5024 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 13:08:40.051534 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.034573 5024 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 13:08:40.051534 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.034576 5024 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 13:08:40.052017 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.034578 5024 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 13:08:40.052017 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.034582 5024 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 13:08:40.052017 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.034586 5024 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 13:08:40.052017 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.034590 5024 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 13:08:40.052017 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.034592 5024 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 13:08:40.052017 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.034596 5024 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 13:08:40.052017 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.034601 5024 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 13:08:40.052017 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.034604 5024 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 13:08:40.052017 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.034607 5024 feature_gate.go:386] unrecognized feature gate: Example Nov 03 13:08:40.052017 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.034609 5024 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 13:08:40.052017 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.034612 5024 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 13:08:40.052017 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.034614 5024 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 13:08:40.052017 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.034617 5024 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 13:08:40.052017 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.034620 5024 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 13:08:40.052017 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.034622 5024 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 13:08:40.052017 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.034625 5024 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 13:08:40.052017 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.034627 5024 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 13:08:40.052017 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.034630 5024 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 13:08:40.052017 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.034632 5024 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 13:08:40.052518 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.034635 5024 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 13:08:40.052518 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.034638 5024 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 13:08:40.052518 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.034640 5024 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 13:08:40.052518 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.034643 5024 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 13:08:40.052518 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.034645 5024 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 13:08:40.052518 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.034654 5024 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 13:08:40.052518 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.034657 5024 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 13:08:40.052518 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.034659 5024 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 13:08:40.052518 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.034662 5024 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 13:08:40.052518 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.034664 5024 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 13:08:40.052518 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.034667 5024 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 13:08:40.052518 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.034669 5024 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 13:08:40.052518 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.034672 5024 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 13:08:40.052518 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.034674 5024 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 13:08:40.052518 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.034677 5024 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 13:08:40.052518 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.034679 5024 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 13:08:40.052518 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.034682 5024 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 13:08:40.052518 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.034684 5024 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 13:08:40.052518 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.034688 5024 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 13:08:40.052518 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.034692 5024 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 13:08:40.053047 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.034695 5024 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 13:08:40.053047 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.034698 5024 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 13:08:40.053047 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.034700 5024 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 13:08:40.053047 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.034703 5024 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 13:08:40.053047 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.034706 5024 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 13:08:40.053047 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.034708 5024 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 13:08:40.053047 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.034711 5024 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 13:08:40.053047 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.034714 5024 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 13:08:40.053047 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.034716 5024 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 13:08:40.053047 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.034719 5024 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 13:08:40.053047 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.034721 5024 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 13:08:40.053047 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.034724 5024 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 13:08:40.053047 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.034727 5024 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 13:08:40.053047 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.034729 5024 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 13:08:40.053047 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.034732 5024 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 13:08:40.053047 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.034734 5024 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 13:08:40.053047 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.034737 5024 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 13:08:40.053047 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.034739 5024 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 13:08:40.053047 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.034748 5024 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 13:08:40.053047 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.034750 5024 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 13:08:40.053562 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.034753 5024 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 13:08:40.053562 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.034756 5024 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 13:08:40.053562 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.034758 5024 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 13:08:40.053562 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.034761 5024 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 13:08:40.053562 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.034764 5024 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 13:08:40.053562 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.034767 5024 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 13:08:40.053562 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.034770 5024 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 13:08:40.053562 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.034772 5024 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 13:08:40.053562 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.034775 5024 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 13:08:40.053562 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.034778 5024 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 13:08:40.053562 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.034783 5024 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 13:08:40.053562 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.034787 5024 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 13:08:40.053562 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.034790 5024 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 13:08:40.053562 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.034792 5024 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 13:08:40.053562 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.034795 5024 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 13:08:40.053562 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.034798 5024 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 13:08:40.053562 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.034800 5024 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 13:08:40.053562 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.034803 5024 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 13:08:40.053562 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.034806 5024 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 13:08:40.054443 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.034810 5024 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 13:08:40.054443 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.034813 5024 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 13:08:40.054443 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.034816 5024 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 13:08:40.054443 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.034820 5024 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 13:08:40.054443 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.034823 5024 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 13:08:40.054443 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:40.034825 5024 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 13:08:40.054443 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.034885 5024 server.go:957] "Client rotation is on, will bootstrap in background" Nov 03 13:08:40.054443 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.035887 5024 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Nov 03 13:08:40.054443 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.035946 5024 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Nov 03 13:08:40.054443 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.036151 5024 server.go:1014] "Starting client certificate rotation" Nov 03 13:08:40.054443 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.036299 5024 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 03 13:08:40.054443 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.036346 5024 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-02 13:02:45 +0000 UTC" deadline="2028-05-16 06:52:48.081523312 +0000 UTC" Nov 03 13:08:40.054443 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.036385 5024 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13433h44m8.045140446s" Nov 03 13:08:40.054443 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.036694 5024 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 13:08:40.054443 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.037901 5024 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 13:08:40.054443 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.040594 5024 log.go:25] "Validated CRI v1 runtime API" Nov 03 13:08:40.055105 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.042544 5024 log.go:25] "Validated CRI v1 image API" Nov 03 13:08:40.055105 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.043302 5024 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 03 13:08:40.055105 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.045430 5024 fs.go:135] Filesystem UUIDs: map[11ac45d8-f758-4da3-834f-f7d549fb1720:/dev/nvme0n1p4 6780481b-d8e9-45eb-9edc-0ebc30424bb0:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2] Nov 03 13:08:40.055258 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.045451 5024 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/080b34934d5649c436eb755c96cb75220add975b57d1a9023728b6782facbc86/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/080b34934d5649c436eb755c96cb75220add975b57d1a9023728b6782facbc86/userdata/shm major:0 minor:74 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/12c34fc486ae00af6cdac52a0c622b9549f772c0e690148eea53070e65b1da62/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/12c34fc486ae00af6cdac52a0c622b9549f772c0e690148eea53070e65b1da62/userdata/shm major:0 minor:234 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1658c9a5bbf6b90cd5b0c2c5ef8bc57b57b5e38c2ebde6e79097b6a75ad221c5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1658c9a5bbf6b90cd5b0c2c5ef8bc57b57b5e38c2ebde6e79097b6a75ad221c5/userdata/shm major:0 minor:238 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/20af2de0975a5d8d3f6cee427d3f37295edf7f21d547d66bb589a540ecc17985/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/20af2de0975a5d8d3f6cee427d3f37295edf7f21d547d66bb589a540ecc17985/userdata/shm major:0 minor:296 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2714947ff73646c1e92b308d87f7aa0ae91cd24f08a9ed96681ba260c6f6d555/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2714947ff73646c1e92b308d87f7aa0ae91cd24f08a9ed96681ba260c6f6d555/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/32b60dc42f674361e6ee2e8b8778e572fd8fc785139362be592e97abee98664a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/32b60dc42f674361e6ee2e8b8778e572fd8fc785139362be592e97abee98664a/userdata/shm major:0 minor:70 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/39aa5c2c75a1203c58dd67810fc2d1718f5535ce2d67634692f6a034d4a8fe41/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/39aa5c2c75a1203c58dd67810fc2d1718f5535ce2d67634692f6a034d4a8fe41/userdata/shm major:0 minor:78 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3cf1dc4f654fe1eddf341ba01ad4d52e81aac0f2f4a81246a19b59883c62f85a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3cf1dc4f654fe1eddf341ba01ad4d52e81aac0f2f4a81246a19b59883c62f85a/userdata/shm major:0 minor:244 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/48ba5bc216484088efd6cef6f8687c4da1e5abc695dc4f3df755ca6977ace6b0/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/48ba5bc216484088efd6cef6f8687c4da1e5abc695dc4f3df755ca6977ace6b0/userdata/shm major:0 minor:294 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5b44ee172423f97b68d439f1263e56f195b4502d78f9d58d2899e3b7eee3152f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5b44ee172423f97b68d439f1263e56f195b4502d78f9d58d2899e3b7eee3152f/userdata/shm major:0 minor:49 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5d20773640ce8e198ac75606401c820525fb0fe1350882ef0b9b2ed7d166824a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5d20773640ce8e198ac75606401c820525fb0fe1350882ef0b9b2ed7d166824a/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5ea2ec30de3f7897100ae5923cb74d11ffc40a71885605a5855258979863426c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5ea2ec30de3f7897100ae5923cb74d11ffc40a71885605a5855258979863426c/userdata/shm major:0 minor:240 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/68fe3802481c8f7a3638809cd85a097698ee57db88bc75d6ddd15414cd29f9b5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/68fe3802481c8f7a3638809cd85a097698ee57db88bc75d6ddd15414cd29f9b5/userdata/shm major:0 minor:232 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6983b50836f972f88268b1ac5ff771dc019eeb5a3416d32647e8046e0e92e2d0/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6983b50836f972f88268b1ac5ff771dc019eeb5a3416d32647e8046e0e92e2d0/userdata/shm major:0 minor:86 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/84fabe8a79acb6040874b2b7e92579078c1dff682320fea274fd83a4f027c1ea/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/84fabe8a79acb6040874b2b7e92579078c1dff682320fea274fd83a4f027c1ea/userdata/shm major:0 minor:228 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a15ed7253699ac60f5504c1ca9afee7dc425b76d3113908f1e8d7116db36f547/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a15ed7253699ac60f5504c1ca9afee7dc425b76d3113908f1e8d7116db36f547/userdata/shm major:0 minor:336 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a670effa2a69f2c5a5f3625db93d19af7b21e7237cd4c41a16088e66cb4104a2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a670effa2a69f2c5a5f3625db93d19af7b21e7237cd4c41a16088e66cb4104a2/userdata/shm major:0 minor:229 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a8456e348c89166330071974486e35de81374e53475426213db288f247638ed1/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a8456e348c89166330071974486e35de81374e53475426213db288f247638ed1/userdata/shm major:0 minor:236 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ab29b318c08555358cf838590f8909d568ac39d5a7ad9c2cbd6830c22a1c9f52/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ab29b318c08555358cf838590f8909d568ac39d5a7ad9c2cbd6830c22a1c9f52/userdata/shm major:0 minor:246 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b60ab4a7af01474429ab8f251670ff485fabd1f7d74bcd699a42190bfa711191/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b60ab4a7af01474429ab8f251670ff485fabd1f7d74bcd699a42190bfa711191/userdata/shm major:0 minor:242 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c29cb1f6582414bd2e24af736205bd2de388faaebcd65096615b22499cfdaf05/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c29cb1f6582414bd2e24af736205bd2de388faaebcd65096615b22499cfdaf05/userdata/shm major:0 minor:82 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e6c2867616afcbdf7d2a5611abc303a745fe5f9dece80d4859a108a05d8f4438/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e6c2867616afcbdf7d2a5611abc303a745fe5f9dece80d4859a108a05d8f4438/userdata/shm major:0 minor:92 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ecf7ff9bffd580730bcdf842700eb81895af4e3d420ec626d39da9261adba3ee/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ecf7ff9bffd580730bcdf842700eb81895af4e3d420ec626d39da9261adba3ee/userdata/shm major:0 minor:95 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/021e9e00-a973-4726-8f48-02a52c4bae14/volumes/kubernetes.io~projected/kube-api-access-qcsxn:{mountpoint:/var/lib/kubelet/pods/021e9e00-a973-4726-8f48-02a52c4bae14/volumes/kubernetes.io~projected/kube-api-access-qcsxn major:0 minor:211 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/07a7e2de-0c09-4819-9670-2be9460a155d/volumes/kubernetes.io~projected/kube-api-access-7jxlt:{mountpoint:/var/lib/kubelet/pods/07a7e2de-0c09-4819-9670-2be9460a155d/volumes/kubernetes.io~projected/kube-api-access-7jxlt major:0 minor:205 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0a892eb3-ecc4-4312-9f4d-73eeba6b6eee/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/0a892eb3-ecc4-4312-9f4d-73eeba6b6eee/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:226 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0a892eb3-ecc4-4312-9f4d-73eeba6b6eee/volumes/kubernetes.io~projected/kube-api-access-xmbd7:{mountpoint:/var/lib/kubelet/pods/0a892eb3-ecc4-4312-9f4d-73eeba6b6eee/volumes/kubernetes.io~projected/kube-api-access-xmbd7 major:0 minor:220 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0a892eb3-ecc4-4312-9f4d-73eeba6b6eee/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/0a892eb3-ecc4-4312-9f4d-73eeba6b6eee/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:217 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0a892eb3-ecc4-4312-9f4d-73eeba6b6eee/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/0a892eb3-ecc4-4312-9f4d-73eeba6b6eee/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:215 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1bfc1afc-5b9f-4447-8d40-fabedbee9c07/volumes/kubernetes.io~projected/kube-api-access-hbfgb:{mountpoint:/var/lib/kubelet/pods/1bfc1afc-5b9f-4447-8d40-fabedbee9c07/volumes/kubernetes.io~projected/kube-api-access-hbfgb major:0 minor:288 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/21618783-0a4e-4b1f-8041-cabdf69f5bea/volumes/kubernetes.io~projected/kube-api-access-vbs5f:{mountpoint:/var/lib/kubelet/pods/21618783-0a4e-4b1f-8041-cabdf69f5bea/volumes/kubernetes.io~projected/kube-api-access-vbs5f major:0 minor:223 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/246971bc-ed35-4fc7-b5f8-941327b90a94/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/246971bc-ed35-4fc7-b5f8-941327b90a94/volumes/kubernetes.io~secret/agent-certs major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/35e0fb40-48ce-4223-b7d4-c9290b6861cd/volumes/kubernetes.io~projected/kube-api-access-mss42:{mountpoint:/var/lib/kubelet/pods/35e0fb40-48ce-4223-b7d4-c9290b6861cd/volumes/kubernetes.io~projected/kube-api-access-mss42 major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/39e11989-e598-4cea-87d9-26b3cf78a66b/volumes/kubernetes.io~projected/kube-api-access-zpq8b:{mountpoint:/var/lib/kubelet/pods/39e11989-e598-4cea-87d9-26b3cf78a66b/volumes/kubernetes.io~projected/kube-api-access-zpq8b major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3fdeb9f3-6c67-4390-8073-17d4d5f802b9/volumes/kubernetes.io~projected/kube-api-access-bddlp:{mountpoint:/var/lib/kubelet/pods/3fdeb9f3-6c67-4390-8073-17d4d5f802b9/volumes/kubernetes.io~projected/kube-api-access-bddlp major:0 minor:225 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/50d68cb7-b766-4674-b295-8f3229880994/volumes/kubernetes.io~projected/kube-api-access-2sxmr:{mountpoint:/var/lib/kubelet/pods/50d68cb7-b766-4674-b295-8f3229880994/volumes/kubernetes.io~projected/kube-api-access-2sxmr major:0 minor:222 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/50d68cb7-b766-4674-b295-8f3229880994/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/50d68cb7-b766-4674-b295-8f3229880994/volumes/kubernetes.io~secret/cert major:0 minor:219 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5c17f5bf-5b7d-4b05-a387-21ac6b5186ca/volumes/kubernetes.io~projected/kube-api-access-k9fhg:{mountpoint:/var/lib/kubelet/pods/5c17f5bf-5b7d-4b05-a387-21ac6b5186ca/volumes/kubernetes.io~projected/kube-api-access-k9fhg major:0 minor:224 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5ea5cf61-ed1c-4f5e-b1a0-5c46c5c3ee18/volumes/kubernetes.io~projected/kube-api-access-vjhjt:{mountpoint:/var/lib/kubelet/pods/5ea5cf61-ed1c-4f5e-b1a0-5c46c5c3ee18/volumes/kubernetes.io~projected/kube-api-access-vjhjt major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6cb478ab-97aa-4881-87b2-b45293cc2349/volumes/kubernetes.io~projected/kube-api-access-x7mw6:{mountpoint:/var/lib/kubelet/pods/6cb478ab-97aa-4881-87b2-b45293cc2349/volumes/kubernetes.io~projected/kube-api-access-x7mw6 major:0 minor:213 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6cb478ab-97aa-4881-87b2-b45293cc2349/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/6cb478ab-97aa-4881-87b2-b45293cc2349/volumes/kubernetes.io~secret/serving-cert major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/74dc4aee-10b1-463c-a9c3-522b7a6c10b7/volumes/kubernetes.io~projected/kube-api-access-cwrs7:{mountpoint:/var/lib/kubelet/pods/74dc4aee-10b1-463c-a9c3-522b7a6c10b7/volumes/kubernetes.io~projected/kube-api-access-cwrs7 major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7e280277-f29f-4a2b-8b85-2491529e5ef2/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/7e280277-f29f-4a2b-8b85-2491529e5ef2/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:293 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/898bd0b5-0d3e-4577-b101-2cc7d75283cb/volumes/kubernetes.io~projected/kube-api-access-bzlml:{mountpoint:/var/lib/kubelet/pods/898bd0b5-0d3e-4577-b101-2cc7d75283cb/volumes/kubernetes.io~projected/kube-api-access-bzlml major:0 minor:227 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/898bd0b5-0d3e-4577-b101-2cc7d75283cb/volumes/kubernetes.io~secret/default-certificate:{mountpoint:/var/lib/kubelet/pods/898bd0b5-0d3e-4577-b101-2cc7d75283cb/volumes/kubernetes.io~secret/default-certificate major:0 minor:214 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/898bd0b5-0d3e-4577-b101-2cc7d75283cb/volumes/kubernetes.io~secret/stats-auth:{mountpoint:/var/lib/kubelet/pods/898bd0b5-0d3e-4577-b101-2cc7d75283cb/volumes/kubernetes.io~secret/stats-auth major:0 minor:218 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9db7d195-220e-4644-8690-9e5ee37fc066/volumes/kubernetes.io~projected/kube-api-access-ftcqc:{mountpoint:/var/lib/kubelet/pods/9db7d195-220e-4644-8690-9e5ee37fc066/volumes/kubernetes.io~projected/kube-api-access-ftcqc major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9f20ddda-41fb-4aba-b257-7ebc59dbe678/volumes/kubernetes.io~projected/kube-api-access-jknsv:{mountpoint:/var/lib/kubelet/pods/9f20ddda-41fb-4aba-b257-7ebc59dbe678/volumes/kubernetes.io~projected/kube-api-access-jknsv major:0 minor:221 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9f20ddda-41fb-4aba-b257-7ebc59dbe678/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/9f20ddda-41fb-4aba-b257-7ebc59dbe678/volumes/kubernetes.io~secret/serving-cert major:0 minor:216 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9fa6d32d-b30b-4b76-9462-5f3f09443d02/volumes/kubernetes.io~projected/kube-api-access-n8x9b:{mountpoint:/var/lib/kubelet/pods/9fa6d32d-b30b-4b76-9462-5f3f09443d02/volumes/kubernetes.io~projected/kube-api-access-n8x9b major:0 minor:209 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9fa6d32d-b30b-4b76-9462-5f3f09443d02/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/9fa6d32d-b30b-4b76-9462-5f3f09443d02/volumes/kubernetes.io~secret/serving-cert major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a3042b60-062f-4f9b-8406-f884c95a8729/volumes/kubernetes.io~projected/kube-api-access-4lj56:{mountpoint:/var/lib/kubelet/pods/a3042b60-062f-4f9b-8406-f884c95a8729/volumes/kubernetes.io~projected/kube-api-access-4lj56 major:0 minor:206 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b8fe3220-d285-487f-9fc0-597438982de7/volumes/kubernetes.io~projected/kube-api-access-zhwtz:{mountpoint:/var/lib/kubelet/pods/b8fe3220-d285-487f-9fc0-597438982de7/volumes/kubernetes.io~projected/kube-api-access-zhwtz major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bd523616-71b2-4b6c-b82c-f1efb585c271/volumes/kubernetes.io~projected/kube-api-access-mtd8p:{mountpoint:/var/lib/kubelet/pods/bd523616-71b2-4b6c-b82c-f1efb585c271/volumes/kubernetes.io~projected/kube-api-access-mtd8p major:0 minor:329 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c704eb02-d156-4695-a55d-7dc96654e4cd/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/c704eb02-d156-4695-a55d-7dc96654e4cd/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c704eb02-d156-4695-a55d-7dc96654e4cd/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/c704eb02-d156-4695-a55d-7dc96654e4cd/volumes/kubernetes.io~empty-dir/tmp major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c704eb02-d156-4695-a55d-7dc96654e4cd/volumes/kubernetes.io~projected/kube-api-access-w4j5g:{mountpoint:/var/lib/kubelet/pods/c704eb02-d156-4695-a55d-7dc96654e4cd/volumes/kubernetes.io~projected/kube-api-access-w4j5g major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c82d9f4b-8c00-4c31-8aa2-99cdc0d87e52/volumes/kubernetes.io~projected/kube-api-access-cb5g7:{mountpoint:/var/lib/kubelet/pods/c82d9f4b-8c00-4c31-8aa2-99cdc0d87e52/volumes/kubernetes.io~projected/kube-api-access-cb5g7 major:0 minor:207 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cc8ef30e-6d4f-4913-8dc8-b09e5aea2f82/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/cc8ef30e-6d4f-4913-8dc8-b09e5aea2f82/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:212 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cc8ef30e-6d4f-4913-8dc8-b09e5aea2f82/volumes/kubernetes.io~projected/kube-api-access-h82mn:{mountpoint:/var/lib/kubelet/pods/cc8ef30e-6d4f-4913-8dc8-b09e5aea2f82/volumes/kubernetes.io~projected/kube-api-access-h82mn major:0 minor:208 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cc8ef30e-6d4f-4913-8dc8-b09e5aea2f82/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/cc8ef30e-6d4f-4913-8dc8-b09e5aea2f82/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cc8ef30e-6d4f-4913-8dc8-b09e5aea2f82/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/cc8ef30e-6d4f-4913-8dc8-b09e5aea2f82/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cd91de08-084e-4667-b13f-2e07b3cff624/volumes/kubernetes.io~projected/kube-api-access-pt6qk:{mountpoint:/var/lib/kubelet/pods/cd91de08-084e-4667-b13f-2e07b3cff624/volumes/kubernetes.io~projected/kube-api-access-pt6qk major:0 minor:210 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cd91de08-084e-4667-b13f-2e07b3cff624/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/cd91de08-084e-4667-b13f-2e07b3cff624/volumes/kubernetes.io~secret/serving-cert major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d926d945-39f4-48f3-b3a4-0b302b15b69b/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/d926d945-39f4-48f3-b3a4-0b302b15b69b/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d926d945-39f4-48f3-b3a4-0b302b15b69b/volumes/kubernetes.io~projected/kube-api-access-6b28l:{mountpoint:/var/lib/kubelet/pods/d926d945-39f4-48f3-b3a4-0b302b15b69b/volumes/kubernetes.io~projected/kube-api-access-6b28l major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d926d945-39f4-48f3-b3a4-0b302b15b69b/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/d926d945-39f4-48f3-b3a4-0b302b15b69b/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e0300f55-f002-4faa-bf6a-c04bea0a043e/volumes/kubernetes.io~projected/kube-api-access-bcz25:{mountpoint:/var/lib/kubelet/pods/e0300f55-f002-4faa-bf6a-c04bea0a043e/volumes/kubernetes.io~projected/kube-api-access-bcz25 major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fd7e8b58-de31-406a-8cdb-9ccfee140d57/volumes/kubernetes.io~projected/kube-api-access-snpf4:{mountpoint:/var/lib/kubelet/pods/fd7e8b58-de31-406a-8cdb-9ccfee140d57/volumes/kubernetes.io~projected/kube-api-access-snpf4 major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fff35d1b-e7b2-4dd3-9824-8b64fad02506/volumes/kubernetes.io~projected/kube-api-access-jssfx:{mountpoint:/var/lib/kubelet/pods/fff35d1b-e7b2-4dd3-9824-8b64fad02506/volumes/kubernetes.io~projected/kube-api-access-jssfx major:0 minor:310 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/0113d7945c38d5a90d7e8ac63810d7a29b7cc9f008108050bfcc91ee97532322/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/e9220517412905f858b64a52e44449b92db79c23128f35970c2b1b10986839db/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/00b809a6b7246627ad60bef055e179429781004497e95e10ff3aad25fe508fd6/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/97c698ed54835c2a0700945fae82b9f0bdcaed19df368c77b970fa5c2f51964c/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/15d446cef0ab08d11bba6709845297818177dce082a795ad0383d9d0264b517e/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/c04dca677d2f964a8a2751b85386902dfb6e206357890ffd2b12ced50c966030/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/038dd52c896c42decedd367ea11930bf8beef2484a5d18e65b62edcc6f0cd824/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/52702056af6974fa309e15019ad0357fbae5a0ce59541c3ddcb72e2821406cde/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-131:{mountpoint:/var/lib/containers/storage/overlay/50ad42a4bcf63f08f967b263f656d5a34db5f26a52d6893b91b94f0c9b882d3f/merged major:0 minor:131 fsType:overlay blockSize:0} overlay_0-134:{mountpoint:/var/lib/containers/storage/overlay/fe4cb3933630ebce1e6b698bfe1c48792d9b7f26f7ffb3bdd222d8ed7934c6a3/merged major:0 minor:134 fsType:overlay blockSize:0} overlay_0-139:{mountpoint:/var/lib/containers/storage/overlay/5f9850ef300f8dae36e6585caf93a945652810439c221cee3c28ebc0584b5cbc/merged major:0 minor:139 fsType:overlay blockSize:0} overlay_0-143:{mountpoint:/var/lib/containers/storage/overlay/ee2cc6585603879a33a344bc99d93cd78741ac68825eea0c423ed4a94ad3c5a3/merged major:0 minor:143 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/d1ac04eb9d5cca14966aee27511d4c8a52ab277fdc0efc3b207eabdd593392df/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-154:{mountpoint:/var/lib/containers/storage/overlay/af0c5335dc69b7fe29378c224b0a9d7c77382183a0cece9d195cb1b393a05617/merged major:0 minor:154 fsType:overlay blockSize:0} overlay_0-159:{mountpoint:/var/lib/containers/storage/overlay/05fe448ff1b199608a60da83343dfd46ed08196f219847cbc1aa2cfcbc28636b/merged major:0 minor:159 fsType:overlay blockSize:0} overlay_0-163:{mountpoint:/var/lib/containers/storage/overlay/dc4ae5b7b6ba6c119664dc43f12340d0963f041d0a072bd0bba50bd6c764eb44/merged major:0 minor:163 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/3dbff9489d902718a916f937cc39b95e18c4d69fcbac421ca7a9eb6b8fa5e880/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/0116a463a8778b471181742e83fc26783fd77026821b6233268353d7b3983ef7/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/25ccca6ed3d19913829165d2c6eb440c1ea27b4d2ee58ce3b27af24b373b6b6a/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/b6978ee221963e64ff0abde044493f389ff97cac93b29bbb447920db14e7f69f/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/a08a7c104c7ab3bc729820283ef05fb5ca0978184e43aa7619c6a0b6aafba149/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-248:{mountpoint:/var/lib/containers/storage/overlay/d35be4bd6733d84bd2883053f3c4c5747f218731e85cb4e01ebf6bf61e1173da/merged major:0 minor:248 fsType:overlay blockSize:0} overlay_0-250:{mountpoint:/var/lib/containers/storage/overlay/7ea6a957c759b29eb4581f76ad9aa84c85881f422e48a664d61a9943b038b4a0/merged major:0 minor:250 fsType:overlay blockSize:0} overlay_0-252:{mountpoint:/var/lib/containers/storage/overlay/8662754f4cffc630a2479a4493ccddf6e8017f37f3e2cc58ccecda363080bf32/merged major:0 minor:252 fsType:overlay blockSize:0} overlay_0-254:{mountpoint:/var/lib/containers/storage/overlay/8eb28f03c06215ba1df1f991c8341e9ed5d0ab05e202764ed88db622b74a9f8e/merged major:0 minor:254 fsType:overlay blockSize:0} overlay_0-256:{mountpoint:/var/lib/containers/storage/overlay/06c680c07515538fcc7fa2ec6a56315258458005665f4c552cc06d2cb367fc9d/merged major:0 minor:256 fsType:overlay blockSize:0} overlay_0-258:{mountpoint:/var/lib/containers/storage/overlay/fdc98fbc7ec97706e2c9eb8df9a5db8686afa33c16cb4ba189297aa6c3761e46/merged major:0 minor:258 fsType:overlay blockSize:0} overlay_0-260:{mountpoint:/var/lib/containers/storage/overlay/387a89a58c6c3b8bb0443685dded7c0dccc5b415f08885b68792134c09940372/merged major:0 minor:260 fsType:overlay blockSize:0} overlay_0-262:{mountpoint:/var/lib/containers/storage/overlay/d215666962afc12ca38a1badd05cf13b626ab4d1549db432fb5b80a60dcc5643/merged major:0 minor:262 fsType:overlay blockSize:0} overlay_0-264:{mountpoint:/var/lib/containers/storage/overlay/91fb3ef5d306d4f51c2a17fa1f43656b0c1da219cb5283e6522ec73a74d61be5/merged major:0 minor:264 fsType:overlay blockSize:0} overlay_0-266:{mountpoint:/var/lib/containers/storage/overlay/8c0af3361b0473bbe3582bec5ba9eb3ba3fbf3ea25e65c7f49c09bb774788ee1/merged major:0 minor:266 fsType:overlay blockSize:0} overlay_0-268:{mountpoint:/var/lib/containers/storage/overlay/285712f9033dcb1285394d1c0558222afa00d8de4011146667dcd017e0d522b0/merged major:0 minor:268 fsType:overlay blockSize:0} overlay_0-270:{mountpoint:/var/lib/containers/storage/overlay/dcaacf5afaa1285bda35f535caf8763441b291935b9f281df85c035e62b98758/merged major:0 minor:270 fsType:overlay blockSize:0} overlay_0-283:{mountpoint:/var/lib/containers/storage/overlay/a35386bc37cf8086d91d44ca53fb7ff6d3e6343d2d771165bd9ebc7ae729bc8c/merged major:0 minor:283 fsType:overlay blockSize:0} overlay_0-285:{mountpoint:/var/lib/containers/storage/overlay/2d22e5ed2c19513ca5801f080d38902e30a8b15bbcd16b8317631ea1195aaf7f/merged major:0 minor:285 fsType:overlay blockSize:0} overlay_0-298:{mountpoint:/var/lib/containers/storage/overlay/dcffa1f908dd191ea90c41cebb7f2797c52903e500c8b73a21c069a558b98caf/merged major:0 minor:298 fsType:overlay blockSize:0} overlay_0-300:{mountpoint:/var/lib/containers/storage/overlay/715706984ec0be2160e17c8fe9ccab066c713f9fa31d76447dcc44ad8e227f9e/merged major:0 minor:300 fsType:overlay blockSize:0} overlay_0-302:{mountpoint:/var/lib/containers/storage/overlay/3739dc334a256b38fcd3e0cc80f43ae9fdcdc5ebf608986a71cb5c3136cae191/merged major:0 minor:302 fsType:overlay blockSize:0} overlay_0-304:{mountpoint:/var/lib/containers/storage/overlay/f7179233858e364d7fd7b3588a3edd66226683e71d542f0bc9c13fc142c29d5d/merged major:0 minor:304 fsType:overlay blockSize:0} overlay_0-315:{mountpoint:/var/lib/containers/storage/overlay/401335f15256770d3dd317c9ee34dd678d619ed0ca90022a52af3972484accc9/merged major:0 minor:315 fsType:overlay blockSize:0} overlay_0-317:{mountpoint:/var/lib/containers/storage/overlay/ec60d64eb1bdd472db77ed4f6046e6e5a32b99ab8667f7911b8d27192c851582/merged major:0 minor:317 fsType:overlay blockSize:0} overlay_0-323:{mountpoint:/var/lib/containers/storage/overlay/5ac59e72cdc849a3c28f05b3787918b248e441c49a608731704397f05c041991/merged major:0 minor:323 fsType:overlay blockSize:0} overlay_0-334:{mountpoint:/var/lib/containers/storage/overlay/a2ab626e59c9fb9f06ee682c273232055d7b0b01e81e9165908ecb6d26d5e80a/merged major:0 minor:334 fsType:overlay blockSize:0} overlay_0-342:{mountpoint:/var/lib/containers/storage/overlay/227008e793570abd319b60df5322f565d117f8666a8fa497aef8ac911c6c459e/merged major:0 minor:342 fsType:overlay blockSize:0} overlay_0-344:{mountpoint:/var/lib/containers/storage/overlay/70378a7c649a90372c9b7d82fdb2eed019f938df6b130d1abdb8dff08d917b60/merged major:0 minor:344 fsType:overlay blockSize:0} overlay_0-346:{mountpoint:/var/lib/containers/storage/overlay/6f15e6be792ff4bed7c18b16f014097e426f8e56bb48a30327c37ac4e4e2f1a4/merged major:0 minor:346 fsType:overlay blockSize:0} overlay_0-50:{mountpoint:/var/lib/containers/storage/overlay/8e327c42b8825c3cad44990a33b3f5ca11330fdf33b11d001beb5277de87b73f/merged major:0 minor:50 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/26f0c36acd2405eb9b7103944de4984d278d95ffb5a1aece0e4b42a52c9c1acd/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-68:{mountpoint:/var/lib/containers/storage/overlay/4b6f03f01982fd179535b990254a8528fca465b2986b0b9b56c3f3faa080c5d1/merged major:0 minor:68 fsType:overlay blockSize:0} overlay_0-72:{mountpoint:/var/lib/containers/storage/overlay/2a8ee1f8c851c2a7613f603f2d5aef8f61ef681aff1b35b3f144fd0eef42787c/merged major:0 minor:72 fsType:overlay blockSize:0} overlay_0-76:{mountpoint:/var/lib/containers/storage/overlay/52dbd07cc190963c5aa8b18cc5dba366b6c0cb2d28bcc433e832bb3a1d14f58d/merged major:0 minor:76 fsType:overlay blockSize:0} overlay_0-80:{mountpoint:/var/lib/containers/storage/overlay/9e9a2eac942c9110b43b815467785c73bd5d919f30cd273683a727c493438de3/merged major:0 minor:80 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/d5812372ec7b58121fadf2d1a32b4ca5ceef06b3051d85de22fc77c1b6d97b67/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/77eee2ee61d38331202610e19bdc4ccd6c93a271110b77bc14cb3c7196b7732e/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-93:{mountpoint:/var/lib/containers/storage/overlay/a000f1f7d0122177af81cb55417268d9356f2c8457eb30d4bbe341f0a268add3/merged major:0 minor:93 fsType:overlay blockSize:0} overlay_0-98:{mountpoint:/var/lib/containers/storage/overlay/fa7f3de60905b5ba23dbbb758c87f17cefcef86120399a5fad6939b4db640ecb/merged major:0 minor:98 fsType:overlay blockSize:0}] Nov 03 13:08:40.055639 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.053114 5024 manager.go:217] Machine: {Timestamp:2026-11-03 13:08:40.05265728 +0000 UTC m=+0.060848548 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099945 MemoryCapacity:33164341248 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec254c61e28ea61f2656b603524d3067 SystemUUID:ec254c61-e28e-a61f-2656-b603524d3067 BootID:2b02f133-fa94-4f95-b9a7-98c3dbc870e5 Filesystems:[{Device:/run/containers/storage/overlay-containers/6983b50836f972f88268b1ac5ff771dc019eeb5a3416d32647e8046e0e92e2d0/userdata/shm DeviceMajor:0 DeviceMinor:86 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0a892eb3-ecc4-4312-9f4d-73eeba6b6eee/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:217 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-283 DeviceMajor:0 DeviceMinor:283 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-270 DeviceMajor:0 DeviceMinor:270 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5ea5cf61-ed1c-4f5e-b1a0-5c46c5c3ee18/volumes/kubernetes.io~projected/kube-api-access-vjhjt DeviceMajor:0 DeviceMinor:66 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/0a892eb3-ecc4-4312-9f4d-73eeba6b6eee/volumes/kubernetes.io~projected/kube-api-access-xmbd7 DeviceMajor:0 DeviceMinor:220 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/84fabe8a79acb6040874b2b7e92579078c1dff682320fea274fd83a4f027c1ea/userdata/shm DeviceMajor:0 DeviceMinor:228 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-256 DeviceMajor:0 DeviceMinor:256 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/cc8ef30e-6d4f-4913-8dc8-b09e5aea2f82/volumes/kubernetes.io~projected/kube-api-access-h82mn DeviceMajor:0 DeviceMinor:208 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/fd7e8b58-de31-406a-8cdb-9ccfee140d57/volumes/kubernetes.io~projected/kube-api-access-snpf4 DeviceMajor:0 DeviceMinor:60 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/cd91de08-084e-4667-b13f-2e07b3cff624/volumes/kubernetes.io~projected/kube-api-access-pt6qk DeviceMajor:0 DeviceMinor:210 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/6cb478ab-97aa-4881-87b2-b45293cc2349/volumes/kubernetes.io~projected/kube-api-access-x7mw6 DeviceMajor:0 DeviceMinor:213 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-143 DeviceMajor:0 DeviceMinor:143 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/cc8ef30e-6d4f-4913-8dc8-b09e5aea2f82/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:197 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/898bd0b5-0d3e-4577-b101-2cc7d75283cb/volumes/kubernetes.io~secret/default-certificate DeviceMajor:0 DeviceMinor:214 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-342 DeviceMajor:0 DeviceMinor:342 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true} {Device:/var/lib/kubelet/pods/246971bc-ed35-4fc7-b5f8-941327b90a94/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:64 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/21618783-0a4e-4b1f-8041-cabdf69f5bea/volumes/kubernetes.io~projected/kube-api-access-vbs5f DeviceMajor:0 DeviceMinor:223 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/9f20ddda-41fb-4aba-b257-7ebc59dbe678/volumes/kubernetes.io~projected/kube-api-access-jknsv DeviceMajor:0 DeviceMinor:221 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5b44ee172423f97b68d439f1263e56f195b4502d78f9d58d2899e3b7eee3152f/userdata/shm DeviceMajor:0 DeviceMinor:49 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/9db7d195-220e-4644-8690-9e5ee37fc066/volumes/kubernetes.io~projected/kube-api-access-ftcqc DeviceMajor:0 DeviceMinor:57 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c29cb1f6582414bd2e24af736205bd2de388faaebcd65096615b22499cfdaf05/userdata/shm DeviceMajor:0 DeviceMinor:82 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-131 DeviceMajor:0 DeviceMinor:131 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9fa6d32d-b30b-4b76-9462-5f3f09443d02/volumes/kubernetes.io~projected/kube-api-access-n8x9b DeviceMajor:0 DeviceMinor:209 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-258 DeviceMajor:0 DeviceMinor:258 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c704eb02-d156-4695-a55d-7dc96654e4cd/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:55 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/74dc4aee-10b1-463c-a9c3-522b7a6c10b7/volumes/kubernetes.io~projected/kube-api-access-cwrs7 DeviceMajor:0 DeviceMinor:61 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-93 DeviceMajor:0 DeviceMinor:93 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/07a7e2de-0c09-4819-9670-2be9460a155d/volumes/kubernetes.io~projected/kube-api-access-7jxlt DeviceMajor:0 DeviceMinor:205 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a8456e348c89166330071974486e35de81374e53475426213db288f247638ed1/userdata/shm DeviceMajor:0 DeviceMinor:236 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1658c9a5bbf6b90cd5b0c2c5ef8bc57b57b5e38c2ebde6e79097b6a75ad221c5/userdata/shm DeviceMajor:0 DeviceMinor:238 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-264 DeviceMajor:0 DeviceMinor:264 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/bd523616-71b2-4b6c-b82c-f1efb585c271/volumes/kubernetes.io~projected/kube-api-access-mtd8p DeviceMajor:0 DeviceMinor:329 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/080b34934d5649c436eb755c96cb75220add975b57d1a9023728b6782facbc86/userdata/shm DeviceMajor:0 DeviceMinor:74 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5ea2ec30de3f7897100ae5923cb74d11ffc40a71885605a5855258979863426c/userdata/shm DeviceMajor:0 DeviceMinor:240 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-304 DeviceMajor:0 DeviceMinor:304 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-317 DeviceMajor:0 DeviceMinor:317 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e0300f55-f002-4faa-bf6a-c04bea0a043e/volumes/kubernetes.io~projected/kube-api-access-bcz25 DeviceMajor:0 DeviceMinor:63 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/cc8ef30e-6d4f-4913-8dc8-b09e5aea2f82/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:201 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-248 DeviceMajor:0 DeviceMinor:248 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-254 DeviceMajor:0 DeviceMinor:254 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/var/lib/kubelet/pods/c704eb02-d156-4695-a55d-7dc96654e4cd/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:56 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/c704eb02-d156-4695-a55d-7dc96654e4cd/volumes/kubernetes.io~projected/kube-api-access-w4j5g DeviceMajor:0 DeviceMinor:62 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/d926d945-39f4-48f3-b3a4-0b302b15b69b/volumes/kubernetes.io~projected/kube-api-access-6b28l DeviceMajor:0 DeviceMinor:67 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b8fe3220-d285-487f-9fc0-597438982de7/volumes/kubernetes.io~projected/kube-api-access-zhwtz DeviceMajor:0 DeviceMinor:204 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-50 DeviceMajor:0 DeviceMinor:50 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/32b60dc42f674361e6ee2e8b8778e572fd8fc785139362be592e97abee98664a/userdata/shm DeviceMajor:0 DeviceMinor:70 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/1bfc1afc-5b9f-4447-8d40-fabedbee9c07/volumes/kubernetes.io~projected/kube-api-access-hbfgb DeviceMajor:0 DeviceMinor:288 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/39e11989-e598-4cea-87d9-26b3cf78a66b/volumes/kubernetes.io~projected/kube-api-access-zpq8b DeviceMajor:0 DeviceMinor:58 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/39aa5c2c75a1203c58dd67810fc2d1718f5535ce2d67634692f6a034d4a8fe41/userdata/shm DeviceMajor:0 DeviceMinor:78 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-98 DeviceMajor:0 DeviceMinor:98 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0a892eb3-ecc4-4312-9f4d-73eeba6b6eee/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:226 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-302 DeviceMajor:0 DeviceMinor:302 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/fff35d1b-e7b2-4dd3-9824-8b64fad02506/volumes/kubernetes.io~projected/kube-api-access-jssfx DeviceMajor:0 DeviceMinor:310 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/50d68cb7-b766-4674-b295-8f3229880994/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:219 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-334 DeviceMajor:0 DeviceMinor:334 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-68 DeviceMajor:0 DeviceMinor:68 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-76 DeviceMajor:0 DeviceMinor:76 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/12c34fc486ae00af6cdac52a0c622b9549f772c0e690148eea53070e65b1da62/userdata/shm DeviceMajor:0 DeviceMinor:234 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-285 DeviceMajor:0 DeviceMinor:285 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-298 DeviceMajor:0 DeviceMinor:298 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ecf7ff9bffd580730bcdf842700eb81895af4e3d420ec626d39da9261adba3ee/userdata/shm DeviceMajor:0 DeviceMinor:95 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-159 DeviceMajor:0 DeviceMinor:159 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/cd91de08-084e-4667-b13f-2e07b3cff624/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:202 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-266 DeviceMajor:0 DeviceMinor:266 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7e280277-f29f-4a2b-8b85-2491529e5ef2/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:293 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ab29b318c08555358cf838590f8909d568ac39d5a7ad9c2cbd6830c22a1c9f52/userdata/shm DeviceMajor:0 DeviceMinor:246 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5d20773640ce8e198ac75606401c820525fb0fe1350882ef0b9b2ed7d166824a/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-139 DeviceMajor:0 DeviceMinor:139 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/68fe3802481c8f7a3638809cd85a097698ee57db88bc75d6ddd15414cd29f9b5/userdata/shm DeviceMajor:0 DeviceMinor:232 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/35e0fb40-48ce-4223-b7d4-c9290b6861cd/volumes/kubernetes.io~projected/kube-api-access-mss42 DeviceMajor:0 DeviceMinor:59 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-154 DeviceMajor:0 DeviceMinor:154 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d926d945-39f4-48f3-b3a4-0b302b15b69b/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/9fa6d32d-b30b-4b76-9462-5f3f09443d02/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:45 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-315 DeviceMajor:0 DeviceMinor:315 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-323 DeviceMajor:0 DeviceMinor:323 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6cb478ab-97aa-4881-87b2-b45293cc2349/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:203 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e6c2867616afcbdf7d2a5611abc303a745fe5f9dece80d4859a108a05d8f4438/userdata/shm DeviceMajor:0 DeviceMinor:92 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3cf1dc4f654fe1eddf341ba01ad4d52e81aac0f2f4a81246a19b59883c62f85a/userdata/shm DeviceMajor:0 DeviceMinor:244 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-268 DeviceMajor:0 DeviceMinor:268 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-300 DeviceMajor:0 DeviceMinor:300 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/cc8ef30e-6d4f-4913-8dc8-b09e5aea2f82/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:212 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/5c17f5bf-5b7d-4b05-a387-21ac6b5186ca/volumes/kubernetes.io~projected/kube-api-access-k9fhg DeviceMajor:0 DeviceMinor:224 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a670effa2a69f2c5a5f3625db93d19af7b21e7237cd4c41a16088e66cb4104a2/userdata/shm DeviceMajor:0 DeviceMinor:229 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-250 DeviceMajor:0 DeviceMinor:250 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-262 DeviceMajor:0 DeviceMinor:262 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-346 DeviceMajor:0 DeviceMinor:346 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-72 DeviceMajor:0 DeviceMinor:72 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-80 DeviceMajor:0 DeviceMinor:80 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c82d9f4b-8c00-4c31-8aa2-99cdc0d87e52/volumes/kubernetes.io~projected/kube-api-access-cb5g7 DeviceMajor:0 DeviceMinor:207 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/50d68cb7-b766-4674-b295-8f3229880994/volumes/kubernetes.io~projected/kube-api-access-2sxmr DeviceMajor:0 DeviceMinor:222 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/3fdeb9f3-6c67-4390-8073-17d4d5f802b9/volumes/kubernetes.io~projected/kube-api-access-bddlp DeviceMajor:0 DeviceMinor:225 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/d926d945-39f4-48f3-b3a4-0b302b15b69b/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:65 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/021e9e00-a973-4726-8f48-02a52c4bae14/volumes/kubernetes.io~projected/kube-api-access-qcsxn DeviceMajor:0 DeviceMinor:211 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2714947ff73646c1e92b308d87f7aa0ae91cd24f08a9ed96681ba260c6f6d555/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0a892eb3-ecc4-4312-9f4d-73eeba6b6eee/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:215 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b60ab4a7af01474429ab8f251670ff485fabd1f7d74bcd699a42190bfa711191/userdata/shm DeviceMajor:0 DeviceMinor:242 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-260 DeviceMajor:0 DeviceMinor:260 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/20af2de0975a5d8d3f6cee427d3f37295edf7f21d547d66bb589a540ecc17985/userdata/shm DeviceMajor:0 DeviceMinor:296 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-134 DeviceMajor:0 DeviceMinor:134 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-163 DeviceMajor:0 DeviceMinor:163 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a3042b60-062f-4f9b-8406-f884c95a8729/volumes/kubernetes.io~projected/kube-api-access-4lj56 DeviceMajor:0 DeviceMinor:206 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/898bd0b5-0d3e-4577-b101-2cc7d75283cb/volumes/kubernetes.io~secret/stats-auth DeviceMajor:0 DeviceMinor:218 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/898bd0b5-0d3e-4577-b101-2cc7d75283cb/volumes/kubernetes.io~projected/kube-api-access-bzlml DeviceMajor:0 DeviceMinor:227 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/48ba5bc216484088efd6cef6f8687c4da1e5abc695dc4f3df755ca6977ace6b0/userdata/shm DeviceMajor:0 DeviceMinor:294 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9f20ddda-41fb-4aba-b257-7ebc59dbe678/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:216 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-252 DeviceMajor:0 DeviceMinor:252 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a15ed7253699ac60f5504c1ca9afee7dc425b76d3113908f1e8d7116db36f547/userdata/shm DeviceMajor:0 DeviceMinor:336 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-344 DeviceMajor:0 DeviceMinor:344 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:12c34fc486ae00a MacAddress:aa:22:a1:ad:e8:88 Speed:10000 Mtu:8901} {Name:1658c9a5bbf6b90 MacAddress:06:12:b9:cb:f2:54 Speed:10000 Mtu:8901} {Name:20af2de0975a5d8 MacAddress:06:88:a3:da:48:82 Speed:10000 Mtu:8901} {Name:3cf1dc4f654fe1e MacAddress:9e:08:78:b1:e4:48 Speed:10000 Mtu:8901} {Name:48ba5bc21648408 MacAddress:da:79:d4:55:cb:ec Speed:10000 Mtu:8901} {Name:5ea2ec30de3f789 MacAddress:36:af:2a:74:df:4e Speed:10000 Mtu:8901} {Name:68fe3802481c8f7 MacAddress:62:fd:49:99:7b:7c Speed:10000 Mtu:8901} {Name:84fabe8a79acb60 MacAddress:d6:4e:9e:db:17:9e Speed:10000 Mtu:8901} {Name:a15ed7253699ac6 MacAddress:aa:13:7d:9c:89:55 Speed:10000 Mtu:8901} {Name:a670effa2a69f2c MacAddress:06:9a:1d:4a:16:0b Speed:10000 Mtu:8901} {Name:a8456e348c89166 MacAddress:26:3a:94:3e:62:98 Speed:10000 Mtu:8901} {Name:ab29b318c085553 MacAddress:a6:ee:36:c8:b1:bb Speed:10000 Mtu:8901} {Name:b60ab4a7af01474 MacAddress:66:b1:b0:53:ec:2e Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:d0:69:7a:87 Speed:0 Mtu:9001} {Name:br-int MacAddress:5a:78:96:73:94:a4 Speed:0 Mtu:8901} {Name:ens5 MacAddress:02:ff:d0:69:7a:87 Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:8a:14:90:fc:bc:ad Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:84:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:56:36:d1:4f:c0:a5 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164341248 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 03 13:08:40.055639 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.053483 5024 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 03 13:08:40.055639 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.053528 5024 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 03 13:08:40.055639 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.053963 5024 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 03 13:08:40.055639 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.053992 5024 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-131-36.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 03 13:08:40.055639 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.054192 5024 topology_manager.go:143] "Creating topology manager with none policy" Nov 03 13:08:40.055639 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.054205 5024 container_manager_linux.go:311] "Creating device plugin manager" Nov 03 13:08:40.055639 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.054220 5024 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 03 13:08:40.055639 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.054238 5024 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 03 13:08:40.055639 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.054268 5024 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 03 13:08:40.055639 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.054568 5024 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 03 13:08:40.055639 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.054738 5024 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 03 13:08:40.055639 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.054781 5024 kubelet.go:493] "Attempting to sync node with API server" Nov 03 13:08:40.055639 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.054797 5024 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 03 13:08:40.055639 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.054811 5024 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 03 13:08:40.055639 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.054824 5024 kubelet.go:405] "Adding apiserver pod source" Nov 03 13:08:40.055639 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.054850 5024 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 03 13:08:40.056536 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.055967 5024 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 03 13:08:40.056536 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.055982 5024 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 03 13:08:40.056596 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.056530 5024 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 03 13:08:40.059819 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.059800 5024 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 03 13:08:40.059819 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.059816 5024 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 03 13:08:40.059932 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.059923 5024 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Nov 03 13:08:40.060122 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.060110 5024 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 03 13:08:40.060158 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.060140 5024 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 03 13:08:40.060254 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.060245 5024 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 03 13:08:40.060297 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.060260 5024 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 03 13:08:40.060297 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.060268 5024 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 03 13:08:40.060297 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.060273 5024 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 03 13:08:40.060297 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.060294 5024 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 03 13:08:40.060407 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.060304 5024 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 03 13:08:40.060407 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.060310 5024 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 03 13:08:40.060407 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.060316 5024 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 03 13:08:40.060407 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.060324 5024 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 03 13:08:40.060407 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.060330 5024 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 03 13:08:40.060407 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.060338 5024 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 03 13:08:40.060407 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.060347 5024 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 03 13:08:40.060407 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.060370 5024 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 03 13:08:40.060407 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.060379 5024 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 03 13:08:40.063988 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.063973 5024 server.go:1263] "Started kubelet" Nov 03 13:08:40.064443 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.064078 5024 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 03 13:08:40.064443 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.064088 5024 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 03 13:08:40.064443 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.064159 5024 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 03 13:08:40.064529 ip-10-0-131-36 systemd[1]: Started Kubernetes Kubelet. Nov 03 13:08:40.064936 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.064910 5024 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 03 13:08:40.065567 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.065393 5024 server.go:317] "Adding debug handlers to kubelet server" Nov 03 13:08:40.068074 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.068040 5024 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 03 13:08:40.068976 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:08:40.068158 5024 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-36.us-east-2.compute.internal\" not found" Nov 03 13:08:40.068976 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.068186 5024 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 03 13:08:40.068976 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.068463 5024 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 13:02:47 +0000 UTC" deadline="2028-08-14 18:49:21.499367606 +0000 UTC" Nov 03 13:08:40.068976 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.068536 5024 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15605h40m41.430835718s" Nov 03 13:08:40.069373 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.069356 5024 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 03 13:08:40.079081 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.072443 5024 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 03 13:08:40.079081 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.072724 5024 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 03 13:08:40.079081 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.073070 5024 factory.go:55] Registering systemd factory Nov 03 13:08:40.079081 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.073088 5024 factory.go:223] Registration of the systemd container factory successfully Nov 03 13:08:40.079081 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.073691 5024 factory.go:153] Registering CRI-O factory Nov 03 13:08:40.079081 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.073714 5024 factory.go:223] Registration of the crio container factory successfully Nov 03 13:08:40.079081 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.073774 5024 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 03 13:08:40.079081 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.073797 5024 factory.go:103] Registering Raw factory Nov 03 13:08:40.079081 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.073823 5024 manager.go:1199] Started watching for new ooms in manager Nov 03 13:08:40.079081 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.074631 5024 manager.go:319] Starting recovery of all containers Nov 03 13:08:40.079081 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.075506 5024 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 13:08:40.079081 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.077956 5024 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="35e0fb40-48ce-4223-b7d4-c9290b6861cd" volumeName="kubernetes.io/configmap/35e0fb40-48ce-4223-b7d4-c9290b6861cd-cni-sysctl-allowlist" seLinuxMountContext="" Nov 03 13:08:40.079081 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.078025 5024 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6cb478ab-97aa-4881-87b2-b45293cc2349" volumeName="kubernetes.io/secret/6cb478ab-97aa-4881-87b2-b45293cc2349-serving-cert" seLinuxMountContext="" Nov 03 13:08:40.079081 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.078056 5024 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="898bd0b5-0d3e-4577-b101-2cc7d75283cb" volumeName="kubernetes.io/secret/898bd0b5-0d3e-4577-b101-2cc7d75283cb-stats-auth" seLinuxMountContext="" Nov 03 13:08:40.079081 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.078080 5024 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c704eb02-d156-4695-a55d-7dc96654e4cd" volumeName="kubernetes.io/empty-dir/c704eb02-d156-4695-a55d-7dc96654e4cd-tmp" seLinuxMountContext="" Nov 03 13:08:40.079081 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.078094 5024 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cd91de08-084e-4667-b13f-2e07b3cff624" volumeName="kubernetes.io/projected/cd91de08-084e-4667-b13f-2e07b3cff624-kube-api-access-pt6qk" seLinuxMountContext="" Nov 03 13:08:40.079081 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.078123 5024 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="21618783-0a4e-4b1f-8041-cabdf69f5bea" volumeName="kubernetes.io/projected/21618783-0a4e-4b1f-8041-cabdf69f5bea-kube-api-access-vbs5f" seLinuxMountContext="" Nov 03 13:08:40.079081 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.078136 5024 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9fa6d32d-b30b-4b76-9462-5f3f09443d02" volumeName="kubernetes.io/empty-dir/9fa6d32d-b30b-4b76-9462-5f3f09443d02-snapshots" seLinuxMountContext="" Nov 03 13:08:40.079081 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.078150 5024 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d926d945-39f4-48f3-b3a4-0b302b15b69b" volumeName="kubernetes.io/configmap/d926d945-39f4-48f3-b3a4-0b302b15b69b-ovnkube-script-lib" seLinuxMountContext="" Nov 03 13:08:40.079081 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.078183 5024 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fd7e8b58-de31-406a-8cdb-9ccfee140d57" volumeName="kubernetes.io/configmap/fd7e8b58-de31-406a-8cdb-9ccfee140d57-iptables-alerter-script" seLinuxMountContext="" Nov 03 13:08:40.079081 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.078208 5024 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bd523616-71b2-4b6c-b82c-f1efb585c271" volumeName="kubernetes.io/projected/bd523616-71b2-4b6c-b82c-f1efb585c271-kube-api-access-mtd8p" seLinuxMountContext="" Nov 03 13:08:40.079081 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.078240 5024 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="021e9e00-a973-4726-8f48-02a52c4bae14" volumeName="kubernetes.io/projected/021e9e00-a973-4726-8f48-02a52c4bae14-kube-api-access-qcsxn" seLinuxMountContext="" Nov 03 13:08:40.079081 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.078312 5024 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0a892eb3-ecc4-4312-9f4d-73eeba6b6eee" volumeName="kubernetes.io/projected/0a892eb3-ecc4-4312-9f4d-73eeba6b6eee-bound-sa-token" seLinuxMountContext="" Nov 03 13:08:40.079081 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.078330 5024 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0a892eb3-ecc4-4312-9f4d-73eeba6b6eee" volumeName="kubernetes.io/secret/0a892eb3-ecc4-4312-9f4d-73eeba6b6eee-image-registry-private-configuration" seLinuxMountContext="" Nov 03 13:08:40.079081 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.078343 5024 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="50d68cb7-b766-4674-b295-8f3229880994" volumeName="kubernetes.io/projected/50d68cb7-b766-4674-b295-8f3229880994-kube-api-access-2sxmr" seLinuxMountContext="" Nov 03 13:08:40.079081 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.078388 5024 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9f20ddda-41fb-4aba-b257-7ebc59dbe678" volumeName="kubernetes.io/secret/9f20ddda-41fb-4aba-b257-7ebc59dbe678-serving-cert" seLinuxMountContext="" Nov 03 13:08:40.079081 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.078404 5024 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9fa6d32d-b30b-4b76-9462-5f3f09443d02" volumeName="kubernetes.io/projected/9fa6d32d-b30b-4b76-9462-5f3f09443d02-kube-api-access-n8x9b" seLinuxMountContext="" Nov 03 13:08:40.079081 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.078469 5024 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cc8ef30e-6d4f-4913-8dc8-b09e5aea2f82" volumeName="kubernetes.io/empty-dir/cc8ef30e-6d4f-4913-8dc8-b09e5aea2f82-ca-trust-extracted" seLinuxMountContext="" Nov 03 13:08:40.079081 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.078511 5024 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b8fe3220-d285-487f-9fc0-597438982de7" volumeName="kubernetes.io/projected/b8fe3220-d285-487f-9fc0-597438982de7-kube-api-access-zhwtz" seLinuxMountContext="" Nov 03 13:08:40.079081 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.078524 5024 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0a892eb3-ecc4-4312-9f4d-73eeba6b6eee" volumeName="kubernetes.io/secret/0a892eb3-ecc4-4312-9f4d-73eeba6b6eee-installation-pull-secrets" seLinuxMountContext="" Nov 03 13:08:40.079081 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.078547 5024 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3fdeb9f3-6c67-4390-8073-17d4d5f802b9" volumeName="kubernetes.io/configmap/3fdeb9f3-6c67-4390-8073-17d4d5f802b9-config-volume" seLinuxMountContext="" Nov 03 13:08:40.079081 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.078574 5024 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="898bd0b5-0d3e-4577-b101-2cc7d75283cb" volumeName="kubernetes.io/projected/898bd0b5-0d3e-4577-b101-2cc7d75283cb-kube-api-access-bzlml" seLinuxMountContext="" Nov 03 13:08:40.079081 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.078593 5024 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9fa6d32d-b30b-4b76-9462-5f3f09443d02" volumeName="kubernetes.io/configmap/9fa6d32d-b30b-4b76-9462-5f3f09443d02-trusted-ca-bundle" seLinuxMountContext="" Nov 03 13:08:40.079081 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.078607 5024 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cc8ef30e-6d4f-4913-8dc8-b09e5aea2f82" volumeName="kubernetes.io/projected/cc8ef30e-6d4f-4913-8dc8-b09e5aea2f82-kube-api-access-h82mn" seLinuxMountContext="" Nov 03 13:08:40.079081 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.078620 5024 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cd91de08-084e-4667-b13f-2e07b3cff624" volumeName="kubernetes.io/configmap/cd91de08-084e-4667-b13f-2e07b3cff624-config" seLinuxMountContext="" Nov 03 13:08:40.079081 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.078641 5024 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d926d945-39f4-48f3-b3a4-0b302b15b69b" volumeName="kubernetes.io/configmap/d926d945-39f4-48f3-b3a4-0b302b15b69b-ovnkube-config" seLinuxMountContext="" Nov 03 13:08:40.079081 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.078655 5024 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cc8ef30e-6d4f-4913-8dc8-b09e5aea2f82" volumeName="kubernetes.io/secret/cc8ef30e-6d4f-4913-8dc8-b09e5aea2f82-image-registry-private-configuration" seLinuxMountContext="" Nov 03 13:08:40.079081 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.078672 5024 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3fdeb9f3-6c67-4390-8073-17d4d5f802b9" volumeName="kubernetes.io/empty-dir/3fdeb9f3-6c67-4390-8073-17d4d5f802b9-tmp-dir" seLinuxMountContext="" Nov 03 13:08:40.079081 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.078689 5024 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5ea5cf61-ed1c-4f5e-b1a0-5c46c5c3ee18" volumeName="kubernetes.io/projected/5ea5cf61-ed1c-4f5e-b1a0-5c46c5c3ee18-kube-api-access-vjhjt" seLinuxMountContext="" Nov 03 13:08:40.079081 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.078729 5024 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="898bd0b5-0d3e-4577-b101-2cc7d75283cb" volumeName="kubernetes.io/secret/898bd0b5-0d3e-4577-b101-2cc7d75283cb-default-certificate" seLinuxMountContext="" Nov 03 13:08:40.079081 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.078741 5024 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a3042b60-062f-4f9b-8406-f884c95a8729" volumeName="kubernetes.io/configmap/a3042b60-062f-4f9b-8406-f884c95a8729-telemetry-config" seLinuxMountContext="" Nov 03 13:08:40.079081 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.078763 5024 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fff35d1b-e7b2-4dd3-9824-8b64fad02506" volumeName="kubernetes.io/configmap/fff35d1b-e7b2-4dd3-9824-8b64fad02506-kube-rbac-proxy-cm" seLinuxMountContext="" Nov 03 13:08:40.079081 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.078777 5024 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="35e0fb40-48ce-4223-b7d4-c9290b6861cd" volumeName="kubernetes.io/configmap/35e0fb40-48ce-4223-b7d4-c9290b6861cd-whereabouts-flatfile-configmap" seLinuxMountContext="" Nov 03 13:08:40.079081 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.078791 5024 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="74dc4aee-10b1-463c-a9c3-522b7a6c10b7" volumeName="kubernetes.io/configmap/74dc4aee-10b1-463c-a9c3-522b7a6c10b7-multus-daemon-config" seLinuxMountContext="" Nov 03 13:08:40.079081 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.078818 5024 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="74dc4aee-10b1-463c-a9c3-522b7a6c10b7" volumeName="kubernetes.io/projected/74dc4aee-10b1-463c-a9c3-522b7a6c10b7-kube-api-access-cwrs7" seLinuxMountContext="" Nov 03 13:08:40.079081 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.078833 5024 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c704eb02-d156-4695-a55d-7dc96654e4cd" volumeName="kubernetes.io/projected/c704eb02-d156-4695-a55d-7dc96654e4cd-kube-api-access-w4j5g" seLinuxMountContext="" Nov 03 13:08:40.079081 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.078853 5024 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cc8ef30e-6d4f-4913-8dc8-b09e5aea2f82" volumeName="kubernetes.io/configmap/cc8ef30e-6d4f-4913-8dc8-b09e5aea2f82-registry-certificates" seLinuxMountContext="" Nov 03 13:08:40.079081 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.078867 5024 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cd91de08-084e-4667-b13f-2e07b3cff624" volumeName="kubernetes.io/secret/cd91de08-084e-4667-b13f-2e07b3cff624-serving-cert" seLinuxMountContext="" Nov 03 13:08:40.079081 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.078897 5024 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1bfc1afc-5b9f-4447-8d40-fabedbee9c07" volumeName="kubernetes.io/projected/1bfc1afc-5b9f-4447-8d40-fabedbee9c07-kube-api-access-hbfgb" seLinuxMountContext="" Nov 03 13:08:40.079081 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.078913 5024 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="39e11989-e598-4cea-87d9-26b3cf78a66b" volumeName="kubernetes.io/configmap/39e11989-e598-4cea-87d9-26b3cf78a66b-serviceca" seLinuxMountContext="" Nov 03 13:08:40.079081 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.078921 5024 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c704eb02-d156-4695-a55d-7dc96654e4cd" volumeName="kubernetes.io/empty-dir/c704eb02-d156-4695-a55d-7dc96654e4cd-etc-tuned" seLinuxMountContext="" Nov 03 13:08:40.079081 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.078933 5024 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c82d9f4b-8c00-4c31-8aa2-99cdc0d87e52" volumeName="kubernetes.io/projected/c82d9f4b-8c00-4c31-8aa2-99cdc0d87e52-kube-api-access-cb5g7" seLinuxMountContext="" Nov 03 13:08:40.079081 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.078944 5024 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0a892eb3-ecc4-4312-9f4d-73eeba6b6eee" volumeName="kubernetes.io/projected/0a892eb3-ecc4-4312-9f4d-73eeba6b6eee-kube-api-access-xmbd7" seLinuxMountContext="" Nov 03 13:08:40.079081 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.078953 5024 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="246971bc-ed35-4fc7-b5f8-941327b90a94" volumeName="kubernetes.io/secret/246971bc-ed35-4fc7-b5f8-941327b90a94-agent-certs" seLinuxMountContext="" Nov 03 13:08:40.079081 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.078984 5024 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="50d68cb7-b766-4674-b295-8f3229880994" volumeName="kubernetes.io/secret/50d68cb7-b766-4674-b295-8f3229880994-cert" seLinuxMountContext="" Nov 03 13:08:40.079081 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.078992 5024 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6cb478ab-97aa-4881-87b2-b45293cc2349" volumeName="kubernetes.io/configmap/6cb478ab-97aa-4881-87b2-b45293cc2349-config" seLinuxMountContext="" Nov 03 13:08:40.079081 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.079002 5024 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7e280277-f29f-4a2b-8b85-2491529e5ef2" volumeName="kubernetes.io/secret/7e280277-f29f-4a2b-8b85-2491529e5ef2-original-pull-secret" seLinuxMountContext="" Nov 03 13:08:40.079081 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.079009 5024 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9f20ddda-41fb-4aba-b257-7ebc59dbe678" volumeName="kubernetes.io/projected/9f20ddda-41fb-4aba-b257-7ebc59dbe678-kube-api-access-jknsv" seLinuxMountContext="" Nov 03 13:08:40.079081 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.079018 5024 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fff35d1b-e7b2-4dd3-9824-8b64fad02506" volumeName="kubernetes.io/projected/fff35d1b-e7b2-4dd3-9824-8b64fad02506-kube-api-access-jssfx" seLinuxMountContext="" Nov 03 13:08:40.079081 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.079028 5024 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="07a7e2de-0c09-4819-9670-2be9460a155d" volumeName="kubernetes.io/projected/07a7e2de-0c09-4819-9670-2be9460a155d-kube-api-access-7jxlt" seLinuxMountContext="" Nov 03 13:08:40.079081 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.079037 5024 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a3042b60-062f-4f9b-8406-f884c95a8729" volumeName="kubernetes.io/projected/a3042b60-062f-4f9b-8406-f884c95a8729-kube-api-access-4lj56" seLinuxMountContext="" Nov 03 13:08:40.079081 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.079049 5024 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cc8ef30e-6d4f-4913-8dc8-b09e5aea2f82" volumeName="kubernetes.io/projected/cc8ef30e-6d4f-4913-8dc8-b09e5aea2f82-bound-sa-token" seLinuxMountContext="" Nov 03 13:08:40.079081 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.079056 5024 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fd7e8b58-de31-406a-8cdb-9ccfee140d57" volumeName="kubernetes.io/projected/fd7e8b58-de31-406a-8cdb-9ccfee140d57-kube-api-access-snpf4" seLinuxMountContext="" Nov 03 13:08:40.079081 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.079071 5024 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="246971bc-ed35-4fc7-b5f8-941327b90a94" volumeName="kubernetes.io/configmap/246971bc-ed35-4fc7-b5f8-941327b90a94-konnectivity-ca" seLinuxMountContext="" Nov 03 13:08:40.079081 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.079103 5024 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="35e0fb40-48ce-4223-b7d4-c9290b6861cd" volumeName="kubernetes.io/configmap/35e0fb40-48ce-4223-b7d4-c9290b6861cd-cni-binary-copy" seLinuxMountContext="" Nov 03 13:08:40.079081 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.079113 5024 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5c17f5bf-5b7d-4b05-a387-21ac6b5186ca" volumeName="kubernetes.io/projected/5c17f5bf-5b7d-4b05-a387-21ac6b5186ca-kube-api-access-k9fhg" seLinuxMountContext="" Nov 03 13:08:40.079081 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.079124 5024 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e0300f55-f002-4faa-bf6a-c04bea0a043e" volumeName="kubernetes.io/projected/e0300f55-f002-4faa-bf6a-c04bea0a043e-kube-api-access-bcz25" seLinuxMountContext="" Nov 03 13:08:40.081348 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.079161 5024 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fa297a30-6da7-4a92-98fb-201323490bb0" volumeName="kubernetes.io/configmap/fa297a30-6da7-4a92-98fb-201323490bb0-nginx-conf" seLinuxMountContext="" Nov 03 13:08:40.081348 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.079171 5024 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0a892eb3-ecc4-4312-9f4d-73eeba6b6eee" volumeName="kubernetes.io/empty-dir/0a892eb3-ecc4-4312-9f4d-73eeba6b6eee-ca-trust-extracted" seLinuxMountContext="" Nov 03 13:08:40.081348 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.079181 5024 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="21618783-0a4e-4b1f-8041-cabdf69f5bea" volumeName="kubernetes.io/configmap/21618783-0a4e-4b1f-8041-cabdf69f5bea-samples-operator-config" seLinuxMountContext="" Nov 03 13:08:40.081348 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.079189 5024 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9f20ddda-41fb-4aba-b257-7ebc59dbe678" volumeName="kubernetes.io/configmap/9f20ddda-41fb-4aba-b257-7ebc59dbe678-config" seLinuxMountContext="" Nov 03 13:08:40.081348 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.079199 5024 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cc8ef30e-6d4f-4913-8dc8-b09e5aea2f82" volumeName="kubernetes.io/configmap/cc8ef30e-6d4f-4913-8dc8-b09e5aea2f82-trusted-ca" seLinuxMountContext="" Nov 03 13:08:40.081348 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.079206 5024 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d926d945-39f4-48f3-b3a4-0b302b15b69b" volumeName="kubernetes.io/configmap/d926d945-39f4-48f3-b3a4-0b302b15b69b-env-overrides" seLinuxMountContext="" Nov 03 13:08:40.081348 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.079215 5024 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0a892eb3-ecc4-4312-9f4d-73eeba6b6eee" volumeName="kubernetes.io/configmap/0a892eb3-ecc4-4312-9f4d-73eeba6b6eee-trusted-ca" seLinuxMountContext="" Nov 03 13:08:40.081348 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.079225 5024 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="39e11989-e598-4cea-87d9-26b3cf78a66b" volumeName="kubernetes.io/projected/39e11989-e598-4cea-87d9-26b3cf78a66b-kube-api-access-zpq8b" seLinuxMountContext="" Nov 03 13:08:40.081348 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.079232 5024 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3fdeb9f3-6c67-4390-8073-17d4d5f802b9" volumeName="kubernetes.io/projected/3fdeb9f3-6c67-4390-8073-17d4d5f802b9-kube-api-access-bddlp" seLinuxMountContext="" Nov 03 13:08:40.081348 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.079244 5024 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5ea5cf61-ed1c-4f5e-b1a0-5c46c5c3ee18" volumeName="kubernetes.io/empty-dir/5ea5cf61-ed1c-4f5e-b1a0-5c46c5c3ee18-tmp-dir" seLinuxMountContext="" Nov 03 13:08:40.081348 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.079251 5024 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9fa6d32d-b30b-4b76-9462-5f3f09443d02" volumeName="kubernetes.io/configmap/9fa6d32d-b30b-4b76-9462-5f3f09443d02-service-ca-bundle" seLinuxMountContext="" Nov 03 13:08:40.081348 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.079259 5024 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9fa6d32d-b30b-4b76-9462-5f3f09443d02" volumeName="kubernetes.io/empty-dir/9fa6d32d-b30b-4b76-9462-5f3f09443d02-tmp" seLinuxMountContext="" Nov 03 13:08:40.081348 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.079270 5024 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cd91de08-084e-4667-b13f-2e07b3cff624" volumeName="kubernetes.io/configmap/cd91de08-084e-4667-b13f-2e07b3cff624-trusted-ca" seLinuxMountContext="" Nov 03 13:08:40.081348 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.079277 5024 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="35e0fb40-48ce-4223-b7d4-c9290b6861cd" volumeName="kubernetes.io/projected/35e0fb40-48ce-4223-b7d4-c9290b6861cd-kube-api-access-mss42" seLinuxMountContext="" Nov 03 13:08:40.081348 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.079305 5024 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9db7d195-220e-4644-8690-9e5ee37fc066" volumeName="kubernetes.io/projected/9db7d195-220e-4644-8690-9e5ee37fc066-kube-api-access-ftcqc" seLinuxMountContext="" Nov 03 13:08:40.081348 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.079314 5024 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cc8ef30e-6d4f-4913-8dc8-b09e5aea2f82" volumeName="kubernetes.io/secret/cc8ef30e-6d4f-4913-8dc8-b09e5aea2f82-installation-pull-secrets" seLinuxMountContext="" Nov 03 13:08:40.081348 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.079321 5024 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d926d945-39f4-48f3-b3a4-0b302b15b69b" volumeName="kubernetes.io/secret/d926d945-39f4-48f3-b3a4-0b302b15b69b-ovn-node-metrics-cert" seLinuxMountContext="" Nov 03 13:08:40.081348 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.079331 5024 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6cb478ab-97aa-4881-87b2-b45293cc2349" volumeName="kubernetes.io/projected/6cb478ab-97aa-4881-87b2-b45293cc2349-kube-api-access-x7mw6" seLinuxMountContext="" Nov 03 13:08:40.081348 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.079342 5024 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="74dc4aee-10b1-463c-a9c3-522b7a6c10b7" volumeName="kubernetes.io/configmap/74dc4aee-10b1-463c-a9c3-522b7a6c10b7-cni-binary-copy" seLinuxMountContext="" Nov 03 13:08:40.081348 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.079352 5024 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d926d945-39f4-48f3-b3a4-0b302b15b69b" volumeName="kubernetes.io/projected/d926d945-39f4-48f3-b3a4-0b302b15b69b-kube-api-access-6b28l" seLinuxMountContext="" Nov 03 13:08:40.081348 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.079359 5024 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fff35d1b-e7b2-4dd3-9824-8b64fad02506" volumeName="kubernetes.io/empty-dir/fff35d1b-e7b2-4dd3-9824-8b64fad02506-data-volume" seLinuxMountContext="" Nov 03 13:08:40.081348 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.079367 5024 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0a892eb3-ecc4-4312-9f4d-73eeba6b6eee" volumeName="kubernetes.io/configmap/0a892eb3-ecc4-4312-9f4d-73eeba6b6eee-registry-certificates" seLinuxMountContext="" Nov 03 13:08:40.081348 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.079377 5024 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9fa6d32d-b30b-4b76-9462-5f3f09443d02" volumeName="kubernetes.io/secret/9fa6d32d-b30b-4b76-9462-5f3f09443d02-serving-cert" seLinuxMountContext="" Nov 03 13:08:40.081348 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.079385 5024 reconstruct.go:97] "Volume reconstruction finished" Nov 03 13:08:40.081348 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.079389 5024 reconciler.go:29] "Reconciler: start to sync state" Nov 03 13:08:40.081348 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.079009 5024 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 13:08:40.082492 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.081794 5024 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 13:08:40.083684 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.083665 5024 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Nov 03 13:08:40.091824 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.091802 5024 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6htht_cd91de08-084e-4667-b13f-2e07b3cff624/console-operator/1.log" Nov 03 13:08:40.094544 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.094525 5024 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6htht_cd91de08-084e-4667-b13f-2e07b3cff624/console-operator/1.log" Nov 03 13:08:40.099101 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.099063 5024 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 03 13:08:40.100467 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.100444 5024 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 03 13:08:40.100542 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.100470 5024 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 03 13:08:40.100542 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.100492 5024 kubelet.go:2521] "Starting kubelet main sync loop" Nov 03 13:08:40.100542 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:08:40.100531 5024 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Nov 03 13:08:40.105232 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.105206 5024 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 13:08:40.138182 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.138158 5024 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6htht_cd91de08-084e-4667-b13f-2e07b3cff624/console-operator/1.log" Nov 03 13:08:40.138331 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.138207 5024 generic.go:356] "Generic (PLEG): container finished" podID="cd91de08-084e-4667-b13f-2e07b3cff624" containerID="b811aececd178145fbcac8b298ed317468857bb459f9492b4b679c31b566f969" exitCode=255 Nov 03 13:08:40.143802 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.143776 5024 generic.go:356] "Generic (PLEG): container finished" podID="4879f52c5c2e658a43379722bc61381e" containerID="7938de99cfecbcdb80378eeeb5f8d60e22897f5556f9e4b29d72b35e122fc315" exitCode=0 Nov 03 13:08:40.150117 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.150086 5024 generic.go:356] "Generic (PLEG): container finished" podID="35e0fb40-48ce-4223-b7d4-c9290b6861cd" containerID="4762aad0f9569e283a300d8804cc2778fe0166f2b826c0cae82766b490647026" exitCode=0 Nov 03 13:08:40.150117 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.150116 5024 generic.go:356] "Generic (PLEG): container finished" podID="35e0fb40-48ce-4223-b7d4-c9290b6861cd" containerID="241d8a6be6bd0a09ab0d84fb2151568b70aa0d9799356f04092d09d4c6cc5295" exitCode=0 Nov 03 13:08:40.150259 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.150126 5024 generic.go:356] "Generic (PLEG): container finished" podID="35e0fb40-48ce-4223-b7d4-c9290b6861cd" containerID="a5a99121c65a6e1a531c5296b16ae624bf4702c16c21d96601a9bab3a7cf17f7" exitCode=0 Nov 03 13:08:40.150259 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.150139 5024 generic.go:356] "Generic (PLEG): container finished" podID="35e0fb40-48ce-4223-b7d4-c9290b6861cd" containerID="30ae7c80b1521afa176588c5f54a629ac760877a8bc34593bf1f0c3725b23fb7" exitCode=0 Nov 03 13:08:40.150259 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.150149 5024 generic.go:356] "Generic (PLEG): container finished" podID="35e0fb40-48ce-4223-b7d4-c9290b6861cd" containerID="e0eb309cf5a77d30354f2d94a35fe246555eda5aa469a2482840a0fc5697a982" exitCode=0 Nov 03 13:08:40.150259 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.150159 5024 generic.go:356] "Generic (PLEG): container finished" podID="35e0fb40-48ce-4223-b7d4-c9290b6861cd" containerID="37ce76298956d213d38dc00b494aca90f17e26a203fe01350b91451ec3da9483" exitCode=0 Nov 03 13:08:40.170791 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.170766 5024 manager.go:324] Recovery completed Nov 03 13:08:40.191553 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.191484 5024 cpu_manager.go:226] "Starting" policy="none" Nov 03 13:08:40.191553 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.191501 5024 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 03 13:08:40.191553 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.191521 5024 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 03 13:08:40.191739 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.191650 5024 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Nov 03 13:08:40.191739 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.191660 5024 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Nov 03 13:08:40.191739 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.191677 5024 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Nov 03 13:08:40.191739 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.191682 5024 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Nov 03 13:08:40.191739 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.191687 5024 policy_none.go:50] "Start" Nov 03 13:08:40.191739 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.191695 5024 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 03 13:08:40.191739 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.191704 5024 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 03 13:08:40.191913 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.191819 5024 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Nov 03 13:08:40.191913 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.191826 5024 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Nov 03 13:08:40.191913 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.191832 5024 policy_none.go:44] "Start" Nov 03 13:08:40.196074 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.196046 5024 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 03 13:08:40.198536 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.198522 5024 manager.go:341] "Starting Device Plugin manager" Nov 03 13:08:40.199008 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:08:40.198548 5024 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 03 13:08:40.199008 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.199010 5024 server.go:87] "Starting device plugin registration server" Nov 03 13:08:40.199184 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.199174 5024 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 03 13:08:40.199233 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.199184 5024 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 03 13:08:40.199308 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.199276 5024 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 03 13:08:40.199432 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.199401 5024 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 03 13:08:40.199535 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.199438 5024 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 03 13:08:40.199535 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.199444 5024 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 03 13:08:40.199640 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.199495 5024 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-03T13:08:40.199425982Z","UUID":"ac4878e8-bfad-4d7e-bacb-80891729f595","Handler":null,"Name":"","Endpoint":""} Nov 03 13:08:40.200724 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.200686 5024 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-36.us-east-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-131-36.us-east-2.compute.internal"] Nov 03 13:08:40.200953 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.200936 5024 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 03 13:08:40.201110 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.200963 5024 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 03 13:08:40.201178 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:08:40.201115 5024 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 03 13:08:40.201574 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.201457 5024 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-36.us-east-2.compute.internal" Nov 03 13:08:40.201651 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.201579 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-36.us-east-2.compute.internal" event={"ID":"4879f52c5c2e658a43379722bc61381e","Type":"ContainerStarted","Data":"6ea16991d8dbeee069c615b16eca7e6ec8aebeaf09d0e7c1bec17e37fdcbfc9b"} Nov 03 13:08:40.201651 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.201629 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-36.us-east-2.compute.internal" event={"ID":"4879f52c5c2e658a43379722bc61381e","Type":"ContainerDied","Data":"7938de99cfecbcdb80378eeeb5f8d60e22897f5556f9e4b29d72b35e122fc315"} Nov 03 13:08:40.201748 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.201650 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-36.us-east-2.compute.internal" event={"ID":"4879f52c5c2e658a43379722bc61381e","Type":"ContainerStarted","Data":"5d20773640ce8e198ac75606401c820525fb0fe1350882ef0b9b2ed7d166824a"} Nov 03 13:08:40.201748 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.201702 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-36.us-east-2.compute.internal" event={"ID":"5ee88558fb38141d796a31757de861ab","Type":"ContainerStarted","Data":"9f04a5587062ce2629d840cb1bf24b80a09c12aabf2775705c79a45bb65216ca"} Nov 03 13:08:40.201748 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.201720 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-36.us-east-2.compute.internal" event={"ID":"5ee88558fb38141d796a31757de861ab","Type":"ContainerStarted","Data":"5b44ee172423f97b68d439f1263e56f195b4502d78f9d58d2899e3b7eee3152f"} Nov 03 13:08:40.201895 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.201822 5024 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-36.us-east-2.compute.internal" Nov 03 13:08:40.213807 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.213786 5024 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 13:08:40.213935 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:08:40.213834 5024 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-131-36.us-east-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-36.us-east-2.compute.internal" Nov 03 13:08:40.214621 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.214597 5024 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 13:08:40.214716 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:08:40.214648 5024 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-131-36.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-36.us-east-2.compute.internal" Nov 03 13:08:40.284633 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.284590 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/4879f52c5c2e658a43379722bc61381e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-36.us-east-2.compute.internal\" (UID: \"4879f52c5c2e658a43379722bc61381e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-36.us-east-2.compute.internal" Nov 03 13:08:40.284633 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.284637 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4879f52c5c2e658a43379722bc61381e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-36.us-east-2.compute.internal\" (UID: \"4879f52c5c2e658a43379722bc61381e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-36.us-east-2.compute.internal" Nov 03 13:08:40.284840 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.284656 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/5ee88558fb38141d796a31757de861ab-config\") pod \"kube-apiserver-proxy-ip-10-0-131-36.us-east-2.compute.internal\" (UID: \"5ee88558fb38141d796a31757de861ab\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-36.us-east-2.compute.internal" Nov 03 13:08:40.299695 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.299672 5024 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 13:08:40.300816 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.300802 5024 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-36.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 13:08:40.300905 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.300827 5024 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-36.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 13:08:40.300905 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.300840 5024 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-36.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 03 13:08:40.300905 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.300879 5024 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-131-36.us-east-2.compute.internal" Nov 03 13:08:40.315743 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.315719 5024 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-131-36.us-east-2.compute.internal" Nov 03 13:08:40.315843 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.315775 5024 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-131-36.us-east-2.compute.internal" Nov 03 13:08:40.385577 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.385538 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/4879f52c5c2e658a43379722bc61381e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-36.us-east-2.compute.internal\" (UID: \"4879f52c5c2e658a43379722bc61381e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-36.us-east-2.compute.internal" Nov 03 13:08:40.385577 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.385560 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/4879f52c5c2e658a43379722bc61381e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-36.us-east-2.compute.internal\" (UID: \"4879f52c5c2e658a43379722bc61381e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-36.us-east-2.compute.internal" Nov 03 13:08:40.385763 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.385589 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4879f52c5c2e658a43379722bc61381e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-36.us-east-2.compute.internal\" (UID: \"4879f52c5c2e658a43379722bc61381e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-36.us-east-2.compute.internal" Nov 03 13:08:40.385763 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.385608 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/5ee88558fb38141d796a31757de861ab-config\") pod \"kube-apiserver-proxy-ip-10-0-131-36.us-east-2.compute.internal\" (UID: \"5ee88558fb38141d796a31757de861ab\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-36.us-east-2.compute.internal" Nov 03 13:08:40.385763 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.385639 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4879f52c5c2e658a43379722bc61381e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-36.us-east-2.compute.internal\" (UID: \"4879f52c5c2e658a43379722bc61381e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-36.us-east-2.compute.internal" Nov 03 13:08:40.385763 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:40.385649 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/5ee88558fb38141d796a31757de861ab-config\") pod \"kube-apiserver-proxy-ip-10-0-131-36.us-east-2.compute.internal\" (UID: \"5ee88558fb38141d796a31757de861ab\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-36.us-east-2.compute.internal" Nov 03 13:08:41.055248 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.055197 5024 apiserver.go:52] "Watching apiserver" Nov 03 13:08:41.069942 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.069912 5024 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 03 13:08:41.070737 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.070716 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-6nz8d","openshift-multus/multus-additional-cni-plugins-x2rm4","kube-system/kas-connection-checker-685895c7c9-vl8fr","openshift-image-registry/node-ca-4tn4j","openshift-ingress/router-default-7f579888d9-ntt6m","openshift-network-operator/iptables-alerter-g4xgp","openshift-network-diagnostics/network-check-source-6f78d59f99-jws95","kube-system/kas-connection-checker-685895c7c9-w5cql","openshift-image-registry/image-registry-7dbb67bd57-74qfj","openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-h46gl","openshift-insights/insights-runtime-extractor-d7mwz","openshift-multus/network-metrics-daemon-f7hmh","openshift-cluster-node-tuning-operator/tuned-hmbpc","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-36.us-east-2.compute.internal","openshift-image-registry/image-registry-555f8976d-mz5ct","openshift-multus/multus-dsjsv","openshift-insights/insights-operator-7c4986d449-4t8lj","openshift-dns/node-resolver-v8gcq","kube-system/konnectivity-agent-bdg96","openshift-network-console/networking-console-plugin-86c478bb88-rz4n9","openshift-kube-storage-version-migrator/migrator-66df5d496b-gkblf","openshift-dns/dns-default-vgnv8","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c9dvm","kube-system/kube-apiserver-proxy-ip-10-0-131-36.us-east-2.compute.internal","openshift-network-diagnostics/network-check-target-8rdgq","openshift-service-ca-operator/service-ca-operator-544d749c8c-cqvdv","openshift-monitoring/cluster-monitoring-operator-884576c57-tm4nb","kube-system/global-pull-secret-syncer-5jz7t","openshift-console-operator/console-operator-7cd56b8dbc-6htht","openshift-ovn-kubernetes/ovnkube-node-fg6vh","kube-system/kas-connection-checker-685895c7c9-hslrf","openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-p6cbj","openshift-ingress-canary/ingress-canary-d8f6l"] Nov 03 13:08:41.072365 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.072346 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-tm4nb" Nov 03 13:08:41.076688 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.076644 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-555f8976d-mz5ct" Nov 03 13:08:41.077025 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.077001 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\"" Nov 03 13:08:41.077139 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.077087 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Nov 03 13:08:41.077616 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.076782 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\"" Nov 03 13:08:41.077616 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.077410 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Nov 03 13:08:41.077616 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.077491 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Nov 03 13:08:41.077616 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.077517 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Nov 03 13:08:41.077830 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.077759 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Nov 03 13:08:41.077912 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.077897 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Nov 03 13:08:41.077990 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.077974 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-h46gl" Nov 03 13:08:41.078456 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.078440 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Nov 03 13:08:41.078519 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.078483 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Nov 03 13:08:41.078560 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.078553 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Nov 03 13:08:41.078615 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.078605 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Nov 03 13:08:41.078645 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.078626 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-7f579888d9-ntt6m" Nov 03 13:08:41.079161 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.079141 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-rz4n9" Nov 03 13:08:41.079532 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.079515 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Nov 03 13:08:41.079676 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.079549 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Nov 03 13:08:41.079676 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.079621 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Nov 03 13:08:41.079676 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.079658 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Nov 03 13:08:41.082009 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.081989 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f7hmh" Nov 03 13:08:41.083336 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.083321 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Nov 03 13:08:41.084349 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.084333 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Nov 03 13:08:41.084669 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.084644 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 03 13:08:41.084782 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.084708 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Nov 03 13:08:41.084935 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.084927 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Nov 03 13:08:41.085012 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.084964 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Nov 03 13:08:41.085012 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.084988 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Nov 03 13:08:41.085012 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.084987 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7dbb67bd57-74qfj" Nov 03 13:08:41.085102 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.085018 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Nov 03 13:08:41.085136 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.084992 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Nov 03 13:08:41.085524 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.085510 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-vgnv8" Nov 03 13:08:41.086691 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.086677 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-d7mwz" Nov 03 13:08:41.088888 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.088860 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qcsxn\" (UniqueName: \"kubernetes.io/projected/021e9e00-a973-4726-8f48-02a52c4bae14-kube-api-access-qcsxn\") pod \"volume-data-source-validator-58c9f5f46d-p6cbj\" (UID: \"021e9e00-a973-4726-8f48-02a52c4bae14\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-p6cbj" Nov 03 13:08:41.088993 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.088916 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a3042b60-062f-4f9b-8406-f884c95a8729-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-tm4nb\" (UID: \"a3042b60-062f-4f9b-8406-f884c95a8729\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-tm4nb" Nov 03 13:08:41.088993 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.088949 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4lj56\" (UniqueName: \"kubernetes.io/projected/a3042b60-062f-4f9b-8406-f884c95a8729-kube-api-access-4lj56\") pod \"cluster-monitoring-operator-884576c57-tm4nb\" (UID: \"a3042b60-062f-4f9b-8406-f884c95a8729\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-tm4nb" Nov 03 13:08:41.088993 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.088980 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6cb478ab-97aa-4881-87b2-b45293cc2349-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-6nz8d\" (UID: \"6cb478ab-97aa-4881-87b2-b45293cc2349\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-6nz8d" Nov 03 13:08:41.089176 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.089010 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7jxlt\" (UniqueName: \"kubernetes.io/projected/07a7e2de-0c09-4819-9670-2be9460a155d-kube-api-access-7jxlt\") pod \"kas-connection-checker-685895c7c9-vl8fr\" (UID: \"07a7e2de-0c09-4819-9670-2be9460a155d\") " pod="kube-system/kas-connection-checker-685895c7c9-vl8fr" Nov 03 13:08:41.089176 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.089036 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/a3042b60-062f-4f9b-8406-f884c95a8729-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-tm4nb\" (UID: \"a3042b60-062f-4f9b-8406-f884c95a8729\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-tm4nb" Nov 03 13:08:41.089176 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.089065 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6cb478ab-97aa-4881-87b2-b45293cc2349-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-6nz8d\" (UID: \"6cb478ab-97aa-4881-87b2-b45293cc2349\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-6nz8d" Nov 03 13:08:41.089176 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.089095 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7mw6\" (UniqueName: \"kubernetes.io/projected/6cb478ab-97aa-4881-87b2-b45293cc2349-kube-api-access-x7mw6\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-6nz8d\" (UID: \"6cb478ab-97aa-4881-87b2-b45293cc2349\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-6nz8d" Nov 03 13:08:41.089176 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.089104 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Nov 03 13:08:41.089176 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.089127 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhwtz\" (UniqueName: \"kubernetes.io/projected/b8fe3220-d285-487f-9fc0-597438982de7-kube-api-access-zhwtz\") pod \"kas-connection-checker-685895c7c9-hslrf\" (UID: \"b8fe3220-d285-487f-9fc0-597438982de7\") " pod="kube-system/kas-connection-checker-685895c7c9-hslrf" Nov 03 13:08:41.089176 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.089156 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cb5g7\" (UniqueName: \"kubernetes.io/projected/c82d9f4b-8c00-4c31-8aa2-99cdc0d87e52-kube-api-access-cb5g7\") pod \"kas-connection-checker-685895c7c9-w5cql\" (UID: \"c82d9f4b-8c00-4c31-8aa2-99cdc0d87e52\") " pod="kube-system/kas-connection-checker-685895c7c9-w5cql" Nov 03 13:08:41.089948 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.089917 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/a3042b60-062f-4f9b-8406-f884c95a8729-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-tm4nb\" (UID: \"a3042b60-062f-4f9b-8406-f884c95a8729\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-tm4nb" Nov 03 13:08:41.090170 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.090151 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6cb478ab-97aa-4881-87b2-b45293cc2349-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-6nz8d\" (UID: \"6cb478ab-97aa-4881-87b2-b45293cc2349\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-6nz8d" Nov 03 13:08:41.090530 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.090508 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-vwt8x\"" Nov 03 13:08:41.090595 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.090547 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 03 13:08:41.090651 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.090636 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 03 13:08:41.090695 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.090654 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 03 13:08:41.090775 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.090753 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6cb478ab-97aa-4881-87b2-b45293cc2349-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-6nz8d\" (UID: \"6cb478ab-97aa-4881-87b2-b45293cc2349\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-6nz8d" Nov 03 13:08:41.090928 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.090833 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 03 13:08:41.090928 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.090854 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 03 13:08:41.090928 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.090877 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 03 13:08:41.091067 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.090989 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-c596b\"" Nov 03 13:08:41.091109 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.091064 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Nov 03 13:08:41.091982 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.091159 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Nov 03 13:08:41.091982 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.091064 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 03 13:08:41.091982 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.091185 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 03 13:08:41.091982 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.091355 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Nov 03 13:08:41.091982 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.091451 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 03 13:08:41.091982 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.091552 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 03 13:08:41.091982 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.091624 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-jqlk5\"" Nov 03 13:08:41.091982 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.091643 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 03 13:08:41.091982 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.091773 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 03 13:08:41.091982 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.091797 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Nov 03 13:08:41.091982 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.091815 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Nov 03 13:08:41.091982 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.091896 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 03 13:08:41.091982 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.091916 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 03 13:08:41.091982 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.091996 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 03 13:08:41.092529 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.092068 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Nov 03 13:08:41.092529 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.092074 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 03 13:08:41.092529 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.092083 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 03 13:08:41.092529 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.092158 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 03 13:08:41.092529 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.092171 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 03 13:08:41.092529 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.092173 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 03 13:08:41.092529 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.092195 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Nov 03 13:08:41.092529 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.092240 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\"" Nov 03 13:08:41.093369 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.093349 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Nov 03 13:08:41.093461 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.093387 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Nov 03 13:08:41.093461 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.093404 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Nov 03 13:08:41.093461 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.093452 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-62cxl\"" Nov 03 13:08:41.093572 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.093516 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 03 13:08:41.093572 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.093556 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 03 13:08:41.093645 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.093585 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-vd7sh\"" Nov 03 13:08:41.093645 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.093608 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 03 13:08:41.093645 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.093630 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 03 13:08:41.093645 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.093641 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 03 13:08:41.093743 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.093658 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-4mrd6\"" Nov 03 13:08:41.093743 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.093667 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Nov 03 13:08:41.093743 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.093720 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-psqgf\"" Nov 03 13:08:41.093743 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.093725 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-569wf\"" Nov 03 13:08:41.093863 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.093749 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Nov 03 13:08:41.093863 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.093784 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 03 13:08:41.093863 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.093811 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-s4cl2\"" Nov 03 13:08:41.093863 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.093820 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-hltl5\"" Nov 03 13:08:41.093983 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.093870 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-wv2nv\"" Nov 03 13:08:41.093983 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.093896 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 03 13:08:41.093983 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.093910 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 03 13:08:41.093983 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.093925 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-d95tp\"" Nov 03 13:08:41.094664 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.094645 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 03 13:08:41.094758 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.094676 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 03 13:08:41.094758 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.094680 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-knm89\"" Nov 03 13:08:41.094758 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.094704 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 03 13:08:41.094758 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.094732 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 03 13:08:41.094758 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.094735 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 03 13:08:41.095939 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.095920 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 03 13:08:41.096185 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.096168 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 03 13:08:41.096185 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.096184 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-gkggt\"" Nov 03 13:08:41.096821 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.096807 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-bbkgd\"" Nov 03 13:08:41.097079 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.097068 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 03 13:08:41.097079 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.097074 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 03 13:08:41.099578 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.099563 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 03 13:08:41.100925 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.100909 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-s2gms\"" Nov 03 13:08:41.101020 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.100964 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 03 13:08:41.101020 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.100989 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Nov 03 13:08:41.101020 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.101008 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Nov 03 13:08:41.101122 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.101030 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-wd8kt\"" Nov 03 13:08:41.101122 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.101080 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Nov 03 13:08:41.101965 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.101938 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Nov 03 13:08:41.102035 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.101970 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-zrpxj\"" Nov 03 13:08:41.102543 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.102514 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cb5g7\" (UniqueName: \"kubernetes.io/projected/c82d9f4b-8c00-4c31-8aa2-99cdc0d87e52-kube-api-access-cb5g7\") pod \"kas-connection-checker-685895c7c9-w5cql\" (UID: \"c82d9f4b-8c00-4c31-8aa2-99cdc0d87e52\") " pod="kube-system/kas-connection-checker-685895c7c9-w5cql" Nov 03 13:08:41.102543 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.102539 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zhwtz\" (UniqueName: \"kubernetes.io/projected/b8fe3220-d285-487f-9fc0-597438982de7-kube-api-access-zhwtz\") pod \"kas-connection-checker-685895c7c9-hslrf\" (UID: \"b8fe3220-d285-487f-9fc0-597438982de7\") " pod="kube-system/kas-connection-checker-685895c7c9-hslrf" Nov 03 13:08:41.102677 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.102631 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x7mw6\" (UniqueName: \"kubernetes.io/projected/6cb478ab-97aa-4881-87b2-b45293cc2349-kube-api-access-x7mw6\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-6nz8d\" (UID: \"6cb478ab-97aa-4881-87b2-b45293cc2349\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-6nz8d" Nov 03 13:08:41.102677 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.102647 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7jxlt\" (UniqueName: \"kubernetes.io/projected/07a7e2de-0c09-4819-9670-2be9460a155d-kube-api-access-7jxlt\") pod \"kas-connection-checker-685895c7c9-vl8fr\" (UID: \"07a7e2de-0c09-4819-9670-2be9460a155d\") " pod="kube-system/kas-connection-checker-685895c7c9-vl8fr" Nov 03 13:08:41.102816 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.102800 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qcsxn\" (UniqueName: \"kubernetes.io/projected/021e9e00-a973-4726-8f48-02a52c4bae14-kube-api-access-qcsxn\") pod \"volume-data-source-validator-58c9f5f46d-p6cbj\" (UID: \"021e9e00-a973-4726-8f48-02a52c4bae14\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-p6cbj" Nov 03 13:08:41.102896 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.102882 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4lj56\" (UniqueName: \"kubernetes.io/projected/a3042b60-062f-4f9b-8406-f884c95a8729-kube-api-access-4lj56\") pod \"cluster-monitoring-operator-884576c57-tm4nb\" (UID: \"a3042b60-062f-4f9b-8406-f884c95a8729\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-tm4nb" Nov 03 13:08:41.160228 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.160190 5024 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-36.us-east-2.compute.internal" Nov 03 13:08:41.160432 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.160377 5024 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-36.us-east-2.compute.internal" Nov 03 13:08:41.172338 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.172314 5024 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 13:08:41.172441 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:08:41.172354 5024 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-131-36.us-east-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-36.us-east-2.compute.internal" Nov 03 13:08:41.173026 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.173009 5024 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 13:08:41.173068 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:08:41.173047 5024 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-131-36.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-36.us-east-2.compute.internal" Nov 03 13:08:41.174312 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.174295 5024 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 03 13:08:41.189395 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.189367 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d926d945-39f4-48f3-b3a4-0b302b15b69b-env-overrides\") pod \"ovnkube-node-fg6vh\" (UID: \"d926d945-39f4-48f3-b3a4-0b302b15b69b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fg6vh" Nov 03 13:08:41.189501 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.189404 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bddlp\" (UniqueName: \"kubernetes.io/projected/3fdeb9f3-6c67-4390-8073-17d4d5f802b9-kube-api-access-bddlp\") pod \"dns-default-vgnv8\" (UID: \"3fdeb9f3-6c67-4390-8073-17d4d5f802b9\") " pod="openshift-dns/dns-default-vgnv8" Nov 03 13:08:41.189567 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.189537 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9fhg\" (UniqueName: \"kubernetes.io/projected/5c17f5bf-5b7d-4b05-a387-21ac6b5186ca-kube-api-access-k9fhg\") pod \"network-check-source-6f78d59f99-jws95\" (UID: \"5c17f5bf-5b7d-4b05-a387-21ac6b5186ca\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-jws95" Nov 03 13:08:41.189619 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.189577 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c704eb02-d156-4695-a55d-7dc96654e4cd-lib-modules\") pod \"tuned-hmbpc\" (UID: \"c704eb02-d156-4695-a55d-7dc96654e4cd\") " pod="openshift-cluster-node-tuning-operator/tuned-hmbpc" Nov 03 13:08:41.189672 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.189611 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/fa297a30-6da7-4a92-98fb-201323490bb0-nginx-conf\") pod \"networking-console-plugin-86c478bb88-rz4n9\" (UID: \"fa297a30-6da7-4a92-98fb-201323490bb0\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-rz4n9" Nov 03 13:08:41.189705 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.189680 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d926d945-39f4-48f3-b3a4-0b302b15b69b-env-overrides\") pod \"ovnkube-node-fg6vh\" (UID: \"d926d945-39f4-48f3-b3a4-0b302b15b69b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fg6vh" Nov 03 13:08:41.189705 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.189694 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/74dc4aee-10b1-463c-a9c3-522b7a6c10b7-multus-daemon-config\") pod \"multus-dsjsv\" (UID: \"74dc4aee-10b1-463c-a9c3-522b7a6c10b7\") " pod="openshift-multus/multus-dsjsv" Nov 03 13:08:41.189765 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.189729 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/898bd0b5-0d3e-4577-b101-2cc7d75283cb-metrics-certs\") pod \"router-default-7f579888d9-ntt6m\" (UID: \"898bd0b5-0d3e-4577-b101-2cc7d75283cb\") " pod="openshift-ingress/router-default-7f579888d9-ntt6m" Nov 03 13:08:41.189866 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.189850 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/fa297a30-6da7-4a92-98fb-201323490bb0-nginx-conf\") pod \"networking-console-plugin-86c478bb88-rz4n9\" (UID: \"fa297a30-6da7-4a92-98fb-201323490bb0\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-rz4n9" Nov 03 13:08:41.189910 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.189876 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d926d945-39f4-48f3-b3a4-0b302b15b69b-ovnkube-script-lib\") pod \"ovnkube-node-fg6vh\" (UID: \"d926d945-39f4-48f3-b3a4-0b302b15b69b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fg6vh" Nov 03 13:08:41.189941 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.189916 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/74dc4aee-10b1-463c-a9c3-522b7a6c10b7-host-var-lib-cni-multus\") pod \"multus-dsjsv\" (UID: \"74dc4aee-10b1-463c-a9c3-522b7a6c10b7\") " pod="openshift-multus/multus-dsjsv" Nov 03 13:08:41.189970 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.189936 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3fdeb9f3-6c67-4390-8073-17d4d5f802b9-metrics-tls\") pod \"dns-default-vgnv8\" (UID: \"3fdeb9f3-6c67-4390-8073-17d4d5f802b9\") " pod="openshift-dns/dns-default-vgnv8" Nov 03 13:08:41.189970 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.189962 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/cc8ef30e-6d4f-4913-8dc8-b09e5aea2f82-installation-pull-secrets\") pod \"image-registry-555f8976d-mz5ct\" (UID: \"cc8ef30e-6d4f-4913-8dc8-b09e5aea2f82\") " pod="openshift-image-registry/image-registry-555f8976d-mz5ct" Nov 03 13:08:41.190034 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.189989 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/74dc4aee-10b1-463c-a9c3-522b7a6c10b7-multus-daemon-config\") pod \"multus-dsjsv\" (UID: \"74dc4aee-10b1-463c-a9c3-522b7a6c10b7\") " pod="openshift-multus/multus-dsjsv" Nov 03 13:08:41.190071 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.190036 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pt6qk\" (UniqueName: \"kubernetes.io/projected/cd91de08-084e-4667-b13f-2e07b3cff624-kube-api-access-pt6qk\") pod \"console-operator-7cd56b8dbc-6htht\" (UID: \"cd91de08-084e-4667-b13f-2e07b3cff624\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-6htht" Nov 03 13:08:41.190071 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.190053 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d926d945-39f4-48f3-b3a4-0b302b15b69b-ovnkube-script-lib\") pod \"ovnkube-node-fg6vh\" (UID: \"d926d945-39f4-48f3-b3a4-0b302b15b69b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fg6vh" Nov 03 13:08:41.190129 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.190091 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3fdeb9f3-6c67-4390-8073-17d4d5f802b9-tmp-dir\") pod \"dns-default-vgnv8\" (UID: \"3fdeb9f3-6c67-4390-8073-17d4d5f802b9\") " pod="openshift-dns/dns-default-vgnv8" Nov 03 13:08:41.190129 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.190125 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9db7d195-220e-4644-8690-9e5ee37fc066-kubelet-dir\") pod \"aws-ebs-csi-driver-node-c9dvm\" (UID: \"9db7d195-220e-4644-8690-9e5ee37fc066\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c9dvm" Nov 03 13:08:41.190190 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.190169 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwrs7\" (UniqueName: \"kubernetes.io/projected/74dc4aee-10b1-463c-a9c3-522b7a6c10b7-kube-api-access-cwrs7\") pod \"multus-dsjsv\" (UID: \"74dc4aee-10b1-463c-a9c3-522b7a6c10b7\") " pod="openshift-multus/multus-dsjsv" Nov 03 13:08:41.190190 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.190174 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/cc8ef30e-6d4f-4913-8dc8-b09e5aea2f82-installation-pull-secrets\") pod \"image-registry-555f8976d-mz5ct\" (UID: \"cc8ef30e-6d4f-4913-8dc8-b09e5aea2f82\") " pod="openshift-image-registry/image-registry-555f8976d-mz5ct" Nov 03 13:08:41.190190 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.190185 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3fdeb9f3-6c67-4390-8073-17d4d5f802b9-tmp-dir\") pod \"dns-default-vgnv8\" (UID: \"3fdeb9f3-6c67-4390-8073-17d4d5f802b9\") " pod="openshift-dns/dns-default-vgnv8" Nov 03 13:08:41.190300 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.190220 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d926d945-39f4-48f3-b3a4-0b302b15b69b-host-run-ovn-kubernetes\") pod \"ovnkube-node-fg6vh\" (UID: \"d926d945-39f4-48f3-b3a4-0b302b15b69b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fg6vh" Nov 03 13:08:41.190300 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.190246 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c704eb02-d156-4695-a55d-7dc96654e4cd-etc-sysctl-d\") pod \"tuned-hmbpc\" (UID: \"c704eb02-d156-4695-a55d-7dc96654e4cd\") " pod="openshift-cluster-node-tuning-operator/tuned-hmbpc" Nov 03 13:08:41.190371 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.190303 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0a892eb3-ecc4-4312-9f4d-73eeba6b6eee-bound-sa-token\") pod \"image-registry-7dbb67bd57-74qfj\" (UID: \"0a892eb3-ecc4-4312-9f4d-73eeba6b6eee\") " pod="openshift-image-registry/image-registry-7dbb67bd57-74qfj" Nov 03 13:08:41.190371 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.190330 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/fff35d1b-e7b2-4dd3-9824-8b64fad02506-data-volume\") pod \"insights-runtime-extractor-d7mwz\" (UID: \"fff35d1b-e7b2-4dd3-9824-8b64fad02506\") " pod="openshift-insights/insights-runtime-extractor-d7mwz" Nov 03 13:08:41.190371 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.190357 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/50d68cb7-b766-4674-b295-8f3229880994-cert\") pod \"ingress-canary-d8f6l\" (UID: \"50d68cb7-b766-4674-b295-8f3229880994\") " pod="openshift-ingress-canary/ingress-canary-d8f6l" Nov 03 13:08:41.190468 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.190378 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/74dc4aee-10b1-463c-a9c3-522b7a6c10b7-etc-kubernetes\") pod \"multus-dsjsv\" (UID: \"74dc4aee-10b1-463c-a9c3-522b7a6c10b7\") " pod="openshift-multus/multus-dsjsv" Nov 03 13:08:41.190468 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.190396 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cc8ef30e-6d4f-4913-8dc8-b09e5aea2f82-registry-tls\") pod \"image-registry-555f8976d-mz5ct\" (UID: \"cc8ef30e-6d4f-4913-8dc8-b09e5aea2f82\") " pod="openshift-image-registry/image-registry-555f8976d-mz5ct" Nov 03 13:08:41.190468 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.190428 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d926d945-39f4-48f3-b3a4-0b302b15b69b-host-run-netns\") pod \"ovnkube-node-fg6vh\" (UID: \"d926d945-39f4-48f3-b3a4-0b302b15b69b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fg6vh" Nov 03 13:08:41.190468 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.190395 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/fff35d1b-e7b2-4dd3-9824-8b64fad02506-data-volume\") pod \"insights-runtime-extractor-d7mwz\" (UID: \"fff35d1b-e7b2-4dd3-9824-8b64fad02506\") " pod="openshift-insights/insights-runtime-extractor-d7mwz" Nov 03 13:08:41.190468 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.190448 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c704eb02-d156-4695-a55d-7dc96654e4cd-etc-sysconfig\") pod \"tuned-hmbpc\" (UID: \"c704eb02-d156-4695-a55d-7dc96654e4cd\") " pod="openshift-cluster-node-tuning-operator/tuned-hmbpc" Nov 03 13:08:41.190613 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.190473 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fd7e8b58-de31-406a-8cdb-9ccfee140d57-host-slash\") pod \"iptables-alerter-g4xgp\" (UID: \"fd7e8b58-de31-406a-8cdb-9ccfee140d57\") " pod="openshift-network-operator/iptables-alerter-g4xgp" Nov 03 13:08:41.190613 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.190497 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7e280277-f29f-4a2b-8b85-2491529e5ef2-original-pull-secret\") pod \"global-pull-secret-syncer-5jz7t\" (UID: \"7e280277-f29f-4a2b-8b85-2491529e5ef2\") " pod="kube-system/global-pull-secret-syncer-5jz7t" Nov 03 13:08:41.190613 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.190520 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8x9b\" (UniqueName: \"kubernetes.io/projected/9fa6d32d-b30b-4b76-9462-5f3f09443d02-kube-api-access-n8x9b\") pod \"insights-operator-7c4986d449-4t8lj\" (UID: \"9fa6d32d-b30b-4b76-9462-5f3f09443d02\") " pod="openshift-insights/insights-operator-7c4986d449-4t8lj" Nov 03 13:08:41.190613 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.190562 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/50d68cb7-b766-4674-b295-8f3229880994-cert\") pod \"ingress-canary-d8f6l\" (UID: \"50d68cb7-b766-4674-b295-8f3229880994\") " pod="openshift-ingress-canary/ingress-canary-d8f6l" Nov 03 13:08:41.190613 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.190562 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/898bd0b5-0d3e-4577-b101-2cc7d75283cb-service-ca-bundle\") pod \"router-default-7f579888d9-ntt6m\" (UID: \"898bd0b5-0d3e-4577-b101-2cc7d75283cb\") " pod="openshift-ingress/router-default-7f579888d9-ntt6m" Nov 03 13:08:41.190832 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.190642 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/898bd0b5-0d3e-4577-b101-2cc7d75283cb-stats-auth\") pod \"router-default-7f579888d9-ntt6m\" (UID: \"898bd0b5-0d3e-4577-b101-2cc7d75283cb\") " pod="openshift-ingress/router-default-7f579888d9-ntt6m" Nov 03 13:08:41.190832 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.190677 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbfgb\" (UniqueName: \"kubernetes.io/projected/1bfc1afc-5b9f-4447-8d40-fabedbee9c07-kube-api-access-hbfgb\") pod \"network-check-target-8rdgq\" (UID: \"1bfc1afc-5b9f-4447-8d40-fabedbee9c07\") " pod="openshift-network-diagnostics/network-check-target-8rdgq" Nov 03 13:08:41.190832 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.190695 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mss42\" (UniqueName: \"kubernetes.io/projected/35e0fb40-48ce-4223-b7d4-c9290b6861cd-kube-api-access-mss42\") pod \"multus-additional-cni-plugins-x2rm4\" (UID: \"35e0fb40-48ce-4223-b7d4-c9290b6861cd\") " pod="openshift-multus/multus-additional-cni-plugins-x2rm4" Nov 03 13:08:41.190832 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.190744 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7e280277-f29f-4a2b-8b85-2491529e5ef2-original-pull-secret\") pod \"global-pull-secret-syncer-5jz7t\" (UID: \"7e280277-f29f-4a2b-8b85-2491529e5ef2\") " pod="kube-system/global-pull-secret-syncer-5jz7t" Nov 03 13:08:41.190832 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.190751 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d926d945-39f4-48f3-b3a4-0b302b15b69b-systemd-units\") pod \"ovnkube-node-fg6vh\" (UID: \"d926d945-39f4-48f3-b3a4-0b302b15b69b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fg6vh" Nov 03 13:08:41.190832 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.190780 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c704eb02-d156-4695-a55d-7dc96654e4cd-etc-modprobe-d\") pod \"tuned-hmbpc\" (UID: \"c704eb02-d156-4695-a55d-7dc96654e4cd\") " pod="openshift-cluster-node-tuning-operator/tuned-hmbpc" Nov 03 13:08:41.191002 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.190823 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c704eb02-d156-4695-a55d-7dc96654e4cd-etc-sysctl-conf\") pod \"tuned-hmbpc\" (UID: \"c704eb02-d156-4695-a55d-7dc96654e4cd\") " pod="openshift-cluster-node-tuning-operator/tuned-hmbpc" Nov 03 13:08:41.191002 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.190866 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/cd91de08-084e-4667-b13f-2e07b3cff624-config\") pod \"console-operator-7cd56b8dbc-6htht\" (UID: \"cd91de08-084e-4667-b13f-2e07b3cff624\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-6htht" Nov 03 13:08:41.191002 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.190886 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/898bd0b5-0d3e-4577-b101-2cc7d75283cb-stats-auth\") pod \"router-default-7f579888d9-ntt6m\" (UID: \"898bd0b5-0d3e-4577-b101-2cc7d75283cb\") " pod="openshift-ingress/router-default-7f579888d9-ntt6m" Nov 03 13:08:41.191002 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.190889 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/39e11989-e598-4cea-87d9-26b3cf78a66b-serviceca\") pod \"node-ca-4tn4j\" (UID: \"39e11989-e598-4cea-87d9-26b3cf78a66b\") " pod="openshift-image-registry/node-ca-4tn4j" Nov 03 13:08:41.191002 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.190919 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/74dc4aee-10b1-463c-a9c3-522b7a6c10b7-host-run-k8s-cni-cncf-io\") pod \"multus-dsjsv\" (UID: \"74dc4aee-10b1-463c-a9c3-522b7a6c10b7\") " pod="openshift-multus/multus-dsjsv" Nov 03 13:08:41.191002 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.190958 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d926d945-39f4-48f3-b3a4-0b302b15b69b-run-systemd\") pod \"ovnkube-node-fg6vh\" (UID: \"d926d945-39f4-48f3-b3a4-0b302b15b69b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fg6vh" Nov 03 13:08:41.191160 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.191007 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c704eb02-d156-4695-a55d-7dc96654e4cd-tmp\") pod \"tuned-hmbpc\" (UID: \"c704eb02-d156-4695-a55d-7dc96654e4cd\") " pod="openshift-cluster-node-tuning-operator/tuned-hmbpc" Nov 03 13:08:41.191160 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.191028 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/cd91de08-084e-4667-b13f-2e07b3cff624-config\") pod \"console-operator-7cd56b8dbc-6htht\" (UID: \"cd91de08-084e-4667-b13f-2e07b3cff624\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-6htht" Nov 03 13:08:41.191160 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.191032 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/74dc4aee-10b1-463c-a9c3-522b7a6c10b7-host-var-lib-cni-bin\") pod \"multus-dsjsv\" (UID: \"74dc4aee-10b1-463c-a9c3-522b7a6c10b7\") " pod="openshift-multus/multus-dsjsv" Nov 03 13:08:41.191160 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.191069 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5ea5cf61-ed1c-4f5e-b1a0-5c46c5c3ee18-tmp-dir\") pod \"node-resolver-v8gcq\" (UID: \"5ea5cf61-ed1c-4f5e-b1a0-5c46c5c3ee18\") " pod="openshift-dns/node-resolver-v8gcq" Nov 03 13:08:41.191160 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.191093 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c704eb02-d156-4695-a55d-7dc96654e4cd-tmp\") pod \"tuned-hmbpc\" (UID: \"c704eb02-d156-4695-a55d-7dc96654e4cd\") " pod="openshift-cluster-node-tuning-operator/tuned-hmbpc" Nov 03 13:08:41.191160 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.191093 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fff35d1b-e7b2-4dd3-9824-8b64fad02506-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-d7mwz\" (UID: \"fff35d1b-e7b2-4dd3-9824-8b64fad02506\") " pod="openshift-insights/insights-runtime-extractor-d7mwz" Nov 03 13:08:41.191160 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.191125 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/74dc4aee-10b1-463c-a9c3-522b7a6c10b7-multus-conf-dir\") pod \"multus-dsjsv\" (UID: \"74dc4aee-10b1-463c-a9c3-522b7a6c10b7\") " pod="openshift-multus/multus-dsjsv" Nov 03 13:08:41.191160 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.191142 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/7e280277-f29f-4a2b-8b85-2491529e5ef2-kubelet-config\") pod \"global-pull-secret-syncer-5jz7t\" (UID: \"7e280277-f29f-4a2b-8b85-2491529e5ef2\") " pod="kube-system/global-pull-secret-syncer-5jz7t" Nov 03 13:08:41.191160 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.191143 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/39e11989-e598-4cea-87d9-26b3cf78a66b-serviceca\") pod \"node-ca-4tn4j\" (UID: \"39e11989-e598-4cea-87d9-26b3cf78a66b\") " pod="openshift-image-registry/node-ca-4tn4j" Nov 03 13:08:41.191160 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.191159 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cc8ef30e-6d4f-4913-8dc8-b09e5aea2f82-bound-sa-token\") pod \"image-registry-555f8976d-mz5ct\" (UID: \"cc8ef30e-6d4f-4913-8dc8-b09e5aea2f82\") " pod="openshift-image-registry/image-registry-555f8976d-mz5ct" Nov 03 13:08:41.191524 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.191180 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d926d945-39f4-48f3-b3a4-0b302b15b69b-node-log\") pod \"ovnkube-node-fg6vh\" (UID: \"d926d945-39f4-48f3-b3a4-0b302b15b69b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fg6vh" Nov 03 13:08:41.191524 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.191217 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5ea5cf61-ed1c-4f5e-b1a0-5c46c5c3ee18-tmp-dir\") pod \"node-resolver-v8gcq\" (UID: \"5ea5cf61-ed1c-4f5e-b1a0-5c46c5c3ee18\") " pod="openshift-dns/node-resolver-v8gcq" Nov 03 13:08:41.191524 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.191266 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fff35d1b-e7b2-4dd3-9824-8b64fad02506-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-d7mwz\" (UID: \"fff35d1b-e7b2-4dd3-9824-8b64fad02506\") " pod="openshift-insights/insights-runtime-extractor-d7mwz" Nov 03 13:08:41.191524 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.191307 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d926d945-39f4-48f3-b3a4-0b302b15b69b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-fg6vh\" (UID: \"d926d945-39f4-48f3-b3a4-0b302b15b69b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fg6vh" Nov 03 13:08:41.191524 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.191338 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9f20ddda-41fb-4aba-b257-7ebc59dbe678-config\") pod \"service-ca-operator-544d749c8c-cqvdv\" (UID: \"9f20ddda-41fb-4aba-b257-7ebc59dbe678\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-cqvdv" Nov 03 13:08:41.191524 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.191359 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xmbd7\" (UniqueName: \"kubernetes.io/projected/0a892eb3-ecc4-4312-9f4d-73eeba6b6eee-kube-api-access-xmbd7\") pod \"image-registry-7dbb67bd57-74qfj\" (UID: \"0a892eb3-ecc4-4312-9f4d-73eeba6b6eee\") " pod="openshift-image-registry/image-registry-7dbb67bd57-74qfj" Nov 03 13:08:41.191524 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.191406 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d926d945-39f4-48f3-b3a4-0b302b15b69b-host-kubelet\") pod \"ovnkube-node-fg6vh\" (UID: \"d926d945-39f4-48f3-b3a4-0b302b15b69b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fg6vh" Nov 03 13:08:41.191524 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.191512 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/21618783-0a4e-4b1f-8041-cabdf69f5bea-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-h46gl\" (UID: \"21618783-0a4e-4b1f-8041-cabdf69f5bea\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-h46gl" Nov 03 13:08:41.191890 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.191541 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/cd91de08-084e-4667-b13f-2e07b3cff624-trusted-ca\") pod \"console-operator-7cd56b8dbc-6htht\" (UID: \"cd91de08-084e-4667-b13f-2e07b3cff624\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-6htht" Nov 03 13:08:41.191890 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.191585 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/fd7e8b58-de31-406a-8cdb-9ccfee140d57-iptables-alerter-script\") pod \"iptables-alerter-g4xgp\" (UID: \"fd7e8b58-de31-406a-8cdb-9ccfee140d57\") " pod="openshift-network-operator/iptables-alerter-g4xgp" Nov 03 13:08:41.191890 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.191589 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9f20ddda-41fb-4aba-b257-7ebc59dbe678-config\") pod \"service-ca-operator-544d749c8c-cqvdv\" (UID: \"9f20ddda-41fb-4aba-b257-7ebc59dbe678\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-cqvdv" Nov 03 13:08:41.191890 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.191652 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jknsv\" (UniqueName: \"kubernetes.io/projected/9f20ddda-41fb-4aba-b257-7ebc59dbe678-kube-api-access-jknsv\") pod \"service-ca-operator-544d749c8c-cqvdv\" (UID: \"9f20ddda-41fb-4aba-b257-7ebc59dbe678\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-cqvdv" Nov 03 13:08:41.192079 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.192055 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d926d945-39f4-48f3-b3a4-0b302b15b69b-etc-openvswitch\") pod \"ovnkube-node-fg6vh\" (UID: \"d926d945-39f4-48f3-b3a4-0b302b15b69b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fg6vh" Nov 03 13:08:41.192229 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.192201 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/cd91de08-084e-4667-b13f-2e07b3cff624-trusted-ca\") pod \"console-operator-7cd56b8dbc-6htht\" (UID: \"cd91de08-084e-4667-b13f-2e07b3cff624\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-6htht" Nov 03 13:08:41.192229 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.192212 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6b28l\" (UniqueName: \"kubernetes.io/projected/d926d945-39f4-48f3-b3a4-0b302b15b69b-kube-api-access-6b28l\") pod \"ovnkube-node-fg6vh\" (UID: \"d926d945-39f4-48f3-b3a4-0b302b15b69b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fg6vh" Nov 03 13:08:41.192360 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.192310 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtd8p\" (UniqueName: \"kubernetes.io/projected/bd523616-71b2-4b6c-b82c-f1efb585c271-kube-api-access-mtd8p\") pod \"migrator-66df5d496b-gkblf\" (UID: \"bd523616-71b2-4b6c-b82c-f1efb585c271\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-gkblf" Nov 03 13:08:41.192547 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.192518 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/cd91de08-084e-4667-b13f-2e07b3cff624-serving-cert\") pod \"console-operator-7cd56b8dbc-6htht\" (UID: \"cd91de08-084e-4667-b13f-2e07b3cff624\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-6htht" Nov 03 13:08:41.192703 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.192682 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/fa297a30-6da7-4a92-98fb-201323490bb0-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-rz4n9\" (UID: \"fa297a30-6da7-4a92-98fb-201323490bb0\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-rz4n9" Nov 03 13:08:41.192761 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.192729 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fff35d1b-e7b2-4dd3-9824-8b64fad02506-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-d7mwz\" (UID: \"fff35d1b-e7b2-4dd3-9824-8b64fad02506\") " pod="openshift-insights/insights-runtime-extractor-d7mwz" Nov 03 13:08:41.192831 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.192789 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9db7d195-220e-4644-8690-9e5ee37fc066-registration-dir\") pod \"aws-ebs-csi-driver-node-c9dvm\" (UID: \"9db7d195-220e-4644-8690-9e5ee37fc066\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c9dvm" Nov 03 13:08:41.192831 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.192822 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d926d945-39f4-48f3-b3a4-0b302b15b69b-run-openvswitch\") pod \"ovnkube-node-fg6vh\" (UID: \"d926d945-39f4-48f3-b3a4-0b302b15b69b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fg6vh" Nov 03 13:08:41.192966 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.192858 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9f20ddda-41fb-4aba-b257-7ebc59dbe678-serving-cert\") pod \"service-ca-operator-544d749c8c-cqvdv\" (UID: \"9f20ddda-41fb-4aba-b257-7ebc59dbe678\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-cqvdv" Nov 03 13:08:41.192966 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.192884 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/74dc4aee-10b1-463c-a9c3-522b7a6c10b7-host-run-netns\") pod \"multus-dsjsv\" (UID: \"74dc4aee-10b1-463c-a9c3-522b7a6c10b7\") " pod="openshift-multus/multus-dsjsv" Nov 03 13:08:41.192966 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.192915 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/898bd0b5-0d3e-4577-b101-2cc7d75283cb-default-certificate\") pod \"router-default-7f579888d9-ntt6m\" (UID: \"898bd0b5-0d3e-4577-b101-2cc7d75283cb\") " pod="openshift-ingress/router-default-7f579888d9-ntt6m" Nov 03 13:08:41.192966 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.192947 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d926d945-39f4-48f3-b3a4-0b302b15b69b-var-lib-openvswitch\") pod \"ovnkube-node-fg6vh\" (UID: \"d926d945-39f4-48f3-b3a4-0b302b15b69b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fg6vh" Nov 03 13:08:41.193191 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.192978 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zpq8b\" (UniqueName: \"kubernetes.io/projected/39e11989-e598-4cea-87d9-26b3cf78a66b-kube-api-access-zpq8b\") pod \"node-ca-4tn4j\" (UID: \"39e11989-e598-4cea-87d9-26b3cf78a66b\") " pod="openshift-image-registry/node-ca-4tn4j" Nov 03 13:08:41.193191 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.193004 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/74dc4aee-10b1-463c-a9c3-522b7a6c10b7-cnibin\") pod \"multus-dsjsv\" (UID: \"74dc4aee-10b1-463c-a9c3-522b7a6c10b7\") " pod="openshift-multus/multus-dsjsv" Nov 03 13:08:41.193191 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.193034 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/35e0fb40-48ce-4223-b7d4-c9290b6861cd-system-cni-dir\") pod \"multus-additional-cni-plugins-x2rm4\" (UID: \"35e0fb40-48ce-4223-b7d4-c9290b6861cd\") " pod="openshift-multus/multus-additional-cni-plugins-x2rm4" Nov 03 13:08:41.193191 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.193034 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/cd91de08-084e-4667-b13f-2e07b3cff624-serving-cert\") pod \"console-operator-7cd56b8dbc-6htht\" (UID: \"cd91de08-084e-4667-b13f-2e07b3cff624\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-6htht" Nov 03 13:08:41.193432 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.193189 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9f20ddda-41fb-4aba-b257-7ebc59dbe678-serving-cert\") pod \"service-ca-operator-544d749c8c-cqvdv\" (UID: \"9f20ddda-41fb-4aba-b257-7ebc59dbe678\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-cqvdv" Nov 03 13:08:41.193432 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.193235 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9db7d195-220e-4644-8690-9e5ee37fc066-socket-dir\") pod \"aws-ebs-csi-driver-node-c9dvm\" (UID: \"9db7d195-220e-4644-8690-9e5ee37fc066\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c9dvm" Nov 03 13:08:41.193432 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.193271 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9db7d195-220e-4644-8690-9e5ee37fc066-etc-selinux\") pod \"aws-ebs-csi-driver-node-c9dvm\" (UID: \"9db7d195-220e-4644-8690-9e5ee37fc066\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c9dvm" Nov 03 13:08:41.193432 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.193351 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/898bd0b5-0d3e-4577-b101-2cc7d75283cb-default-certificate\") pod \"router-default-7f579888d9-ntt6m\" (UID: \"898bd0b5-0d3e-4577-b101-2cc7d75283cb\") " pod="openshift-ingress/router-default-7f579888d9-ntt6m" Nov 03 13:08:41.193798 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.193779 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d926d945-39f4-48f3-b3a4-0b302b15b69b-run-ovn\") pod \"ovnkube-node-fg6vh\" (UID: \"d926d945-39f4-48f3-b3a4-0b302b15b69b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fg6vh" Nov 03 13:08:41.193872 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.193820 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c704eb02-d156-4695-a55d-7dc96654e4cd-var-lib-kubelet\") pod \"tuned-hmbpc\" (UID: \"c704eb02-d156-4695-a55d-7dc96654e4cd\") " pod="openshift-cluster-node-tuning-operator/tuned-hmbpc" Nov 03 13:08:41.193872 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.193855 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bcz25\" (UniqueName: \"kubernetes.io/projected/e0300f55-f002-4faa-bf6a-c04bea0a043e-kube-api-access-bcz25\") pod \"network-metrics-daemon-f7hmh\" (UID: \"e0300f55-f002-4faa-bf6a-c04bea0a043e\") " pod="openshift-multus/network-metrics-daemon-f7hmh" Nov 03 13:08:41.193985 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.193885 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jssfx\" (UniqueName: \"kubernetes.io/projected/fff35d1b-e7b2-4dd3-9824-8b64fad02506-kube-api-access-jssfx\") pod \"insights-runtime-extractor-d7mwz\" (UID: \"fff35d1b-e7b2-4dd3-9824-8b64fad02506\") " pod="openshift-insights/insights-runtime-extractor-d7mwz" Nov 03 13:08:41.193985 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.193940 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h82mn\" (UniqueName: \"kubernetes.io/projected/cc8ef30e-6d4f-4913-8dc8-b09e5aea2f82-kube-api-access-h82mn\") pod \"image-registry-555f8976d-mz5ct\" (UID: \"cc8ef30e-6d4f-4913-8dc8-b09e5aea2f82\") " pod="openshift-image-registry/image-registry-555f8976d-mz5ct" Nov 03 13:08:41.193985 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.193980 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d926d945-39f4-48f3-b3a4-0b302b15b69b-host-slash\") pod \"ovnkube-node-fg6vh\" (UID: \"d926d945-39f4-48f3-b3a4-0b302b15b69b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fg6vh" Nov 03 13:08:41.194217 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.194009 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/74dc4aee-10b1-463c-a9c3-522b7a6c10b7-system-cni-dir\") pod \"multus-dsjsv\" (UID: \"74dc4aee-10b1-463c-a9c3-522b7a6c10b7\") " pod="openshift-multus/multus-dsjsv" Nov 03 13:08:41.194217 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.194045 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/74dc4aee-10b1-463c-a9c3-522b7a6c10b7-cni-binary-copy\") pod \"multus-dsjsv\" (UID: \"74dc4aee-10b1-463c-a9c3-522b7a6c10b7\") " pod="openshift-multus/multus-dsjsv" Nov 03 13:08:41.194217 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.194072 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0a892eb3-ecc4-4312-9f4d-73eeba6b6eee-registry-certificates\") pod \"image-registry-7dbb67bd57-74qfj\" (UID: \"0a892eb3-ecc4-4312-9f4d-73eeba6b6eee\") " pod="openshift-image-registry/image-registry-7dbb67bd57-74qfj" Nov 03 13:08:41.194217 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.194106 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d926d945-39f4-48f3-b3a4-0b302b15b69b-ovn-node-metrics-cert\") pod \"ovnkube-node-fg6vh\" (UID: \"d926d945-39f4-48f3-b3a4-0b302b15b69b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fg6vh" Nov 03 13:08:41.194217 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.194139 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2sxmr\" (UniqueName: \"kubernetes.io/projected/50d68cb7-b766-4674-b295-8f3229880994-kube-api-access-2sxmr\") pod \"ingress-canary-d8f6l\" (UID: \"50d68cb7-b766-4674-b295-8f3229880994\") " pod="openshift-ingress-canary/ingress-canary-d8f6l" Nov 03 13:08:41.194217 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.194170 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0a892eb3-ecc4-4312-9f4d-73eeba6b6eee-installation-pull-secrets\") pod \"image-registry-7dbb67bd57-74qfj\" (UID: \"0a892eb3-ecc4-4312-9f4d-73eeba6b6eee\") " pod="openshift-image-registry/image-registry-7dbb67bd57-74qfj" Nov 03 13:08:41.194217 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.194196 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/cc8ef30e-6d4f-4913-8dc8-b09e5aea2f82-image-registry-private-configuration\") pod \"image-registry-555f8976d-mz5ct\" (UID: \"cc8ef30e-6d4f-4913-8dc8-b09e5aea2f82\") " pod="openshift-image-registry/image-registry-555f8976d-mz5ct" Nov 03 13:08:41.194646 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.194226 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/cc8ef30e-6d4f-4913-8dc8-b09e5aea2f82-trusted-ca\") pod \"image-registry-555f8976d-mz5ct\" (UID: \"cc8ef30e-6d4f-4913-8dc8-b09e5aea2f82\") " pod="openshift-image-registry/image-registry-555f8976d-mz5ct" Nov 03 13:08:41.194646 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.194257 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9db7d195-220e-4644-8690-9e5ee37fc066-sys-fs\") pod \"aws-ebs-csi-driver-node-c9dvm\" (UID: \"9db7d195-220e-4644-8690-9e5ee37fc066\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c9dvm" Nov 03 13:08:41.194646 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.194312 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-snpf4\" (UniqueName: \"kubernetes.io/projected/fd7e8b58-de31-406a-8cdb-9ccfee140d57-kube-api-access-snpf4\") pod \"iptables-alerter-g4xgp\" (UID: \"fd7e8b58-de31-406a-8cdb-9ccfee140d57\") " pod="openshift-network-operator/iptables-alerter-g4xgp" Nov 03 13:08:41.194646 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.194345 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/35e0fb40-48ce-4223-b7d4-c9290b6861cd-cnibin\") pod \"multus-additional-cni-plugins-x2rm4\" (UID: \"35e0fb40-48ce-4223-b7d4-c9290b6861cd\") " pod="openshift-multus/multus-additional-cni-plugins-x2rm4" Nov 03 13:08:41.194646 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.194371 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/fff35d1b-e7b2-4dd3-9824-8b64fad02506-crio-socket\") pod \"insights-runtime-extractor-d7mwz\" (UID: \"fff35d1b-e7b2-4dd3-9824-8b64fad02506\") " pod="openshift-insights/insights-runtime-extractor-d7mwz" Nov 03 13:08:41.194646 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.194404 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/cc8ef30e-6d4f-4913-8dc8-b09e5aea2f82-registry-certificates\") pod \"image-registry-555f8976d-mz5ct\" (UID: \"cc8ef30e-6d4f-4913-8dc8-b09e5aea2f82\") " pod="openshift-image-registry/image-registry-555f8976d-mz5ct" Nov 03 13:08:41.194646 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.194436 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4j5g\" (UniqueName: \"kubernetes.io/projected/c704eb02-d156-4695-a55d-7dc96654e4cd-kube-api-access-w4j5g\") pod \"tuned-hmbpc\" (UID: \"c704eb02-d156-4695-a55d-7dc96654e4cd\") " pod="openshift-cluster-node-tuning-operator/tuned-hmbpc" Nov 03 13:08:41.194646 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.194438 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0a892eb3-ecc4-4312-9f4d-73eeba6b6eee-registry-certificates\") pod \"image-registry-7dbb67bd57-74qfj\" (UID: \"0a892eb3-ecc4-4312-9f4d-73eeba6b6eee\") " pod="openshift-image-registry/image-registry-7dbb67bd57-74qfj" Nov 03 13:08:41.194646 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.194471 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/7e280277-f29f-4a2b-8b85-2491529e5ef2-dbus\") pod \"global-pull-secret-syncer-5jz7t\" (UID: \"7e280277-f29f-4a2b-8b85-2491529e5ef2\") " pod="kube-system/global-pull-secret-syncer-5jz7t" Nov 03 13:08:41.194646 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.194502 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9fa6d32d-b30b-4b76-9462-5f3f09443d02-tmp\") pod \"insights-operator-7c4986d449-4t8lj\" (UID: \"9fa6d32d-b30b-4b76-9462-5f3f09443d02\") " pod="openshift-insights/insights-operator-7c4986d449-4t8lj" Nov 03 13:08:41.194646 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.194565 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/74dc4aee-10b1-463c-a9c3-522b7a6c10b7-hostroot\") pod \"multus-dsjsv\" (UID: \"74dc4aee-10b1-463c-a9c3-522b7a6c10b7\") " pod="openshift-multus/multus-dsjsv" Nov 03 13:08:41.194646 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.194605 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/9fa6d32d-b30b-4b76-9462-5f3f09443d02-snapshots\") pod \"insights-operator-7c4986d449-4t8lj\" (UID: \"9fa6d32d-b30b-4b76-9462-5f3f09443d02\") " pod="openshift-insights/insights-operator-7c4986d449-4t8lj" Nov 03 13:08:41.194646 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.194644 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/35e0fb40-48ce-4223-b7d4-c9290b6861cd-cni-binary-copy\") pod \"multus-additional-cni-plugins-x2rm4\" (UID: \"35e0fb40-48ce-4223-b7d4-c9290b6861cd\") " pod="openshift-multus/multus-additional-cni-plugins-x2rm4" Nov 03 13:08:41.195336 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.194681 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/35e0fb40-48ce-4223-b7d4-c9290b6861cd-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-x2rm4\" (UID: \"35e0fb40-48ce-4223-b7d4-c9290b6861cd\") " pod="openshift-multus/multus-additional-cni-plugins-x2rm4" Nov 03 13:08:41.195336 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.194714 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjhjt\" (UniqueName: \"kubernetes.io/projected/5ea5cf61-ed1c-4f5e-b1a0-5c46c5c3ee18-kube-api-access-vjhjt\") pod \"node-resolver-v8gcq\" (UID: \"5ea5cf61-ed1c-4f5e-b1a0-5c46c5c3ee18\") " pod="openshift-dns/node-resolver-v8gcq" Nov 03 13:08:41.195336 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.194756 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/3fdeb9f3-6c67-4390-8073-17d4d5f802b9-config-volume\") pod \"dns-default-vgnv8\" (UID: \"3fdeb9f3-6c67-4390-8073-17d4d5f802b9\") " pod="openshift-dns/dns-default-vgnv8" Nov 03 13:08:41.195336 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.194799 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d926d945-39f4-48f3-b3a4-0b302b15b69b-host-cni-bin\") pod \"ovnkube-node-fg6vh\" (UID: \"d926d945-39f4-48f3-b3a4-0b302b15b69b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fg6vh" Nov 03 13:08:41.195336 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.194950 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/cc8ef30e-6d4f-4913-8dc8-b09e5aea2f82-registry-certificates\") pod \"image-registry-555f8976d-mz5ct\" (UID: \"cc8ef30e-6d4f-4913-8dc8-b09e5aea2f82\") " pod="openshift-image-registry/image-registry-555f8976d-mz5ct" Nov 03 13:08:41.195336 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.195001 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ftcqc\" (UniqueName: \"kubernetes.io/projected/9db7d195-220e-4644-8690-9e5ee37fc066-kube-api-access-ftcqc\") pod \"aws-ebs-csi-driver-node-c9dvm\" (UID: \"9db7d195-220e-4644-8690-9e5ee37fc066\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c9dvm" Nov 03 13:08:41.195336 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.195030 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c704eb02-d156-4695-a55d-7dc96654e4cd-etc-kubernetes\") pod \"tuned-hmbpc\" (UID: \"c704eb02-d156-4695-a55d-7dc96654e4cd\") " pod="openshift-cluster-node-tuning-operator/tuned-hmbpc" Nov 03 13:08:41.195336 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.195060 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/74dc4aee-10b1-463c-a9c3-522b7a6c10b7-multus-socket-dir-parent\") pod \"multus-dsjsv\" (UID: \"74dc4aee-10b1-463c-a9c3-522b7a6c10b7\") " pod="openshift-multus/multus-dsjsv" Nov 03 13:08:41.195336 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.195077 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/74dc4aee-10b1-463c-a9c3-522b7a6c10b7-cni-binary-copy\") pod \"multus-dsjsv\" (UID: \"74dc4aee-10b1-463c-a9c3-522b7a6c10b7\") " pod="openshift-multus/multus-dsjsv" Nov 03 13:08:41.195336 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.195096 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzlml\" (UniqueName: \"kubernetes.io/projected/898bd0b5-0d3e-4577-b101-2cc7d75283cb-kube-api-access-bzlml\") pod \"router-default-7f579888d9-ntt6m\" (UID: \"898bd0b5-0d3e-4577-b101-2cc7d75283cb\") " pod="openshift-ingress/router-default-7f579888d9-ntt6m" Nov 03 13:08:41.195336 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.195162 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9fa6d32d-b30b-4b76-9462-5f3f09443d02-tmp\") pod \"insights-operator-7c4986d449-4t8lj\" (UID: \"9fa6d32d-b30b-4b76-9462-5f3f09443d02\") " pod="openshift-insights/insights-operator-7c4986d449-4t8lj" Nov 03 13:08:41.195336 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.195205 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/35e0fb40-48ce-4223-b7d4-c9290b6861cd-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-x2rm4\" (UID: \"35e0fb40-48ce-4223-b7d4-c9290b6861cd\") " pod="openshift-multus/multus-additional-cni-plugins-x2rm4" Nov 03 13:08:41.195336 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.195252 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0a892eb3-ecc4-4312-9f4d-73eeba6b6eee-ca-trust-extracted\") pod \"image-registry-7dbb67bd57-74qfj\" (UID: \"0a892eb3-ecc4-4312-9f4d-73eeba6b6eee\") " pod="openshift-image-registry/image-registry-7dbb67bd57-74qfj" Nov 03 13:08:41.195336 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.195322 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbs5f\" (UniqueName: \"kubernetes.io/projected/21618783-0a4e-4b1f-8041-cabdf69f5bea-kube-api-access-vbs5f\") pod \"cluster-samples-operator-6ddb6c8b94-h46gl\" (UID: \"21618783-0a4e-4b1f-8041-cabdf69f5bea\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-h46gl" Nov 03 13:08:41.196035 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.195360 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0a892eb3-ecc4-4312-9f4d-73eeba6b6eee-image-registry-private-configuration\") pod \"image-registry-7dbb67bd57-74qfj\" (UID: \"0a892eb3-ecc4-4312-9f4d-73eeba6b6eee\") " pod="openshift-image-registry/image-registry-7dbb67bd57-74qfj" Nov 03 13:08:41.196035 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.195398 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0a892eb3-ecc4-4312-9f4d-73eeba6b6eee-registry-tls\") pod \"image-registry-7dbb67bd57-74qfj\" (UID: \"0a892eb3-ecc4-4312-9f4d-73eeba6b6eee\") " pod="openshift-image-registry/image-registry-7dbb67bd57-74qfj" Nov 03 13:08:41.196035 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.195432 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0a892eb3-ecc4-4312-9f4d-73eeba6b6eee-trusted-ca\") pod \"image-registry-7dbb67bd57-74qfj\" (UID: \"0a892eb3-ecc4-4312-9f4d-73eeba6b6eee\") " pod="openshift-image-registry/image-registry-7dbb67bd57-74qfj" Nov 03 13:08:41.196035 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.195470 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c704eb02-d156-4695-a55d-7dc96654e4cd-sys\") pod \"tuned-hmbpc\" (UID: \"c704eb02-d156-4695-a55d-7dc96654e4cd\") " pod="openshift-cluster-node-tuning-operator/tuned-hmbpc" Nov 03 13:08:41.196035 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.195503 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/cc8ef30e-6d4f-4913-8dc8-b09e5aea2f82-image-registry-private-configuration\") pod \"image-registry-555f8976d-mz5ct\" (UID: \"cc8ef30e-6d4f-4913-8dc8-b09e5aea2f82\") " pod="openshift-image-registry/image-registry-555f8976d-mz5ct" Nov 03 13:08:41.196035 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.195660 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0a892eb3-ecc4-4312-9f4d-73eeba6b6eee-ca-trust-extracted\") pod \"image-registry-7dbb67bd57-74qfj\" (UID: \"0a892eb3-ecc4-4312-9f4d-73eeba6b6eee\") " pod="openshift-image-registry/image-registry-7dbb67bd57-74qfj" Nov 03 13:08:41.196035 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.195739 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/3fdeb9f3-6c67-4390-8073-17d4d5f802b9-config-volume\") pod \"dns-default-vgnv8\" (UID: \"3fdeb9f3-6c67-4390-8073-17d4d5f802b9\") " pod="openshift-dns/dns-default-vgnv8" Nov 03 13:08:41.196035 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.195883 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/35e0fb40-48ce-4223-b7d4-c9290b6861cd-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-x2rm4\" (UID: \"35e0fb40-48ce-4223-b7d4-c9290b6861cd\") " pod="openshift-multus/multus-additional-cni-plugins-x2rm4" Nov 03 13:08:41.196453 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.196047 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/cc8ef30e-6d4f-4913-8dc8-b09e5aea2f82-trusted-ca\") pod \"image-registry-555f8976d-mz5ct\" (UID: \"cc8ef30e-6d4f-4913-8dc8-b09e5aea2f82\") " pod="openshift-image-registry/image-registry-555f8976d-mz5ct" Nov 03 13:08:41.196453 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.196126 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d926d945-39f4-48f3-b3a4-0b302b15b69b-host-cni-netd\") pod \"ovnkube-node-fg6vh\" (UID: \"d926d945-39f4-48f3-b3a4-0b302b15b69b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fg6vh" Nov 03 13:08:41.196453 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.196163 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/21618783-0a4e-4b1f-8041-cabdf69f5bea-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-h46gl\" (UID: \"21618783-0a4e-4b1f-8041-cabdf69f5bea\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-h46gl" Nov 03 13:08:41.196453 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.196187 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0a892eb3-ecc4-4312-9f4d-73eeba6b6eee-trusted-ca\") pod \"image-registry-7dbb67bd57-74qfj\" (UID: \"0a892eb3-ecc4-4312-9f4d-73eeba6b6eee\") " pod="openshift-image-registry/image-registry-7dbb67bd57-74qfj" Nov 03 13:08:41.196453 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.196199 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/246971bc-ed35-4fc7-b5f8-941327b90a94-konnectivity-ca\") pod \"konnectivity-agent-bdg96\" (UID: \"246971bc-ed35-4fc7-b5f8-941327b90a94\") " pod="kube-system/konnectivity-agent-bdg96" Nov 03 13:08:41.196453 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.196247 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/74dc4aee-10b1-463c-a9c3-522b7a6c10b7-host-var-lib-kubelet\") pod \"multus-dsjsv\" (UID: \"74dc4aee-10b1-463c-a9c3-522b7a6c10b7\") " pod="openshift-multus/multus-dsjsv" Nov 03 13:08:41.196453 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.196351 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/35e0fb40-48ce-4223-b7d4-c9290b6861cd-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-x2rm4\" (UID: \"35e0fb40-48ce-4223-b7d4-c9290b6861cd\") " pod="openshift-multus/multus-additional-cni-plugins-x2rm4" Nov 03 13:08:41.196453 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.196435 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/21618783-0a4e-4b1f-8041-cabdf69f5bea-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-h46gl\" (UID: \"21618783-0a4e-4b1f-8041-cabdf69f5bea\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-h46gl" Nov 03 13:08:41.196835 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.196470 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d926d945-39f4-48f3-b3a4-0b302b15b69b-ovnkube-config\") pod \"ovnkube-node-fg6vh\" (UID: \"d926d945-39f4-48f3-b3a4-0b302b15b69b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fg6vh" Nov 03 13:08:41.196835 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.196509 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c704eb02-d156-4695-a55d-7dc96654e4cd-host\") pod \"tuned-hmbpc\" (UID: \"c704eb02-d156-4695-a55d-7dc96654e4cd\") " pod="openshift-cluster-node-tuning-operator/tuned-hmbpc" Nov 03 13:08:41.196835 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.196543 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c704eb02-d156-4695-a55d-7dc96654e4cd-etc-tuned\") pod \"tuned-hmbpc\" (UID: \"c704eb02-d156-4695-a55d-7dc96654e4cd\") " pod="openshift-cluster-node-tuning-operator/tuned-hmbpc" Nov 03 13:08:41.196835 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.196581 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/246971bc-ed35-4fc7-b5f8-941327b90a94-agent-certs\") pod \"konnectivity-agent-bdg96\" (UID: \"246971bc-ed35-4fc7-b5f8-941327b90a94\") " pod="kube-system/konnectivity-agent-bdg96" Nov 03 13:08:41.196835 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.196613 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/39e11989-e598-4cea-87d9-26b3cf78a66b-host\") pod \"node-ca-4tn4j\" (UID: \"39e11989-e598-4cea-87d9-26b3cf78a66b\") " pod="openshift-image-registry/node-ca-4tn4j" Nov 03 13:08:41.196835 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.196646 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/cc8ef30e-6d4f-4913-8dc8-b09e5aea2f82-ca-trust-extracted\") pod \"image-registry-555f8976d-mz5ct\" (UID: \"cc8ef30e-6d4f-4913-8dc8-b09e5aea2f82\") " pod="openshift-image-registry/image-registry-555f8976d-mz5ct" Nov 03 13:08:41.196835 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.196673 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/35e0fb40-48ce-4223-b7d4-c9290b6861cd-cni-binary-copy\") pod \"multus-additional-cni-plugins-x2rm4\" (UID: \"35e0fb40-48ce-4223-b7d4-c9290b6861cd\") " pod="openshift-multus/multus-additional-cni-plugins-x2rm4" Nov 03 13:08:41.196835 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.196744 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/cc8ef30e-6d4f-4913-8dc8-b09e5aea2f82-ca-trust-extracted\") pod \"image-registry-555f8976d-mz5ct\" (UID: \"cc8ef30e-6d4f-4913-8dc8-b09e5aea2f82\") " pod="openshift-image-registry/image-registry-555f8976d-mz5ct" Nov 03 13:08:41.196835 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.196745 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bddlp\" (UniqueName: \"kubernetes.io/projected/3fdeb9f3-6c67-4390-8073-17d4d5f802b9-kube-api-access-bddlp\") pod \"dns-default-vgnv8\" (UID: \"3fdeb9f3-6c67-4390-8073-17d4d5f802b9\") " pod="openshift-dns/dns-default-vgnv8" Nov 03 13:08:41.196835 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.196832 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0a892eb3-ecc4-4312-9f4d-73eeba6b6eee-image-registry-private-configuration\") pod \"image-registry-7dbb67bd57-74qfj\" (UID: \"0a892eb3-ecc4-4312-9f4d-73eeba6b6eee\") " pod="openshift-image-registry/image-registry-7dbb67bd57-74qfj" Nov 03 13:08:41.197304 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.196876 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9db7d195-220e-4644-8690-9e5ee37fc066-device-dir\") pod \"aws-ebs-csi-driver-node-c9dvm\" (UID: \"9db7d195-220e-4644-8690-9e5ee37fc066\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c9dvm" Nov 03 13:08:41.197304 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.196901 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d926d945-39f4-48f3-b3a4-0b302b15b69b-ovnkube-config\") pod \"ovnkube-node-fg6vh\" (UID: \"d926d945-39f4-48f3-b3a4-0b302b15b69b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fg6vh" Nov 03 13:08:41.197304 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.196906 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d926d945-39f4-48f3-b3a4-0b302b15b69b-log-socket\") pod \"ovnkube-node-fg6vh\" (UID: \"d926d945-39f4-48f3-b3a4-0b302b15b69b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fg6vh" Nov 03 13:08:41.197304 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.197038 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/246971bc-ed35-4fc7-b5f8-941327b90a94-konnectivity-ca\") pod \"konnectivity-agent-bdg96\" (UID: \"246971bc-ed35-4fc7-b5f8-941327b90a94\") " pod="kube-system/konnectivity-agent-bdg96" Nov 03 13:08:41.197304 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.197086 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/74dc4aee-10b1-463c-a9c3-522b7a6c10b7-multus-cni-dir\") pod \"multus-dsjsv\" (UID: \"74dc4aee-10b1-463c-a9c3-522b7a6c10b7\") " pod="openshift-multus/multus-dsjsv" Nov 03 13:08:41.197304 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.197091 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/246971bc-ed35-4fc7-b5f8-941327b90a94-agent-certs\") pod \"konnectivity-agent-bdg96\" (UID: \"246971bc-ed35-4fc7-b5f8-941327b90a94\") " pod="kube-system/konnectivity-agent-bdg96" Nov 03 13:08:41.197304 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.197122 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/5ea5cf61-ed1c-4f5e-b1a0-5c46c5c3ee18-hosts-file\") pod \"node-resolver-v8gcq\" (UID: \"5ea5cf61-ed1c-4f5e-b1a0-5c46c5c3ee18\") " pod="openshift-dns/node-resolver-v8gcq" Nov 03 13:08:41.197304 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.197153 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c704eb02-d156-4695-a55d-7dc96654e4cd-etc-tuned\") pod \"tuned-hmbpc\" (UID: \"c704eb02-d156-4695-a55d-7dc96654e4cd\") " pod="openshift-cluster-node-tuning-operator/tuned-hmbpc" Nov 03 13:08:41.197304 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.197270 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a3042b60-062f-4f9b-8406-f884c95a8729-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-tm4nb\" (UID: \"a3042b60-062f-4f9b-8406-f884c95a8729\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-tm4nb" Nov 03 13:08:41.197728 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.197318 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c704eb02-d156-4695-a55d-7dc96654e4cd-etc-systemd\") pod \"tuned-hmbpc\" (UID: \"c704eb02-d156-4695-a55d-7dc96654e4cd\") " pod="openshift-cluster-node-tuning-operator/tuned-hmbpc" Nov 03 13:08:41.197728 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.197349 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c704eb02-d156-4695-a55d-7dc96654e4cd-run\") pod \"tuned-hmbpc\" (UID: \"c704eb02-d156-4695-a55d-7dc96654e4cd\") " pod="openshift-cluster-node-tuning-operator/tuned-hmbpc" Nov 03 13:08:41.197728 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.197401 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9fa6d32d-b30b-4b76-9462-5f3f09443d02-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-4t8lj\" (UID: \"9fa6d32d-b30b-4b76-9462-5f3f09443d02\") " pod="openshift-insights/insights-operator-7c4986d449-4t8lj" Nov 03 13:08:41.197728 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.197318 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0a892eb3-ecc4-4312-9f4d-73eeba6b6eee-installation-pull-secrets\") pod \"image-registry-7dbb67bd57-74qfj\" (UID: \"0a892eb3-ecc4-4312-9f4d-73eeba6b6eee\") " pod="openshift-image-registry/image-registry-7dbb67bd57-74qfj" Nov 03 13:08:41.197728 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.197556 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9fhg\" (UniqueName: \"kubernetes.io/projected/5c17f5bf-5b7d-4b05-a387-21ac6b5186ca-kube-api-access-k9fhg\") pod \"network-check-source-6f78d59f99-jws95\" (UID: \"5c17f5bf-5b7d-4b05-a387-21ac6b5186ca\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-jws95" Nov 03 13:08:41.197728 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.197585 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d926d945-39f4-48f3-b3a4-0b302b15b69b-ovn-node-metrics-cert\") pod \"ovnkube-node-fg6vh\" (UID: \"d926d945-39f4-48f3-b3a4-0b302b15b69b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fg6vh" Nov 03 13:08:41.197728 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.197594 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/9fa6d32d-b30b-4b76-9462-5f3f09443d02-snapshots\") pod \"insights-operator-7c4986d449-4t8lj\" (UID: \"9fa6d32d-b30b-4b76-9462-5f3f09443d02\") " pod="openshift-insights/insights-operator-7c4986d449-4t8lj" Nov 03 13:08:41.198053 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.197860 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwrs7\" (UniqueName: \"kubernetes.io/projected/74dc4aee-10b1-463c-a9c3-522b7a6c10b7-kube-api-access-cwrs7\") pod \"multus-dsjsv\" (UID: \"74dc4aee-10b1-463c-a9c3-522b7a6c10b7\") " pod="openshift-multus/multus-dsjsv" Nov 03 13:08:41.198053 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.198025 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0a892eb3-ecc4-4312-9f4d-73eeba6b6eee-bound-sa-token\") pod \"image-registry-7dbb67bd57-74qfj\" (UID: \"0a892eb3-ecc4-4312-9f4d-73eeba6b6eee\") " pod="openshift-image-registry/image-registry-7dbb67bd57-74qfj" Nov 03 13:08:41.198445 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.198318 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9fa6d32d-b30b-4b76-9462-5f3f09443d02-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-4t8lj\" (UID: \"9fa6d32d-b30b-4b76-9462-5f3f09443d02\") " pod="openshift-insights/insights-operator-7c4986d449-4t8lj" Nov 03 13:08:41.198445 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.198433 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pt6qk\" (UniqueName: \"kubernetes.io/projected/cd91de08-084e-4667-b13f-2e07b3cff624-kube-api-access-pt6qk\") pod \"console-operator-7cd56b8dbc-6htht\" (UID: \"cd91de08-084e-4667-b13f-2e07b3cff624\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-6htht" Nov 03 13:08:41.198607 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.198583 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mss42\" (UniqueName: \"kubernetes.io/projected/35e0fb40-48ce-4223-b7d4-c9290b6861cd-kube-api-access-mss42\") pod \"multus-additional-cni-plugins-x2rm4\" (UID: \"35e0fb40-48ce-4223-b7d4-c9290b6861cd\") " pod="openshift-multus/multus-additional-cni-plugins-x2rm4" Nov 03 13:08:41.198659 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.198635 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9fa6d32d-b30b-4b76-9462-5f3f09443d02-service-ca-bundle\") pod \"insights-operator-7c4986d449-4t8lj\" (UID: \"9fa6d32d-b30b-4b76-9462-5f3f09443d02\") " pod="openshift-insights/insights-operator-7c4986d449-4t8lj" Nov 03 13:08:41.198716 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.198664 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/35e0fb40-48ce-4223-b7d4-c9290b6861cd-tuning-conf-dir\") pod \"multus-additional-cni-plugins-x2rm4\" (UID: \"35e0fb40-48ce-4223-b7d4-c9290b6861cd\") " pod="openshift-multus/multus-additional-cni-plugins-x2rm4" Nov 03 13:08:41.198716 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.198681 5024 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 03 13:08:41.198812 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.198695 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/74dc4aee-10b1-463c-a9c3-522b7a6c10b7-host-run-multus-certs\") pod \"multus-dsjsv\" (UID: \"74dc4aee-10b1-463c-a9c3-522b7a6c10b7\") " pod="openshift-multus/multus-dsjsv" Nov 03 13:08:41.198812 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.198791 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9fa6d32d-b30b-4b76-9462-5f3f09443d02-serving-cert\") pod \"insights-operator-7c4986d449-4t8lj\" (UID: \"9fa6d32d-b30b-4b76-9462-5f3f09443d02\") " pod="openshift-insights/insights-operator-7c4986d449-4t8lj" Nov 03 13:08:41.198912 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.198835 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cc8ef30e-6d4f-4913-8dc8-b09e5aea2f82-bound-sa-token\") pod \"image-registry-555f8976d-mz5ct\" (UID: \"cc8ef30e-6d4f-4913-8dc8-b09e5aea2f82\") " pod="openshift-image-registry/image-registry-555f8976d-mz5ct" Nov 03 13:08:41.198912 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.198846 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e0300f55-f002-4faa-bf6a-c04bea0a043e-metrics-certs\") pod \"network-metrics-daemon-f7hmh\" (UID: \"e0300f55-f002-4faa-bf6a-c04bea0a043e\") " pod="openshift-multus/network-metrics-daemon-f7hmh" Nov 03 13:08:41.199443 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.199124 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8x9b\" (UniqueName: \"kubernetes.io/projected/9fa6d32d-b30b-4b76-9462-5f3f09443d02-kube-api-access-n8x9b\") pod \"insights-operator-7c4986d449-4t8lj\" (UID: \"9fa6d32d-b30b-4b76-9462-5f3f09443d02\") " pod="openshift-insights/insights-operator-7c4986d449-4t8lj" Nov 03 13:08:41.199443 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.199340 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/fd7e8b58-de31-406a-8cdb-9ccfee140d57-iptables-alerter-script\") pod \"iptables-alerter-g4xgp\" (UID: \"fd7e8b58-de31-406a-8cdb-9ccfee140d57\") " pod="openshift-network-operator/iptables-alerter-g4xgp" Nov 03 13:08:41.199574 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.199452 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hbfgb\" (UniqueName: \"kubernetes.io/projected/1bfc1afc-5b9f-4447-8d40-fabedbee9c07-kube-api-access-hbfgb\") pod \"network-check-target-8rdgq\" (UID: \"1bfc1afc-5b9f-4447-8d40-fabedbee9c07\") " pod="openshift-network-diagnostics/network-check-target-8rdgq" Nov 03 13:08:41.199574 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.199524 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9fa6d32d-b30b-4b76-9462-5f3f09443d02-serving-cert\") pod \"insights-operator-7c4986d449-4t8lj\" (UID: \"9fa6d32d-b30b-4b76-9462-5f3f09443d02\") " pod="openshift-insights/insights-operator-7c4986d449-4t8lj" Nov 03 13:08:41.200069 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.200046 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xmbd7\" (UniqueName: \"kubernetes.io/projected/0a892eb3-ecc4-4312-9f4d-73eeba6b6eee-kube-api-access-xmbd7\") pod \"image-registry-7dbb67bd57-74qfj\" (UID: \"0a892eb3-ecc4-4312-9f4d-73eeba6b6eee\") " pod="openshift-image-registry/image-registry-7dbb67bd57-74qfj" Nov 03 13:08:41.200137 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.200113 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9fa6d32d-b30b-4b76-9462-5f3f09443d02-service-ca-bundle\") pod \"insights-operator-7c4986d449-4t8lj\" (UID: \"9fa6d32d-b30b-4b76-9462-5f3f09443d02\") " pod="openshift-insights/insights-operator-7c4986d449-4t8lj" Nov 03 13:08:41.201122 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.201101 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a3042b60-062f-4f9b-8406-f884c95a8729-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-tm4nb\" (UID: \"a3042b60-062f-4f9b-8406-f884c95a8729\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-tm4nb" Nov 03 13:08:41.203936 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.203917 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bcz25\" (UniqueName: \"kubernetes.io/projected/e0300f55-f002-4faa-bf6a-c04bea0a043e-kube-api-access-bcz25\") pod \"network-metrics-daemon-f7hmh\" (UID: \"e0300f55-f002-4faa-bf6a-c04bea0a043e\") " pod="openshift-multus/network-metrics-daemon-f7hmh" Nov 03 13:08:41.204032 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.204005 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-snpf4\" (UniqueName: \"kubernetes.io/projected/fd7e8b58-de31-406a-8cdb-9ccfee140d57-kube-api-access-snpf4\") pod \"iptables-alerter-g4xgp\" (UID: \"fd7e8b58-de31-406a-8cdb-9ccfee140d57\") " pod="openshift-network-operator/iptables-alerter-g4xgp" Nov 03 13:08:41.204080 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.204046 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jknsv\" (UniqueName: \"kubernetes.io/projected/9f20ddda-41fb-4aba-b257-7ebc59dbe678-kube-api-access-jknsv\") pod \"service-ca-operator-544d749c8c-cqvdv\" (UID: \"9f20ddda-41fb-4aba-b257-7ebc59dbe678\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-cqvdv" Nov 03 13:08:41.204520 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.204501 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ftcqc\" (UniqueName: \"kubernetes.io/projected/9db7d195-220e-4644-8690-9e5ee37fc066-kube-api-access-ftcqc\") pod \"aws-ebs-csi-driver-node-c9dvm\" (UID: \"9db7d195-220e-4644-8690-9e5ee37fc066\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c9dvm" Nov 03 13:08:41.204925 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.204899 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bzlml\" (UniqueName: \"kubernetes.io/projected/898bd0b5-0d3e-4577-b101-2cc7d75283cb-kube-api-access-bzlml\") pod \"router-default-7f579888d9-ntt6m\" (UID: \"898bd0b5-0d3e-4577-b101-2cc7d75283cb\") " pod="openshift-ingress/router-default-7f579888d9-ntt6m" Nov 03 13:08:41.205008 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.204963 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jssfx\" (UniqueName: \"kubernetes.io/projected/fff35d1b-e7b2-4dd3-9824-8b64fad02506-kube-api-access-jssfx\") pod \"insights-runtime-extractor-d7mwz\" (UID: \"fff35d1b-e7b2-4dd3-9824-8b64fad02506\") " pod="openshift-insights/insights-runtime-extractor-d7mwz" Nov 03 13:08:41.205180 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.205159 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zpq8b\" (UniqueName: \"kubernetes.io/projected/39e11989-e598-4cea-87d9-26b3cf78a66b-kube-api-access-zpq8b\") pod \"node-ca-4tn4j\" (UID: \"39e11989-e598-4cea-87d9-26b3cf78a66b\") " pod="openshift-image-registry/node-ca-4tn4j" Nov 03 13:08:41.205384 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.205362 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2sxmr\" (UniqueName: \"kubernetes.io/projected/50d68cb7-b766-4674-b295-8f3229880994-kube-api-access-2sxmr\") pod \"ingress-canary-d8f6l\" (UID: \"50d68cb7-b766-4674-b295-8f3229880994\") " pod="openshift-ingress-canary/ingress-canary-d8f6l" Nov 03 13:08:41.205811 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.205789 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6b28l\" (UniqueName: \"kubernetes.io/projected/d926d945-39f4-48f3-b3a4-0b302b15b69b-kube-api-access-6b28l\") pod \"ovnkube-node-fg6vh\" (UID: \"d926d945-39f4-48f3-b3a4-0b302b15b69b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fg6vh" Nov 03 13:08:41.205914 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.205894 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mtd8p\" (UniqueName: \"kubernetes.io/projected/bd523616-71b2-4b6c-b82c-f1efb585c271-kube-api-access-mtd8p\") pod \"migrator-66df5d496b-gkblf\" (UID: \"bd523616-71b2-4b6c-b82c-f1efb585c271\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-gkblf" Nov 03 13:08:41.206745 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.206083 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4j5g\" (UniqueName: \"kubernetes.io/projected/c704eb02-d156-4695-a55d-7dc96654e4cd-kube-api-access-w4j5g\") pod \"tuned-hmbpc\" (UID: \"c704eb02-d156-4695-a55d-7dc96654e4cd\") " pod="openshift-cluster-node-tuning-operator/tuned-hmbpc" Nov 03 13:08:41.206745 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.206527 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbs5f\" (UniqueName: \"kubernetes.io/projected/21618783-0a4e-4b1f-8041-cabdf69f5bea-kube-api-access-vbs5f\") pod \"cluster-samples-operator-6ddb6c8b94-h46gl\" (UID: \"21618783-0a4e-4b1f-8041-cabdf69f5bea\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-h46gl" Nov 03 13:08:41.206745 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.206710 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjhjt\" (UniqueName: \"kubernetes.io/projected/5ea5cf61-ed1c-4f5e-b1a0-5c46c5c3ee18-kube-api-access-vjhjt\") pod \"node-resolver-v8gcq\" (UID: \"5ea5cf61-ed1c-4f5e-b1a0-5c46c5c3ee18\") " pod="openshift-dns/node-resolver-v8gcq" Nov 03 13:08:41.206967 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.206951 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h82mn\" (UniqueName: \"kubernetes.io/projected/cc8ef30e-6d4f-4913-8dc8-b09e5aea2f82-kube-api-access-h82mn\") pod \"image-registry-555f8976d-mz5ct\" (UID: \"cc8ef30e-6d4f-4913-8dc8-b09e5aea2f82\") " pod="openshift-image-registry/image-registry-555f8976d-mz5ct" Nov 03 13:08:41.300022 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.299983 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d926d945-39f4-48f3-b3a4-0b302b15b69b-host-kubelet\") pod \"ovnkube-node-fg6vh\" (UID: \"d926d945-39f4-48f3-b3a4-0b302b15b69b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fg6vh" Nov 03 13:08:41.300208 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.300024 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/21618783-0a4e-4b1f-8041-cabdf69f5bea-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-h46gl\" (UID: \"21618783-0a4e-4b1f-8041-cabdf69f5bea\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-h46gl" Nov 03 13:08:41.300208 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.300048 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d926d945-39f4-48f3-b3a4-0b302b15b69b-host-kubelet\") pod \"ovnkube-node-fg6vh\" (UID: \"d926d945-39f4-48f3-b3a4-0b302b15b69b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fg6vh" Nov 03 13:08:41.300208 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.300061 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d926d945-39f4-48f3-b3a4-0b302b15b69b-etc-openvswitch\") pod \"ovnkube-node-fg6vh\" (UID: \"d926d945-39f4-48f3-b3a4-0b302b15b69b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fg6vh" Nov 03 13:08:41.300208 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.300094 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/fa297a30-6da7-4a92-98fb-201323490bb0-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-rz4n9\" (UID: \"fa297a30-6da7-4a92-98fb-201323490bb0\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-rz4n9" Nov 03 13:08:41.300208 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.300125 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fff35d1b-e7b2-4dd3-9824-8b64fad02506-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-d7mwz\" (UID: \"fff35d1b-e7b2-4dd3-9824-8b64fad02506\") " pod="openshift-insights/insights-runtime-extractor-d7mwz" Nov 03 13:08:41.300208 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.300096 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d926d945-39f4-48f3-b3a4-0b302b15b69b-etc-openvswitch\") pod \"ovnkube-node-fg6vh\" (UID: \"d926d945-39f4-48f3-b3a4-0b302b15b69b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fg6vh" Nov 03 13:08:41.300208 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.300161 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9db7d195-220e-4644-8690-9e5ee37fc066-registration-dir\") pod \"aws-ebs-csi-driver-node-c9dvm\" (UID: \"9db7d195-220e-4644-8690-9e5ee37fc066\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c9dvm" Nov 03 13:08:41.300581 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.300241 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9db7d195-220e-4644-8690-9e5ee37fc066-registration-dir\") pod \"aws-ebs-csi-driver-node-c9dvm\" (UID: \"9db7d195-220e-4644-8690-9e5ee37fc066\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c9dvm" Nov 03 13:08:41.300581 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.300204 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d926d945-39f4-48f3-b3a4-0b302b15b69b-run-openvswitch\") pod \"ovnkube-node-fg6vh\" (UID: \"d926d945-39f4-48f3-b3a4-0b302b15b69b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fg6vh" Nov 03 13:08:41.300581 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.300319 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/74dc4aee-10b1-463c-a9c3-522b7a6c10b7-host-run-netns\") pod \"multus-dsjsv\" (UID: \"74dc4aee-10b1-463c-a9c3-522b7a6c10b7\") " pod="openshift-multus/multus-dsjsv" Nov 03 13:08:41.300581 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.300335 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d926d945-39f4-48f3-b3a4-0b302b15b69b-run-openvswitch\") pod \"ovnkube-node-fg6vh\" (UID: \"d926d945-39f4-48f3-b3a4-0b302b15b69b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fg6vh" Nov 03 13:08:41.300581 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.300361 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/74dc4aee-10b1-463c-a9c3-522b7a6c10b7-host-run-netns\") pod \"multus-dsjsv\" (UID: \"74dc4aee-10b1-463c-a9c3-522b7a6c10b7\") " pod="openshift-multus/multus-dsjsv" Nov 03 13:08:41.300581 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.300346 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d926d945-39f4-48f3-b3a4-0b302b15b69b-var-lib-openvswitch\") pod \"ovnkube-node-fg6vh\" (UID: \"d926d945-39f4-48f3-b3a4-0b302b15b69b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fg6vh" Nov 03 13:08:41.300581 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.300379 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d926d945-39f4-48f3-b3a4-0b302b15b69b-var-lib-openvswitch\") pod \"ovnkube-node-fg6vh\" (UID: \"d926d945-39f4-48f3-b3a4-0b302b15b69b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fg6vh" Nov 03 13:08:41.300581 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.300406 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/74dc4aee-10b1-463c-a9c3-522b7a6c10b7-cnibin\") pod \"multus-dsjsv\" (UID: \"74dc4aee-10b1-463c-a9c3-522b7a6c10b7\") " pod="openshift-multus/multus-dsjsv" Nov 03 13:08:41.300581 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.300425 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/35e0fb40-48ce-4223-b7d4-c9290b6861cd-system-cni-dir\") pod \"multus-additional-cni-plugins-x2rm4\" (UID: \"35e0fb40-48ce-4223-b7d4-c9290b6861cd\") " pod="openshift-multus/multus-additional-cni-plugins-x2rm4" Nov 03 13:08:41.300581 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.300444 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9db7d195-220e-4644-8690-9e5ee37fc066-socket-dir\") pod \"aws-ebs-csi-driver-node-c9dvm\" (UID: \"9db7d195-220e-4644-8690-9e5ee37fc066\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c9dvm" Nov 03 13:08:41.300581 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.300467 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9db7d195-220e-4644-8690-9e5ee37fc066-etc-selinux\") pod \"aws-ebs-csi-driver-node-c9dvm\" (UID: \"9db7d195-220e-4644-8690-9e5ee37fc066\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c9dvm" Nov 03 13:08:41.300581 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.300491 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d926d945-39f4-48f3-b3a4-0b302b15b69b-run-ovn\") pod \"ovnkube-node-fg6vh\" (UID: \"d926d945-39f4-48f3-b3a4-0b302b15b69b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fg6vh" Nov 03 13:08:41.300581 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.300492 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/74dc4aee-10b1-463c-a9c3-522b7a6c10b7-cnibin\") pod \"multus-dsjsv\" (UID: \"74dc4aee-10b1-463c-a9c3-522b7a6c10b7\") " pod="openshift-multus/multus-dsjsv" Nov 03 13:08:41.300581 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.300518 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c704eb02-d156-4695-a55d-7dc96654e4cd-var-lib-kubelet\") pod \"tuned-hmbpc\" (UID: \"c704eb02-d156-4695-a55d-7dc96654e4cd\") " pod="openshift-cluster-node-tuning-operator/tuned-hmbpc" Nov 03 13:08:41.300581 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.300489 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/35e0fb40-48ce-4223-b7d4-c9290b6861cd-system-cni-dir\") pod \"multus-additional-cni-plugins-x2rm4\" (UID: \"35e0fb40-48ce-4223-b7d4-c9290b6861cd\") " pod="openshift-multus/multus-additional-cni-plugins-x2rm4" Nov 03 13:08:41.300581 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.300527 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9db7d195-220e-4644-8690-9e5ee37fc066-socket-dir\") pod \"aws-ebs-csi-driver-node-c9dvm\" (UID: \"9db7d195-220e-4644-8690-9e5ee37fc066\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c9dvm" Nov 03 13:08:41.300581 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.300546 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d926d945-39f4-48f3-b3a4-0b302b15b69b-host-slash\") pod \"ovnkube-node-fg6vh\" (UID: \"d926d945-39f4-48f3-b3a4-0b302b15b69b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fg6vh" Nov 03 13:08:41.300581 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.300552 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9db7d195-220e-4644-8690-9e5ee37fc066-etc-selinux\") pod \"aws-ebs-csi-driver-node-c9dvm\" (UID: \"9db7d195-220e-4644-8690-9e5ee37fc066\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c9dvm" Nov 03 13:08:41.300581 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.300566 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d926d945-39f4-48f3-b3a4-0b302b15b69b-run-ovn\") pod \"ovnkube-node-fg6vh\" (UID: \"d926d945-39f4-48f3-b3a4-0b302b15b69b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fg6vh" Nov 03 13:08:41.300581 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.300576 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/74dc4aee-10b1-463c-a9c3-522b7a6c10b7-system-cni-dir\") pod \"multus-dsjsv\" (UID: \"74dc4aee-10b1-463c-a9c3-522b7a6c10b7\") " pod="openshift-multus/multus-dsjsv" Nov 03 13:08:41.300581 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.300585 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c704eb02-d156-4695-a55d-7dc96654e4cd-var-lib-kubelet\") pod \"tuned-hmbpc\" (UID: \"c704eb02-d156-4695-a55d-7dc96654e4cd\") " pod="openshift-cluster-node-tuning-operator/tuned-hmbpc" Nov 03 13:08:41.300581 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.300591 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d926d945-39f4-48f3-b3a4-0b302b15b69b-host-slash\") pod \"ovnkube-node-fg6vh\" (UID: \"d926d945-39f4-48f3-b3a4-0b302b15b69b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fg6vh" Nov 03 13:08:41.300581 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.300605 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9db7d195-220e-4644-8690-9e5ee37fc066-sys-fs\") pod \"aws-ebs-csi-driver-node-c9dvm\" (UID: \"9db7d195-220e-4644-8690-9e5ee37fc066\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c9dvm" Nov 03 13:08:41.301911 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.300632 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/74dc4aee-10b1-463c-a9c3-522b7a6c10b7-system-cni-dir\") pod \"multus-dsjsv\" (UID: \"74dc4aee-10b1-463c-a9c3-522b7a6c10b7\") " pod="openshift-multus/multus-dsjsv" Nov 03 13:08:41.301911 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.300632 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/35e0fb40-48ce-4223-b7d4-c9290b6861cd-cnibin\") pod \"multus-additional-cni-plugins-x2rm4\" (UID: \"35e0fb40-48ce-4223-b7d4-c9290b6861cd\") " pod="openshift-multus/multus-additional-cni-plugins-x2rm4" Nov 03 13:08:41.301911 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.300664 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/35e0fb40-48ce-4223-b7d4-c9290b6861cd-cnibin\") pod \"multus-additional-cni-plugins-x2rm4\" (UID: \"35e0fb40-48ce-4223-b7d4-c9290b6861cd\") " pod="openshift-multus/multus-additional-cni-plugins-x2rm4" Nov 03 13:08:41.301911 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.300668 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/fff35d1b-e7b2-4dd3-9824-8b64fad02506-crio-socket\") pod \"insights-runtime-extractor-d7mwz\" (UID: \"fff35d1b-e7b2-4dd3-9824-8b64fad02506\") " pod="openshift-insights/insights-runtime-extractor-d7mwz" Nov 03 13:08:41.301911 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.300706 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/7e280277-f29f-4a2b-8b85-2491529e5ef2-dbus\") pod \"global-pull-secret-syncer-5jz7t\" (UID: \"7e280277-f29f-4a2b-8b85-2491529e5ef2\") " pod="kube-system/global-pull-secret-syncer-5jz7t" Nov 03 13:08:41.301911 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.300714 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9db7d195-220e-4644-8690-9e5ee37fc066-sys-fs\") pod \"aws-ebs-csi-driver-node-c9dvm\" (UID: \"9db7d195-220e-4644-8690-9e5ee37fc066\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c9dvm" Nov 03 13:08:41.301911 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.300726 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/74dc4aee-10b1-463c-a9c3-522b7a6c10b7-hostroot\") pod \"multus-dsjsv\" (UID: \"74dc4aee-10b1-463c-a9c3-522b7a6c10b7\") " pod="openshift-multus/multus-dsjsv" Nov 03 13:08:41.301911 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.300761 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/fff35d1b-e7b2-4dd3-9824-8b64fad02506-crio-socket\") pod \"insights-runtime-extractor-d7mwz\" (UID: \"fff35d1b-e7b2-4dd3-9824-8b64fad02506\") " pod="openshift-insights/insights-runtime-extractor-d7mwz" Nov 03 13:08:41.301911 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.300773 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d926d945-39f4-48f3-b3a4-0b302b15b69b-host-cni-bin\") pod \"ovnkube-node-fg6vh\" (UID: \"d926d945-39f4-48f3-b3a4-0b302b15b69b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fg6vh" Nov 03 13:08:41.301911 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.300814 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/7e280277-f29f-4a2b-8b85-2491529e5ef2-dbus\") pod \"global-pull-secret-syncer-5jz7t\" (UID: \"7e280277-f29f-4a2b-8b85-2491529e5ef2\") " pod="kube-system/global-pull-secret-syncer-5jz7t" Nov 03 13:08:41.301911 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.300831 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/74dc4aee-10b1-463c-a9c3-522b7a6c10b7-hostroot\") pod \"multus-dsjsv\" (UID: \"74dc4aee-10b1-463c-a9c3-522b7a6c10b7\") " pod="openshift-multus/multus-dsjsv" Nov 03 13:08:41.301911 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.300864 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d926d945-39f4-48f3-b3a4-0b302b15b69b-host-cni-bin\") pod \"ovnkube-node-fg6vh\" (UID: \"d926d945-39f4-48f3-b3a4-0b302b15b69b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fg6vh" Nov 03 13:08:41.301911 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.300922 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c704eb02-d156-4695-a55d-7dc96654e4cd-etc-kubernetes\") pod \"tuned-hmbpc\" (UID: \"c704eb02-d156-4695-a55d-7dc96654e4cd\") " pod="openshift-cluster-node-tuning-operator/tuned-hmbpc" Nov 03 13:08:41.301911 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.300951 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/74dc4aee-10b1-463c-a9c3-522b7a6c10b7-multus-socket-dir-parent\") pod \"multus-dsjsv\" (UID: \"74dc4aee-10b1-463c-a9c3-522b7a6c10b7\") " pod="openshift-multus/multus-dsjsv" Nov 03 13:08:41.301911 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.301007 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/74dc4aee-10b1-463c-a9c3-522b7a6c10b7-multus-socket-dir-parent\") pod \"multus-dsjsv\" (UID: \"74dc4aee-10b1-463c-a9c3-522b7a6c10b7\") " pod="openshift-multus/multus-dsjsv" Nov 03 13:08:41.301911 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.301009 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c704eb02-d156-4695-a55d-7dc96654e4cd-etc-kubernetes\") pod \"tuned-hmbpc\" (UID: \"c704eb02-d156-4695-a55d-7dc96654e4cd\") " pod="openshift-cluster-node-tuning-operator/tuned-hmbpc" Nov 03 13:08:41.301911 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.301022 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0a892eb3-ecc4-4312-9f4d-73eeba6b6eee-registry-tls\") pod \"image-registry-7dbb67bd57-74qfj\" (UID: \"0a892eb3-ecc4-4312-9f4d-73eeba6b6eee\") " pod="openshift-image-registry/image-registry-7dbb67bd57-74qfj" Nov 03 13:08:41.301911 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.301052 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c704eb02-d156-4695-a55d-7dc96654e4cd-sys\") pod \"tuned-hmbpc\" (UID: \"c704eb02-d156-4695-a55d-7dc96654e4cd\") " pod="openshift-cluster-node-tuning-operator/tuned-hmbpc" Nov 03 13:08:41.301911 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.301075 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d926d945-39f4-48f3-b3a4-0b302b15b69b-host-cni-netd\") pod \"ovnkube-node-fg6vh\" (UID: \"d926d945-39f4-48f3-b3a4-0b302b15b69b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fg6vh" Nov 03 13:08:41.301911 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.301101 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/74dc4aee-10b1-463c-a9c3-522b7a6c10b7-host-var-lib-kubelet\") pod \"multus-dsjsv\" (UID: \"74dc4aee-10b1-463c-a9c3-522b7a6c10b7\") " pod="openshift-multus/multus-dsjsv" Nov 03 13:08:41.301911 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:08:41.301100 5024 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 13:08:41.301911 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.301120 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c704eb02-d156-4695-a55d-7dc96654e4cd-host\") pod \"tuned-hmbpc\" (UID: \"c704eb02-d156-4695-a55d-7dc96654e4cd\") " pod="openshift-cluster-node-tuning-operator/tuned-hmbpc" Nov 03 13:08:41.301911 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:08:41.301128 5024 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7dbb67bd57-74qfj: secret "image-registry-tls" not found Nov 03 13:08:41.301911 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.301140 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c704eb02-d156-4695-a55d-7dc96654e4cd-sys\") pod \"tuned-hmbpc\" (UID: \"c704eb02-d156-4695-a55d-7dc96654e4cd\") " pod="openshift-cluster-node-tuning-operator/tuned-hmbpc" Nov 03 13:08:41.301911 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.301143 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/39e11989-e598-4cea-87d9-26b3cf78a66b-host\") pod \"node-ca-4tn4j\" (UID: \"39e11989-e598-4cea-87d9-26b3cf78a66b\") " pod="openshift-image-registry/node-ca-4tn4j" Nov 03 13:08:41.301911 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.301175 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9db7d195-220e-4644-8690-9e5ee37fc066-device-dir\") pod \"aws-ebs-csi-driver-node-c9dvm\" (UID: \"9db7d195-220e-4644-8690-9e5ee37fc066\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c9dvm" Nov 03 13:08:41.301911 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.301177 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/74dc4aee-10b1-463c-a9c3-522b7a6c10b7-host-var-lib-kubelet\") pod \"multus-dsjsv\" (UID: \"74dc4aee-10b1-463c-a9c3-522b7a6c10b7\") " pod="openshift-multus/multus-dsjsv" Nov 03 13:08:41.301911 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.301182 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c704eb02-d156-4695-a55d-7dc96654e4cd-host\") pod \"tuned-hmbpc\" (UID: \"c704eb02-d156-4695-a55d-7dc96654e4cd\") " pod="openshift-cluster-node-tuning-operator/tuned-hmbpc" Nov 03 13:08:41.301911 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:08:41.301201 5024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0a892eb3-ecc4-4312-9f4d-73eeba6b6eee-registry-tls podName:0a892eb3-ecc4-4312-9f4d-73eeba6b6eee nodeName:}" failed. No retries permitted until 2026-11-03 13:08:41.80117989 +0000 UTC m=+1.809371165 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/0a892eb3-ecc4-4312-9f4d-73eeba6b6eee-registry-tls") pod "image-registry-7dbb67bd57-74qfj" (UID: "0a892eb3-ecc4-4312-9f4d-73eeba6b6eee") : secret "image-registry-tls" not found Nov 03 13:08:41.301911 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.301213 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d926d945-39f4-48f3-b3a4-0b302b15b69b-host-cni-netd\") pod \"ovnkube-node-fg6vh\" (UID: \"d926d945-39f4-48f3-b3a4-0b302b15b69b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fg6vh" Nov 03 13:08:41.301911 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.301229 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9db7d195-220e-4644-8690-9e5ee37fc066-device-dir\") pod \"aws-ebs-csi-driver-node-c9dvm\" (UID: \"9db7d195-220e-4644-8690-9e5ee37fc066\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c9dvm" Nov 03 13:08:41.301911 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.301242 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d926d945-39f4-48f3-b3a4-0b302b15b69b-log-socket\") pod \"ovnkube-node-fg6vh\" (UID: \"d926d945-39f4-48f3-b3a4-0b302b15b69b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fg6vh" Nov 03 13:08:41.301911 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.301265 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d926d945-39f4-48f3-b3a4-0b302b15b69b-log-socket\") pod \"ovnkube-node-fg6vh\" (UID: \"d926d945-39f4-48f3-b3a4-0b302b15b69b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fg6vh" Nov 03 13:08:41.301911 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.301230 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/39e11989-e598-4cea-87d9-26b3cf78a66b-host\") pod \"node-ca-4tn4j\" (UID: \"39e11989-e598-4cea-87d9-26b3cf78a66b\") " pod="openshift-image-registry/node-ca-4tn4j" Nov 03 13:08:41.301911 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.301266 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/74dc4aee-10b1-463c-a9c3-522b7a6c10b7-multus-cni-dir\") pod \"multus-dsjsv\" (UID: \"74dc4aee-10b1-463c-a9c3-522b7a6c10b7\") " pod="openshift-multus/multus-dsjsv" Nov 03 13:08:41.301911 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.301332 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/5ea5cf61-ed1c-4f5e-b1a0-5c46c5c3ee18-hosts-file\") pod \"node-resolver-v8gcq\" (UID: \"5ea5cf61-ed1c-4f5e-b1a0-5c46c5c3ee18\") " pod="openshift-dns/node-resolver-v8gcq" Nov 03 13:08:41.301911 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.301336 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/74dc4aee-10b1-463c-a9c3-522b7a6c10b7-multus-cni-dir\") pod \"multus-dsjsv\" (UID: \"74dc4aee-10b1-463c-a9c3-522b7a6c10b7\") " pod="openshift-multus/multus-dsjsv" Nov 03 13:08:41.301911 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.301357 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c704eb02-d156-4695-a55d-7dc96654e4cd-etc-systemd\") pod \"tuned-hmbpc\" (UID: \"c704eb02-d156-4695-a55d-7dc96654e4cd\") " pod="openshift-cluster-node-tuning-operator/tuned-hmbpc" Nov 03 13:08:41.301911 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.301388 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/5ea5cf61-ed1c-4f5e-b1a0-5c46c5c3ee18-hosts-file\") pod \"node-resolver-v8gcq\" (UID: \"5ea5cf61-ed1c-4f5e-b1a0-5c46c5c3ee18\") " pod="openshift-dns/node-resolver-v8gcq" Nov 03 13:08:41.301911 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.301403 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c704eb02-d156-4695-a55d-7dc96654e4cd-run\") pod \"tuned-hmbpc\" (UID: \"c704eb02-d156-4695-a55d-7dc96654e4cd\") " pod="openshift-cluster-node-tuning-operator/tuned-hmbpc" Nov 03 13:08:41.301911 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.301424 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/35e0fb40-48ce-4223-b7d4-c9290b6861cd-tuning-conf-dir\") pod \"multus-additional-cni-plugins-x2rm4\" (UID: \"35e0fb40-48ce-4223-b7d4-c9290b6861cd\") " pod="openshift-multus/multus-additional-cni-plugins-x2rm4" Nov 03 13:08:41.301911 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.301430 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c704eb02-d156-4695-a55d-7dc96654e4cd-etc-systemd\") pod \"tuned-hmbpc\" (UID: \"c704eb02-d156-4695-a55d-7dc96654e4cd\") " pod="openshift-cluster-node-tuning-operator/tuned-hmbpc" Nov 03 13:08:41.301911 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.301439 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/74dc4aee-10b1-463c-a9c3-522b7a6c10b7-host-run-multus-certs\") pod \"multus-dsjsv\" (UID: \"74dc4aee-10b1-463c-a9c3-522b7a6c10b7\") " pod="openshift-multus/multus-dsjsv" Nov 03 13:08:41.301911 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.301472 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c704eb02-d156-4695-a55d-7dc96654e4cd-run\") pod \"tuned-hmbpc\" (UID: \"c704eb02-d156-4695-a55d-7dc96654e4cd\") " pod="openshift-cluster-node-tuning-operator/tuned-hmbpc" Nov 03 13:08:41.301911 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.301485 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/35e0fb40-48ce-4223-b7d4-c9290b6861cd-tuning-conf-dir\") pod \"multus-additional-cni-plugins-x2rm4\" (UID: \"35e0fb40-48ce-4223-b7d4-c9290b6861cd\") " pod="openshift-multus/multus-additional-cni-plugins-x2rm4" Nov 03 13:08:41.301911 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.301477 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e0300f55-f002-4faa-bf6a-c04bea0a043e-metrics-certs\") pod \"network-metrics-daemon-f7hmh\" (UID: \"e0300f55-f002-4faa-bf6a-c04bea0a043e\") " pod="openshift-multus/network-metrics-daemon-f7hmh" Nov 03 13:08:41.301911 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.301504 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/74dc4aee-10b1-463c-a9c3-522b7a6c10b7-host-run-multus-certs\") pod \"multus-dsjsv\" (UID: \"74dc4aee-10b1-463c-a9c3-522b7a6c10b7\") " pod="openshift-multus/multus-dsjsv" Nov 03 13:08:41.301911 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.301527 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c704eb02-d156-4695-a55d-7dc96654e4cd-lib-modules\") pod \"tuned-hmbpc\" (UID: \"c704eb02-d156-4695-a55d-7dc96654e4cd\") " pod="openshift-cluster-node-tuning-operator/tuned-hmbpc" Nov 03 13:08:41.301911 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:08:41.301546 5024 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 03 13:08:41.301911 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.301553 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/898bd0b5-0d3e-4577-b101-2cc7d75283cb-metrics-certs\") pod \"router-default-7f579888d9-ntt6m\" (UID: \"898bd0b5-0d3e-4577-b101-2cc7d75283cb\") " pod="openshift-ingress/router-default-7f579888d9-ntt6m" Nov 03 13:08:41.301911 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.301580 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/74dc4aee-10b1-463c-a9c3-522b7a6c10b7-host-var-lib-cni-multus\") pod \"multus-dsjsv\" (UID: \"74dc4aee-10b1-463c-a9c3-522b7a6c10b7\") " pod="openshift-multus/multus-dsjsv" Nov 03 13:08:41.301911 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.301606 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3fdeb9f3-6c67-4390-8073-17d4d5f802b9-metrics-tls\") pod \"dns-default-vgnv8\" (UID: \"3fdeb9f3-6c67-4390-8073-17d4d5f802b9\") " pod="openshift-dns/dns-default-vgnv8" Nov 03 13:08:41.301911 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.301634 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9db7d195-220e-4644-8690-9e5ee37fc066-kubelet-dir\") pod \"aws-ebs-csi-driver-node-c9dvm\" (UID: \"9db7d195-220e-4644-8690-9e5ee37fc066\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c9dvm" Nov 03 13:08:41.301911 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.301660 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d926d945-39f4-48f3-b3a4-0b302b15b69b-host-run-ovn-kubernetes\") pod \"ovnkube-node-fg6vh\" (UID: \"d926d945-39f4-48f3-b3a4-0b302b15b69b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fg6vh" Nov 03 13:08:41.301911 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.301636 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/74dc4aee-10b1-463c-a9c3-522b7a6c10b7-host-var-lib-cni-multus\") pod \"multus-dsjsv\" (UID: \"74dc4aee-10b1-463c-a9c3-522b7a6c10b7\") " pod="openshift-multus/multus-dsjsv" Nov 03 13:08:41.301911 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.301691 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c704eb02-d156-4695-a55d-7dc96654e4cd-lib-modules\") pod \"tuned-hmbpc\" (UID: \"c704eb02-d156-4695-a55d-7dc96654e4cd\") " pod="openshift-cluster-node-tuning-operator/tuned-hmbpc" Nov 03 13:08:41.301911 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.301680 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9db7d195-220e-4644-8690-9e5ee37fc066-kubelet-dir\") pod \"aws-ebs-csi-driver-node-c9dvm\" (UID: \"9db7d195-220e-4644-8690-9e5ee37fc066\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c9dvm" Nov 03 13:08:41.301911 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:08:41.301715 5024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e0300f55-f002-4faa-bf6a-c04bea0a043e-metrics-certs podName:e0300f55-f002-4faa-bf6a-c04bea0a043e nodeName:}" failed. No retries permitted until 2026-11-03 13:08:41.801697245 +0000 UTC m=+1.809888507 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e0300f55-f002-4faa-bf6a-c04bea0a043e-metrics-certs") pod "network-metrics-daemon-f7hmh" (UID: "e0300f55-f002-4faa-bf6a-c04bea0a043e") : secret "metrics-daemon-secret" not found Nov 03 13:08:41.301911 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.301727 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d926d945-39f4-48f3-b3a4-0b302b15b69b-host-run-ovn-kubernetes\") pod \"ovnkube-node-fg6vh\" (UID: \"d926d945-39f4-48f3-b3a4-0b302b15b69b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fg6vh" Nov 03 13:08:41.301911 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.301737 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c704eb02-d156-4695-a55d-7dc96654e4cd-etc-sysctl-d\") pod \"tuned-hmbpc\" (UID: \"c704eb02-d156-4695-a55d-7dc96654e4cd\") " pod="openshift-cluster-node-tuning-operator/tuned-hmbpc" Nov 03 13:08:41.301911 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.301770 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/74dc4aee-10b1-463c-a9c3-522b7a6c10b7-etc-kubernetes\") pod \"multus-dsjsv\" (UID: \"74dc4aee-10b1-463c-a9c3-522b7a6c10b7\") " pod="openshift-multus/multus-dsjsv" Nov 03 13:08:41.301911 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.301794 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cc8ef30e-6d4f-4913-8dc8-b09e5aea2f82-registry-tls\") pod \"image-registry-555f8976d-mz5ct\" (UID: \"cc8ef30e-6d4f-4913-8dc8-b09e5aea2f82\") " pod="openshift-image-registry/image-registry-555f8976d-mz5ct" Nov 03 13:08:41.301911 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.301812 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c704eb02-d156-4695-a55d-7dc96654e4cd-etc-sysctl-d\") pod \"tuned-hmbpc\" (UID: \"c704eb02-d156-4695-a55d-7dc96654e4cd\") " pod="openshift-cluster-node-tuning-operator/tuned-hmbpc" Nov 03 13:08:41.301911 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.301817 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d926d945-39f4-48f3-b3a4-0b302b15b69b-host-run-netns\") pod \"ovnkube-node-fg6vh\" (UID: \"d926d945-39f4-48f3-b3a4-0b302b15b69b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fg6vh" Nov 03 13:08:41.301911 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.301838 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/74dc4aee-10b1-463c-a9c3-522b7a6c10b7-etc-kubernetes\") pod \"multus-dsjsv\" (UID: \"74dc4aee-10b1-463c-a9c3-522b7a6c10b7\") " pod="openshift-multus/multus-dsjsv" Nov 03 13:08:41.301911 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.301843 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c704eb02-d156-4695-a55d-7dc96654e4cd-etc-sysconfig\") pod \"tuned-hmbpc\" (UID: \"c704eb02-d156-4695-a55d-7dc96654e4cd\") " pod="openshift-cluster-node-tuning-operator/tuned-hmbpc" Nov 03 13:08:41.301911 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.301878 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fd7e8b58-de31-406a-8cdb-9ccfee140d57-host-slash\") pod \"iptables-alerter-g4xgp\" (UID: \"fd7e8b58-de31-406a-8cdb-9ccfee140d57\") " pod="openshift-network-operator/iptables-alerter-g4xgp" Nov 03 13:08:41.301911 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:08:41.301885 5024 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 13:08:41.301911 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.301890 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c704eb02-d156-4695-a55d-7dc96654e4cd-etc-sysconfig\") pod \"tuned-hmbpc\" (UID: \"c704eb02-d156-4695-a55d-7dc96654e4cd\") " pod="openshift-cluster-node-tuning-operator/tuned-hmbpc" Nov 03 13:08:41.301911 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:08:41.301897 5024 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-555f8976d-mz5ct: secret "image-registry-tls" not found Nov 03 13:08:41.301911 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.301937 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fd7e8b58-de31-406a-8cdb-9ccfee140d57-host-slash\") pod \"iptables-alerter-g4xgp\" (UID: \"fd7e8b58-de31-406a-8cdb-9ccfee140d57\") " pod="openshift-network-operator/iptables-alerter-g4xgp" Nov 03 13:08:41.301911 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:08:41.301949 5024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cc8ef30e-6d4f-4913-8dc8-b09e5aea2f82-registry-tls podName:cc8ef30e-6d4f-4913-8dc8-b09e5aea2f82 nodeName:}" failed. No retries permitted until 2026-11-03 13:08:41.801936361 +0000 UTC m=+1.810127640 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/cc8ef30e-6d4f-4913-8dc8-b09e5aea2f82-registry-tls") pod "image-registry-555f8976d-mz5ct" (UID: "cc8ef30e-6d4f-4913-8dc8-b09e5aea2f82") : secret "image-registry-tls" not found Nov 03 13:08:41.301911 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.301908 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/898bd0b5-0d3e-4577-b101-2cc7d75283cb-service-ca-bundle\") pod \"router-default-7f579888d9-ntt6m\" (UID: \"898bd0b5-0d3e-4577-b101-2cc7d75283cb\") " pod="openshift-ingress/router-default-7f579888d9-ntt6m" Nov 03 13:08:41.301911 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.301970 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d926d945-39f4-48f3-b3a4-0b302b15b69b-host-run-netns\") pod \"ovnkube-node-fg6vh\" (UID: \"d926d945-39f4-48f3-b3a4-0b302b15b69b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fg6vh" Nov 03 13:08:41.301911 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:08:41.301985 5024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/898bd0b5-0d3e-4577-b101-2cc7d75283cb-service-ca-bundle podName:898bd0b5-0d3e-4577-b101-2cc7d75283cb nodeName:}" failed. No retries permitted until 2026-11-03 13:08:41.801975658 +0000 UTC m=+1.810166931 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/898bd0b5-0d3e-4577-b101-2cc7d75283cb-service-ca-bundle") pod "router-default-7f579888d9-ntt6m" (UID: "898bd0b5-0d3e-4577-b101-2cc7d75283cb") : configmap references non-existent config key: service-ca.crt Nov 03 13:08:41.304902 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:08:41.302013 5024 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 03 13:08:41.304902 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.302020 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d926d945-39f4-48f3-b3a4-0b302b15b69b-systemd-units\") pod \"ovnkube-node-fg6vh\" (UID: \"d926d945-39f4-48f3-b3a4-0b302b15b69b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fg6vh" Nov 03 13:08:41.304902 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.302044 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c704eb02-d156-4695-a55d-7dc96654e4cd-etc-modprobe-d\") pod \"tuned-hmbpc\" (UID: \"c704eb02-d156-4695-a55d-7dc96654e4cd\") " pod="openshift-cluster-node-tuning-operator/tuned-hmbpc" Nov 03 13:08:41.304902 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:08:41.302069 5024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/898bd0b5-0d3e-4577-b101-2cc7d75283cb-metrics-certs podName:898bd0b5-0d3e-4577-b101-2cc7d75283cb nodeName:}" failed. No retries permitted until 2026-11-03 13:08:41.802042504 +0000 UTC m=+1.810233758 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/898bd0b5-0d3e-4577-b101-2cc7d75283cb-metrics-certs") pod "router-default-7f579888d9-ntt6m" (UID: "898bd0b5-0d3e-4577-b101-2cc7d75283cb") : secret "router-metrics-certs-default" not found Nov 03 13:08:41.304902 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.302095 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c704eb02-d156-4695-a55d-7dc96654e4cd-etc-sysctl-conf\") pod \"tuned-hmbpc\" (UID: \"c704eb02-d156-4695-a55d-7dc96654e4cd\") " pod="openshift-cluster-node-tuning-operator/tuned-hmbpc" Nov 03 13:08:41.304902 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.302124 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/74dc4aee-10b1-463c-a9c3-522b7a6c10b7-host-run-k8s-cni-cncf-io\") pod \"multus-dsjsv\" (UID: \"74dc4aee-10b1-463c-a9c3-522b7a6c10b7\") " pod="openshift-multus/multus-dsjsv" Nov 03 13:08:41.304902 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.302159 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d926d945-39f4-48f3-b3a4-0b302b15b69b-run-systemd\") pod \"ovnkube-node-fg6vh\" (UID: \"d926d945-39f4-48f3-b3a4-0b302b15b69b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fg6vh" Nov 03 13:08:41.304902 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.302125 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c704eb02-d156-4695-a55d-7dc96654e4cd-etc-modprobe-d\") pod \"tuned-hmbpc\" (UID: \"c704eb02-d156-4695-a55d-7dc96654e4cd\") " pod="openshift-cluster-node-tuning-operator/tuned-hmbpc" Nov 03 13:08:41.304902 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.302160 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/74dc4aee-10b1-463c-a9c3-522b7a6c10b7-host-run-k8s-cni-cncf-io\") pod \"multus-dsjsv\" (UID: \"74dc4aee-10b1-463c-a9c3-522b7a6c10b7\") " pod="openshift-multus/multus-dsjsv" Nov 03 13:08:41.304902 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.302183 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/74dc4aee-10b1-463c-a9c3-522b7a6c10b7-host-var-lib-cni-bin\") pod \"multus-dsjsv\" (UID: \"74dc4aee-10b1-463c-a9c3-522b7a6c10b7\") " pod="openshift-multus/multus-dsjsv" Nov 03 13:08:41.304902 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.302210 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d926d945-39f4-48f3-b3a4-0b302b15b69b-run-systemd\") pod \"ovnkube-node-fg6vh\" (UID: \"d926d945-39f4-48f3-b3a4-0b302b15b69b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fg6vh" Nov 03 13:08:41.304902 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.302097 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d926d945-39f4-48f3-b3a4-0b302b15b69b-systemd-units\") pod \"ovnkube-node-fg6vh\" (UID: \"d926d945-39f4-48f3-b3a4-0b302b15b69b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fg6vh" Nov 03 13:08:41.304902 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.302214 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c704eb02-d156-4695-a55d-7dc96654e4cd-etc-sysctl-conf\") pod \"tuned-hmbpc\" (UID: \"c704eb02-d156-4695-a55d-7dc96654e4cd\") " pod="openshift-cluster-node-tuning-operator/tuned-hmbpc" Nov 03 13:08:41.304902 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.302220 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/74dc4aee-10b1-463c-a9c3-522b7a6c10b7-multus-conf-dir\") pod \"multus-dsjsv\" (UID: \"74dc4aee-10b1-463c-a9c3-522b7a6c10b7\") " pod="openshift-multus/multus-dsjsv" Nov 03 13:08:41.304902 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.302252 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/74dc4aee-10b1-463c-a9c3-522b7a6c10b7-multus-conf-dir\") pod \"multus-dsjsv\" (UID: \"74dc4aee-10b1-463c-a9c3-522b7a6c10b7\") " pod="openshift-multus/multus-dsjsv" Nov 03 13:08:41.304902 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.302255 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/74dc4aee-10b1-463c-a9c3-522b7a6c10b7-host-var-lib-cni-bin\") pod \"multus-dsjsv\" (UID: \"74dc4aee-10b1-463c-a9c3-522b7a6c10b7\") " pod="openshift-multus/multus-dsjsv" Nov 03 13:08:41.304902 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.302254 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/7e280277-f29f-4a2b-8b85-2491529e5ef2-kubelet-config\") pod \"global-pull-secret-syncer-5jz7t\" (UID: \"7e280277-f29f-4a2b-8b85-2491529e5ef2\") " pod="kube-system/global-pull-secret-syncer-5jz7t" Nov 03 13:08:41.304902 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.302306 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/7e280277-f29f-4a2b-8b85-2491529e5ef2-kubelet-config\") pod \"global-pull-secret-syncer-5jz7t\" (UID: \"7e280277-f29f-4a2b-8b85-2491529e5ef2\") " pod="kube-system/global-pull-secret-syncer-5jz7t" Nov 03 13:08:41.304902 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.302323 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d926d945-39f4-48f3-b3a4-0b302b15b69b-node-log\") pod \"ovnkube-node-fg6vh\" (UID: \"d926d945-39f4-48f3-b3a4-0b302b15b69b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fg6vh" Nov 03 13:08:41.304902 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.302343 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d926d945-39f4-48f3-b3a4-0b302b15b69b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-fg6vh\" (UID: \"d926d945-39f4-48f3-b3a4-0b302b15b69b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fg6vh" Nov 03 13:08:41.304902 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.302405 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d926d945-39f4-48f3-b3a4-0b302b15b69b-node-log\") pod \"ovnkube-node-fg6vh\" (UID: \"d926d945-39f4-48f3-b3a4-0b302b15b69b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fg6vh" Nov 03 13:08:41.304902 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.302415 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d926d945-39f4-48f3-b3a4-0b302b15b69b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-fg6vh\" (UID: \"d926d945-39f4-48f3-b3a4-0b302b15b69b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fg6vh" Nov 03 13:08:41.304902 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.303326 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/fa297a30-6da7-4a92-98fb-201323490bb0-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-rz4n9\" (UID: \"fa297a30-6da7-4a92-98fb-201323490bb0\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-rz4n9" Nov 03 13:08:41.304902 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.303651 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fff35d1b-e7b2-4dd3-9824-8b64fad02506-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-d7mwz\" (UID: \"fff35d1b-e7b2-4dd3-9824-8b64fad02506\") " pod="openshift-insights/insights-runtime-extractor-d7mwz" Nov 03 13:08:41.304902 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.303849 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/21618783-0a4e-4b1f-8041-cabdf69f5bea-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-h46gl\" (UID: \"21618783-0a4e-4b1f-8041-cabdf69f5bea\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-h46gl" Nov 03 13:08:41.304902 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.303933 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3fdeb9f3-6c67-4390-8073-17d4d5f802b9-metrics-tls\") pod \"dns-default-vgnv8\" (UID: \"3fdeb9f3-6c67-4390-8073-17d4d5f802b9\") " pod="openshift-dns/dns-default-vgnv8" Nov 03 13:08:41.372607 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.372578 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-tm4nb" Nov 03 13:08:41.374354 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.374336 5024 scope.go:122] "RemoveContainer" containerID="b811aececd178145fbcac8b298ed317468857bb459f9492b4b679c31b566f969" Nov 03 13:08:41.378405 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.378387 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-h46gl" Nov 03 13:08:41.379356 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.379340 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-rz4n9" Nov 03 13:08:41.386758 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.386737 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-vgnv8" Nov 03 13:08:41.387738 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.387721 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-d7mwz" Nov 03 13:08:41.678346 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.678255 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-rz4n9"] Nov 03 13:08:41.681701 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.681674 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-tm4nb"] Nov 03 13:08:41.682180 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.682152 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-h46gl"] Nov 03 13:08:41.682272 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:41.682218 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa297a30_6da7_4a92_98fb_201323490bb0.slice/crio-261247e43faca763a238188dd4310f58dd68dc5aac45517362444152e847ce95 WatchSource:0}: Error finding container 261247e43faca763a238188dd4310f58dd68dc5aac45517362444152e847ce95: Status 404 returned error can't find the container with id 261247e43faca763a238188dd4310f58dd68dc5aac45517362444152e847ce95 Nov 03 13:08:41.684665 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.684611 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-d7mwz"] Nov 03 13:08:41.685330 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.685309 5024 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 13:08:41.687802 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:41.687756 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3042b60_062f_4f9b_8406_f884c95a8729.slice/crio-5b1ecf95dfba6638e8ae2c69e9c32e6d543a7fe6a6d930205e3f184e0f22636d WatchSource:0}: Error finding container 5b1ecf95dfba6638e8ae2c69e9c32e6d543a7fe6a6d930205e3f184e0f22636d: Status 404 returned error can't find the container with id 5b1ecf95dfba6638e8ae2c69e9c32e6d543a7fe6a6d930205e3f184e0f22636d Nov 03 13:08:41.688950 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:41.688922 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfff35d1b_e7b2_4dd3_9824_8b64fad02506.slice/crio-344825fba43454845813ca19e2a90bdb5497119875281dffcb4d9bccb3afc2bf WatchSource:0}: Error finding container 344825fba43454845813ca19e2a90bdb5497119875281dffcb4d9bccb3afc2bf: Status 404 returned error can't find the container with id 344825fba43454845813ca19e2a90bdb5497119875281dffcb4d9bccb3afc2bf Nov 03 13:08:41.699836 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.699811 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-vgnv8"] Nov 03 13:08:41.705725 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:41.705702 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3fdeb9f3_6c67_4390_8073_17d4d5f802b9.slice/crio-a95035061266bc1e4d83116dc9f3676627d28d9690e0728a701a339d9f942d7d WatchSource:0}: Error finding container a95035061266bc1e4d83116dc9f3676627d28d9690e0728a701a339d9f942d7d: Status 404 returned error can't find the container with id a95035061266bc1e4d83116dc9f3676627d28d9690e0728a701a339d9f942d7d Nov 03 13:08:41.807267 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.807226 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0a892eb3-ecc4-4312-9f4d-73eeba6b6eee-registry-tls\") pod \"image-registry-7dbb67bd57-74qfj\" (UID: \"0a892eb3-ecc4-4312-9f4d-73eeba6b6eee\") " pod="openshift-image-registry/image-registry-7dbb67bd57-74qfj" Nov 03 13:08:41.807267 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.807273 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e0300f55-f002-4faa-bf6a-c04bea0a043e-metrics-certs\") pod \"network-metrics-daemon-f7hmh\" (UID: \"e0300f55-f002-4faa-bf6a-c04bea0a043e\") " pod="openshift-multus/network-metrics-daemon-f7hmh" Nov 03 13:08:41.807528 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:08:41.807384 5024 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 03 13:08:41.807528 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:08:41.807390 5024 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 13:08:41.807528 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:08:41.807409 5024 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7dbb67bd57-74qfj: secret "image-registry-tls" not found Nov 03 13:08:41.807528 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:08:41.807435 5024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e0300f55-f002-4faa-bf6a-c04bea0a043e-metrics-certs podName:e0300f55-f002-4faa-bf6a-c04bea0a043e nodeName:}" failed. No retries permitted until 2026-11-03 13:08:42.807422418 +0000 UTC m=+2.815613674 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e0300f55-f002-4faa-bf6a-c04bea0a043e-metrics-certs") pod "network-metrics-daemon-f7hmh" (UID: "e0300f55-f002-4faa-bf6a-c04bea0a043e") : secret "metrics-daemon-secret" not found Nov 03 13:08:41.807528 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:08:41.807453 5024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0a892eb3-ecc4-4312-9f4d-73eeba6b6eee-registry-tls podName:0a892eb3-ecc4-4312-9f4d-73eeba6b6eee nodeName:}" failed. No retries permitted until 2026-11-03 13:08:42.807440477 +0000 UTC m=+2.815631750 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/0a892eb3-ecc4-4312-9f4d-73eeba6b6eee-registry-tls") pod "image-registry-7dbb67bd57-74qfj" (UID: "0a892eb3-ecc4-4312-9f4d-73eeba6b6eee") : secret "image-registry-tls" not found Nov 03 13:08:41.807528 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.807471 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/898bd0b5-0d3e-4577-b101-2cc7d75283cb-metrics-certs\") pod \"router-default-7f579888d9-ntt6m\" (UID: \"898bd0b5-0d3e-4577-b101-2cc7d75283cb\") " pod="openshift-ingress/router-default-7f579888d9-ntt6m" Nov 03 13:08:41.807528 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.807506 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cc8ef30e-6d4f-4913-8dc8-b09e5aea2f82-registry-tls\") pod \"image-registry-555f8976d-mz5ct\" (UID: \"cc8ef30e-6d4f-4913-8dc8-b09e5aea2f82\") " pod="openshift-image-registry/image-registry-555f8976d-mz5ct" Nov 03 13:08:41.807528 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:41.807529 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/898bd0b5-0d3e-4577-b101-2cc7d75283cb-service-ca-bundle\") pod \"router-default-7f579888d9-ntt6m\" (UID: \"898bd0b5-0d3e-4577-b101-2cc7d75283cb\") " pod="openshift-ingress/router-default-7f579888d9-ntt6m" Nov 03 13:08:41.807898 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:08:41.807612 5024 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 13:08:41.807898 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:08:41.807629 5024 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-555f8976d-mz5ct: secret "image-registry-tls" not found Nov 03 13:08:41.807898 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:08:41.807635 5024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/898bd0b5-0d3e-4577-b101-2cc7d75283cb-service-ca-bundle podName:898bd0b5-0d3e-4577-b101-2cc7d75283cb nodeName:}" failed. No retries permitted until 2026-11-03 13:08:42.807624798 +0000 UTC m=+2.815816076 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/898bd0b5-0d3e-4577-b101-2cc7d75283cb-service-ca-bundle") pod "router-default-7f579888d9-ntt6m" (UID: "898bd0b5-0d3e-4577-b101-2cc7d75283cb") : configmap references non-existent config key: service-ca.crt Nov 03 13:08:41.807898 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:08:41.807662 5024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cc8ef30e-6d4f-4913-8dc8-b09e5aea2f82-registry-tls podName:cc8ef30e-6d4f-4913-8dc8-b09e5aea2f82 nodeName:}" failed. No retries permitted until 2026-11-03 13:08:42.807652682 +0000 UTC m=+2.815843937 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/cc8ef30e-6d4f-4913-8dc8-b09e5aea2f82-registry-tls") pod "image-registry-555f8976d-mz5ct" (UID: "cc8ef30e-6d4f-4913-8dc8-b09e5aea2f82") : secret "image-registry-tls" not found Nov 03 13:08:41.807898 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:08:41.807722 5024 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 03 13:08:41.807898 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:08:41.807777 5024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/898bd0b5-0d3e-4577-b101-2cc7d75283cb-metrics-certs podName:898bd0b5-0d3e-4577-b101-2cc7d75283cb nodeName:}" failed. No retries permitted until 2026-11-03 13:08:42.807763808 +0000 UTC m=+2.815955079 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/898bd0b5-0d3e-4577-b101-2cc7d75283cb-metrics-certs") pod "router-default-7f579888d9-ntt6m" (UID: "898bd0b5-0d3e-4577-b101-2cc7d75283cb") : secret "router-metrics-certs-default" not found Nov 03 13:08:42.165784 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:42.165750 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-vgnv8" event={"ID":"3fdeb9f3-6c67-4390-8073-17d4d5f802b9","Type":"ContainerStarted","Data":"a95035061266bc1e4d83116dc9f3676627d28d9690e0728a701a339d9f942d7d"} Nov 03 13:08:42.167582 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:42.167501 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-tm4nb" event={"ID":"a3042b60-062f-4f9b-8406-f884c95a8729","Type":"ContainerStarted","Data":"5b1ecf95dfba6638e8ae2c69e9c32e6d543a7fe6a6d930205e3f184e0f22636d"} Nov 03 13:08:42.169839 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:42.169805 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-d7mwz" event={"ID":"fff35d1b-e7b2-4dd3-9824-8b64fad02506","Type":"ContainerStarted","Data":"565ca085cab0b7c959e74c533c7e5b98f0864cd40783b49e92a95bf326dca1c1"} Nov 03 13:08:42.169955 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:42.169867 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-d7mwz" event={"ID":"fff35d1b-e7b2-4dd3-9824-8b64fad02506","Type":"ContainerStarted","Data":"344825fba43454845813ca19e2a90bdb5497119875281dffcb4d9bccb3afc2bf"} Nov 03 13:08:42.171488 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:42.171416 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-rz4n9" event={"ID":"fa297a30-6da7-4a92-98fb-201323490bb0","Type":"ContainerStarted","Data":"261247e43faca763a238188dd4310f58dd68dc5aac45517362444152e847ce95"} Nov 03 13:08:42.173189 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:42.173162 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-h46gl" event={"ID":"21618783-0a4e-4b1f-8041-cabdf69f5bea","Type":"ContainerStarted","Data":"9e14c33c58d4d89c8d579faaec635f0e88a36acc506dcb906437a7190bcc04e6"} Nov 03 13:08:42.175792 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:42.175772 5024 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6htht_cd91de08-084e-4667-b13f-2e07b3cff624/console-operator/1.log" Nov 03 13:08:42.175892 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:42.175824 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-6htht" event={"ID":"cd91de08-084e-4667-b13f-2e07b3cff624","Type":"ContainerStarted","Data":"0ca721f8e581004c02261102494e0d99c0664b9cd03fc7545e1532a4520b8f12"} Nov 03 13:08:42.819429 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:42.819387 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cc8ef30e-6d4f-4913-8dc8-b09e5aea2f82-registry-tls\") pod \"image-registry-555f8976d-mz5ct\" (UID: \"cc8ef30e-6d4f-4913-8dc8-b09e5aea2f82\") " pod="openshift-image-registry/image-registry-555f8976d-mz5ct" Nov 03 13:08:42.819603 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:42.819447 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/898bd0b5-0d3e-4577-b101-2cc7d75283cb-service-ca-bundle\") pod \"router-default-7f579888d9-ntt6m\" (UID: \"898bd0b5-0d3e-4577-b101-2cc7d75283cb\") " pod="openshift-ingress/router-default-7f579888d9-ntt6m" Nov 03 13:08:42.819667 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:42.819625 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0a892eb3-ecc4-4312-9f4d-73eeba6b6eee-registry-tls\") pod \"image-registry-7dbb67bd57-74qfj\" (UID: \"0a892eb3-ecc4-4312-9f4d-73eeba6b6eee\") " pod="openshift-image-registry/image-registry-7dbb67bd57-74qfj" Nov 03 13:08:42.819719 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:42.819674 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e0300f55-f002-4faa-bf6a-c04bea0a043e-metrics-certs\") pod \"network-metrics-daemon-f7hmh\" (UID: \"e0300f55-f002-4faa-bf6a-c04bea0a043e\") " pod="openshift-multus/network-metrics-daemon-f7hmh" Nov 03 13:08:42.819719 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:42.819707 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/898bd0b5-0d3e-4577-b101-2cc7d75283cb-metrics-certs\") pod \"router-default-7f579888d9-ntt6m\" (UID: \"898bd0b5-0d3e-4577-b101-2cc7d75283cb\") " pod="openshift-ingress/router-default-7f579888d9-ntt6m" Nov 03 13:08:42.819897 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:08:42.819839 5024 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 03 13:08:42.819957 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:08:42.819908 5024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/898bd0b5-0d3e-4577-b101-2cc7d75283cb-metrics-certs podName:898bd0b5-0d3e-4577-b101-2cc7d75283cb nodeName:}" failed. No retries permitted until 2026-11-03 13:08:44.819889804 +0000 UTC m=+4.828081060 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/898bd0b5-0d3e-4577-b101-2cc7d75283cb-metrics-certs") pod "router-default-7f579888d9-ntt6m" (UID: "898bd0b5-0d3e-4577-b101-2cc7d75283cb") : secret "router-metrics-certs-default" not found Nov 03 13:08:42.821082 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:08:42.821059 5024 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 03 13:08:42.821199 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:08:42.821125 5024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e0300f55-f002-4faa-bf6a-c04bea0a043e-metrics-certs podName:e0300f55-f002-4faa-bf6a-c04bea0a043e nodeName:}" failed. No retries permitted until 2026-11-03 13:08:44.821108352 +0000 UTC m=+4.829299625 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e0300f55-f002-4faa-bf6a-c04bea0a043e-metrics-certs") pod "network-metrics-daemon-f7hmh" (UID: "e0300f55-f002-4faa-bf6a-c04bea0a043e") : secret "metrics-daemon-secret" not found Nov 03 13:08:42.821273 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:08:42.821255 5024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/898bd0b5-0d3e-4577-b101-2cc7d75283cb-service-ca-bundle podName:898bd0b5-0d3e-4577-b101-2cc7d75283cb nodeName:}" failed. No retries permitted until 2026-11-03 13:08:44.821216637 +0000 UTC m=+4.829407905 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/898bd0b5-0d3e-4577-b101-2cc7d75283cb-service-ca-bundle") pod "router-default-7f579888d9-ntt6m" (UID: "898bd0b5-0d3e-4577-b101-2cc7d75283cb") : configmap references non-existent config key: service-ca.crt Nov 03 13:08:42.823454 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:42.823431 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cc8ef30e-6d4f-4913-8dc8-b09e5aea2f82-registry-tls\") pod \"image-registry-555f8976d-mz5ct\" (UID: \"cc8ef30e-6d4f-4913-8dc8-b09e5aea2f82\") " pod="openshift-image-registry/image-registry-555f8976d-mz5ct" Nov 03 13:08:42.823864 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:42.823847 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0a892eb3-ecc4-4312-9f4d-73eeba6b6eee-registry-tls\") pod \"image-registry-7dbb67bd57-74qfj\" (UID: \"0a892eb3-ecc4-4312-9f4d-73eeba6b6eee\") " pod="openshift-image-registry/image-registry-7dbb67bd57-74qfj" Nov 03 13:08:42.878534 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:42.878449 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-555f8976d-mz5ct" Nov 03 13:08:42.885752 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:42.885721 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7dbb67bd57-74qfj" Nov 03 13:08:42.975156 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:42.973945 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-fg6vh" Nov 03 13:08:43.184208 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:43.184094 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-d7mwz" event={"ID":"fff35d1b-e7b2-4dd3-9824-8b64fad02506","Type":"ContainerStarted","Data":"0225f493e47885a8e5871f0e4c64c2e447d8d3b9ba87707702a27cb0d90b07b0"} Nov 03 13:08:43.193609 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:43.190194 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7dbb67bd57-74qfj"] Nov 03 13:08:43.193609 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:43.193461 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-555f8976d-mz5ct"] Nov 03 13:08:43.203335 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:43.203267 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc8ef30e_6d4f_4913_8dc8_b09e5aea2f82.slice/crio-4278f1b6983e7acc4d4d3d30127f25a11e647c906157a6127809999b409890a2 WatchSource:0}: Error finding container 4278f1b6983e7acc4d4d3d30127f25a11e647c906157a6127809999b409890a2: Status 404 returned error can't find the container with id 4278f1b6983e7acc4d4d3d30127f25a11e647c906157a6127809999b409890a2 Nov 03 13:08:43.204264 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:43.204234 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a892eb3_ecc4_4312_9f4d_73eeba6b6eee.slice/crio-b13ded60514581be37c7f4965199d2ba2c9988e1bc8866f5a6ad43fe344af28b WatchSource:0}: Error finding container b13ded60514581be37c7f4965199d2ba2c9988e1bc8866f5a6ad43fe344af28b: Status 404 returned error can't find the container with id b13ded60514581be37c7f4965199d2ba2c9988e1bc8866f5a6ad43fe344af28b Nov 03 13:08:44.190811 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:44.190669 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-555f8976d-mz5ct" event={"ID":"cc8ef30e-6d4f-4913-8dc8-b09e5aea2f82","Type":"ContainerStarted","Data":"7c251d5c4ec97f54c433212e0c2983af902326bb7d9b279efbdbcae45f1e0acb"} Nov 03 13:08:44.190811 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:44.190731 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-555f8976d-mz5ct" event={"ID":"cc8ef30e-6d4f-4913-8dc8-b09e5aea2f82","Type":"ContainerStarted","Data":"4278f1b6983e7acc4d4d3d30127f25a11e647c906157a6127809999b409890a2"} Nov 03 13:08:44.193992 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:44.193945 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-rz4n9" event={"ID":"fa297a30-6da7-4a92-98fb-201323490bb0","Type":"ContainerStarted","Data":"ef0661c7f727a8aedc521a2cbbcd53769b4875b54ef80374a689ce0e1fd689b8"} Nov 03 13:08:44.195856 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:44.195831 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7dbb67bd57-74qfj" event={"ID":"0a892eb3-ecc4-4312-9f4d-73eeba6b6eee","Type":"ContainerStarted","Data":"e05f253dbd0d764b33d094ec44e84d449c11ff3d57a84848cc2970460a1f1c35"} Nov 03 13:08:44.195941 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:44.195870 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7dbb67bd57-74qfj" event={"ID":"0a892eb3-ecc4-4312-9f4d-73eeba6b6eee","Type":"ContainerStarted","Data":"b13ded60514581be37c7f4965199d2ba2c9988e1bc8866f5a6ad43fe344af28b"} Nov 03 13:08:44.570160 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:44.570105 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-6htht" Nov 03 13:08:44.570362 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:44.570319 5024 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 03 13:08:44.574231 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:44.574205 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-6htht" Nov 03 13:08:44.774982 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:44.774963 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-fg6vh" Nov 03 13:08:44.775243 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:44.775231 5024 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 03 13:08:44.853306 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:44.853257 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e0300f55-f002-4faa-bf6a-c04bea0a043e-metrics-certs\") pod \"network-metrics-daemon-f7hmh\" (UID: \"e0300f55-f002-4faa-bf6a-c04bea0a043e\") " pod="openshift-multus/network-metrics-daemon-f7hmh" Nov 03 13:08:44.853459 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:44.853339 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/898bd0b5-0d3e-4577-b101-2cc7d75283cb-metrics-certs\") pod \"router-default-7f579888d9-ntt6m\" (UID: \"898bd0b5-0d3e-4577-b101-2cc7d75283cb\") " pod="openshift-ingress/router-default-7f579888d9-ntt6m" Nov 03 13:08:44.853627 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:44.853597 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/898bd0b5-0d3e-4577-b101-2cc7d75283cb-service-ca-bundle\") pod \"router-default-7f579888d9-ntt6m\" (UID: \"898bd0b5-0d3e-4577-b101-2cc7d75283cb\") " pod="openshift-ingress/router-default-7f579888d9-ntt6m" Nov 03 13:08:44.853716 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:08:44.853695 5024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/898bd0b5-0d3e-4577-b101-2cc7d75283cb-service-ca-bundle podName:898bd0b5-0d3e-4577-b101-2cc7d75283cb nodeName:}" failed. No retries permitted until 2026-11-03 13:08:48.853676887 +0000 UTC m=+8.861868157 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/898bd0b5-0d3e-4577-b101-2cc7d75283cb-service-ca-bundle") pod "router-default-7f579888d9-ntt6m" (UID: "898bd0b5-0d3e-4577-b101-2cc7d75283cb") : configmap references non-existent config key: service-ca.crt Nov 03 13:08:44.855724 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:44.855695 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e0300f55-f002-4faa-bf6a-c04bea0a043e-metrics-certs\") pod \"network-metrics-daemon-f7hmh\" (UID: \"e0300f55-f002-4faa-bf6a-c04bea0a043e\") " pod="openshift-multus/network-metrics-daemon-f7hmh" Nov 03 13:08:44.855810 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:44.855729 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/898bd0b5-0d3e-4577-b101-2cc7d75283cb-metrics-certs\") pod \"router-default-7f579888d9-ntt6m\" (UID: \"898bd0b5-0d3e-4577-b101-2cc7d75283cb\") " pod="openshift-ingress/router-default-7f579888d9-ntt6m" Nov 03 13:08:44.982891 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:44.982821 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f7hmh" Nov 03 13:08:45.201433 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:45.201398 5024 generic.go:356] "Generic (PLEG): container finished" podID="9fa6d32d-b30b-4b76-9462-5f3f09443d02" containerID="61609a0eadc2e90859796d714a04559f7442b35166cf465ad0da1508ed8e60ec" exitCode=0 Nov 03 13:08:45.201824 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:45.201474 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-4t8lj" event={"ID":"9fa6d32d-b30b-4b76-9462-5f3f09443d02","Type":"ContainerDied","Data":"61609a0eadc2e90859796d714a04559f7442b35166cf465ad0da1508ed8e60ec"} Nov 03 13:08:45.201898 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:45.201883 5024 scope.go:122] "RemoveContainer" containerID="61609a0eadc2e90859796d714a04559f7442b35166cf465ad0da1508ed8e60ec" Nov 03 13:08:45.203582 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:45.203546 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-vgnv8" event={"ID":"3fdeb9f3-6c67-4390-8073-17d4d5f802b9","Type":"ContainerStarted","Data":"07e50c79f392e7e5735c5f03d91284270756ea458a5eb2cce2b472e30eeea8af"} Nov 03 13:08:45.203690 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:45.203596 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-vgnv8" event={"ID":"3fdeb9f3-6c67-4390-8073-17d4d5f802b9","Type":"ContainerStarted","Data":"0aad8faff1ed7b78d520a1d41c81b32c96fc6627292c49d77ce0501eade51dd5"} Nov 03 13:08:45.203690 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:45.203664 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-vgnv8" Nov 03 13:08:45.205193 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:45.205166 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-tm4nb" event={"ID":"a3042b60-062f-4f9b-8406-f884c95a8729","Type":"ContainerStarted","Data":"92dfe49cd70793b77a8a3bb649a6ecd97475b101d9a2e67320a55a64adf6fba7"} Nov 03 13:08:45.207386 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:45.207362 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-d7mwz" event={"ID":"fff35d1b-e7b2-4dd3-9824-8b64fad02506","Type":"ContainerStarted","Data":"a47187c6f05ad444068556227c4f3c1f629ce96cee17624fe49ac34b887656c3"} Nov 03 13:08:45.209167 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:45.209145 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-h46gl" event={"ID":"21618783-0a4e-4b1f-8041-cabdf69f5bea","Type":"ContainerStarted","Data":"9c425c84f08d2552fb7b3b510e14b225d1156f8aeb8ea4ab7d95600f70af1781"} Nov 03 13:08:45.209249 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:45.209187 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-h46gl" event={"ID":"21618783-0a4e-4b1f-8041-cabdf69f5bea","Type":"ContainerStarted","Data":"732841007db228101c5f95e4defac0a3084f6d818f2e55bbaa63957a7fcf9481"} Nov 03 13:08:45.209249 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:45.209238 5024 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 03 13:08:45.209372 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:45.209301 5024 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 03 13:08:45.247217 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:45.247183 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-7dbb67bd57-74qfj" Nov 03 13:08:45.256071 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:45.256048 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-f7hmh"] Nov 03 13:08:45.265365 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:45.265342 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0300f55_f002_4faa_bf6a_c04bea0a043e.slice/crio-35fe6303538508513d23ae0db0d313b9f317df718524ad10ebc5ecc65c879a58 WatchSource:0}: Error finding container 35fe6303538508513d23ae0db0d313b9f317df718524ad10ebc5ecc65c879a58: Status 404 returned error can't find the container with id 35fe6303538508513d23ae0db0d313b9f317df718524ad10ebc5ecc65c879a58 Nov 03 13:08:45.355974 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:45.355943 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-fg6vh" Nov 03 13:08:46.214756 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:46.214721 5024 generic.go:356] "Generic (PLEG): container finished" podID="6cb478ab-97aa-4881-87b2-b45293cc2349" containerID="402758d12c8d4eaafa8121774c00f0b07ba63f8606dc2f3933548434390c58a4" exitCode=0 Nov 03 13:08:46.215210 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:46.214797 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-6nz8d" event={"ID":"6cb478ab-97aa-4881-87b2-b45293cc2349","Type":"ContainerDied","Data":"402758d12c8d4eaafa8121774c00f0b07ba63f8606dc2f3933548434390c58a4"} Nov 03 13:08:46.215210 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:46.215206 5024 scope.go:122] "RemoveContainer" containerID="402758d12c8d4eaafa8121774c00f0b07ba63f8606dc2f3933548434390c58a4" Nov 03 13:08:46.216795 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:46.216772 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-4t8lj" event={"ID":"9fa6d32d-b30b-4b76-9462-5f3f09443d02","Type":"ContainerStarted","Data":"38fe2165f6f7db74dcc7eeeb77243c793837b0499c3d1eb201d8ff3f0db023ef"} Nov 03 13:08:46.217849 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:46.217825 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-f7hmh" event={"ID":"e0300f55-f002-4faa-bf6a-c04bea0a043e","Type":"ContainerStarted","Data":"35fe6303538508513d23ae0db0d313b9f317df718524ad10ebc5ecc65c879a58"} Nov 03 13:08:47.223954 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:47.223907 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-f7hmh" event={"ID":"e0300f55-f002-4faa-bf6a-c04bea0a043e","Type":"ContainerStarted","Data":"c7e4bf9a825f728b93e56e1186078509c7cce63ea95106ad7ed974546620d924"} Nov 03 13:08:47.224446 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:47.224008 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-f7hmh" event={"ID":"e0300f55-f002-4faa-bf6a-c04bea0a043e","Type":"ContainerStarted","Data":"f01c2b4ecab3765ddef2f56f175acab9e449dc0a21a30de2e70b4265dfd93398"} Nov 03 13:08:47.225825 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:47.225799 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-6nz8d" event={"ID":"6cb478ab-97aa-4881-87b2-b45293cc2349","Type":"ContainerStarted","Data":"62d893fc20dd85ec2c806eb86b8662f308e216483901d56df1eec25025f6c01c"} Nov 03 13:08:48.890268 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:48.890224 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/898bd0b5-0d3e-4577-b101-2cc7d75283cb-service-ca-bundle\") pod \"router-default-7f579888d9-ntt6m\" (UID: \"898bd0b5-0d3e-4577-b101-2cc7d75283cb\") " pod="openshift-ingress/router-default-7f579888d9-ntt6m" Nov 03 13:08:48.890885 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:48.890862 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/898bd0b5-0d3e-4577-b101-2cc7d75283cb-service-ca-bundle\") pod \"router-default-7f579888d9-ntt6m\" (UID: \"898bd0b5-0d3e-4577-b101-2cc7d75283cb\") " pod="openshift-ingress/router-default-7f579888d9-ntt6m" Nov 03 13:08:49.179632 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:49.179535 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-7f579888d9-ntt6m" Nov 03 13:08:49.456856 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:49.456782 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-7f579888d9-ntt6m"] Nov 03 13:08:49.459823 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:08:49.459795 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod898bd0b5_0d3e_4577_b101_2cc7d75283cb.slice/crio-08a08dc5764eb8ea9052374d860f86729bce31446fe725c6884c7d658d522f5b WatchSource:0}: Error finding container 08a08dc5764eb8ea9052374d860f86729bce31446fe725c6884c7d658d522f5b: Status 404 returned error can't find the container with id 08a08dc5764eb8ea9052374d860f86729bce31446fe725c6884c7d658d522f5b Nov 03 13:08:50.242041 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:50.242002 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-7f579888d9-ntt6m" event={"ID":"898bd0b5-0d3e-4577-b101-2cc7d75283cb","Type":"ContainerStarted","Data":"9d49479da21e51fdb1fbea6d68318a548c61fbe4aa59071dc481ca83be24ddfa"} Nov 03 13:08:50.242549 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:50.242057 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-7f579888d9-ntt6m" event={"ID":"898bd0b5-0d3e-4577-b101-2cc7d75283cb","Type":"ContainerStarted","Data":"08a08dc5764eb8ea9052374d860f86729bce31446fe725c6884c7d658d522f5b"} Nov 03 13:08:50.704101 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:50.704064 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-555f8976d-mz5ct" Nov 03 13:08:51.180091 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:51.180054 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-7f579888d9-ntt6m" Nov 03 13:08:51.182428 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:51.182404 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-7f579888d9-ntt6m" Nov 03 13:08:51.419020 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:51.418985 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-7f579888d9-ntt6m" Nov 03 13:08:51.420351 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:51.420330 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-7f579888d9-ntt6m" Nov 03 13:08:53.419647 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:53.419617 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-555f8976d-mz5ct"] Nov 03 13:08:55.221202 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:55.221172 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-vgnv8" Nov 03 13:08:55.292107 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:55.292075 5024 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-vgnv8_3fdeb9f3-6c67-4390-8073-17d4d5f802b9/dns/0.log" Nov 03 13:08:55.478895 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:55.478822 5024 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-vgnv8_3fdeb9f3-6c67-4390-8073-17d4d5f802b9/kube-rbac-proxy/0.log" Nov 03 13:08:56.076990 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:56.076954 5024 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-v8gcq_5ea5cf61-ed1c-4f5e-b1a0-5c46c5c3ee18/dns-node-resolver/0.log" Nov 03 13:08:56.284442 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:56.284406 5024 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-555f8976d-mz5ct_cc8ef30e-6d4f-4913-8dc8-b09e5aea2f82/registry/0.log" Nov 03 13:08:56.675193 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:56.675144 5024 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-7dbb67bd57-74qfj_0a892eb3-ecc4-4312-9f4d-73eeba6b6eee/registry/0.log" Nov 03 13:08:56.874608 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:56.874579 5024 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-4tn4j_39e11989-e598-4cea-87d9-26b3cf78a66b/node-ca/0.log" Nov 03 13:08:57.474729 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:57.474690 5024 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7f579888d9-ntt6m_898bd0b5-0d3e-4577-b101-2cc7d75283cb/router/0.log" Nov 03 13:08:57.674457 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:57.674428 5024 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-d8f6l_50d68cb7-b766-4674-b295-8f3229880994/serve-healthcheck-canary/0.log" Nov 03 13:08:58.673077 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:08:58.673043 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-8rdgq" Nov 03 13:09:00.493931 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:09:00.493897 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-r9pzz"] Nov 03 13:09:00.497051 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:09:00.497029 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-r9pzz" Nov 03 13:09:00.577221 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:09:00.577193 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-pjbx2\"" Nov 03 13:09:00.577423 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:09:00.577403 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Nov 03 13:09:00.577527 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:09:00.577508 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Nov 03 13:09:00.587856 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:09:00.587827 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/dbe43d42-0886-402c-bc38-080acd180803-node-exporter-textfile\") pod \"node-exporter-r9pzz\" (UID: \"dbe43d42-0886-402c-bc38-080acd180803\") " pod="openshift-monitoring/node-exporter-r9pzz" Nov 03 13:09:00.587970 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:09:00.587857 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/dbe43d42-0886-402c-bc38-080acd180803-node-exporter-tls\") pod \"node-exporter-r9pzz\" (UID: \"dbe43d42-0886-402c-bc38-080acd180803\") " pod="openshift-monitoring/node-exporter-r9pzz" Nov 03 13:09:00.587970 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:09:00.587883 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-95vg9\" (UniqueName: \"kubernetes.io/projected/dbe43d42-0886-402c-bc38-080acd180803-kube-api-access-95vg9\") pod \"node-exporter-r9pzz\" (UID: \"dbe43d42-0886-402c-bc38-080acd180803\") " pod="openshift-monitoring/node-exporter-r9pzz" Nov 03 13:09:00.587970 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:09:00.587950 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/dbe43d42-0886-402c-bc38-080acd180803-root\") pod \"node-exporter-r9pzz\" (UID: \"dbe43d42-0886-402c-bc38-080acd180803\") " pod="openshift-monitoring/node-exporter-r9pzz" Nov 03 13:09:00.588063 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:09:00.587991 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/dbe43d42-0886-402c-bc38-080acd180803-node-exporter-accelerators-collector-config\") pod \"node-exporter-r9pzz\" (UID: \"dbe43d42-0886-402c-bc38-080acd180803\") " pod="openshift-monitoring/node-exporter-r9pzz" Nov 03 13:09:00.588099 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:09:00.588065 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/dbe43d42-0886-402c-bc38-080acd180803-node-exporter-wtmp\") pod \"node-exporter-r9pzz\" (UID: \"dbe43d42-0886-402c-bc38-080acd180803\") " pod="openshift-monitoring/node-exporter-r9pzz" Nov 03 13:09:00.588131 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:09:00.588103 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/dbe43d42-0886-402c-bc38-080acd180803-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-r9pzz\" (UID: \"dbe43d42-0886-402c-bc38-080acd180803\") " pod="openshift-monitoring/node-exporter-r9pzz" Nov 03 13:09:00.588186 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:09:00.588170 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dbe43d42-0886-402c-bc38-080acd180803-metrics-client-ca\") pod \"node-exporter-r9pzz\" (UID: \"dbe43d42-0886-402c-bc38-080acd180803\") " pod="openshift-monitoring/node-exporter-r9pzz" Nov 03 13:09:00.588231 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:09:00.588216 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/dbe43d42-0886-402c-bc38-080acd180803-sys\") pod \"node-exporter-r9pzz\" (UID: \"dbe43d42-0886-402c-bc38-080acd180803\") " pod="openshift-monitoring/node-exporter-r9pzz" Nov 03 13:09:00.590082 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:09:00.590065 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Nov 03 13:09:00.590145 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:09:00.590133 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Nov 03 13:09:00.688620 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:09:00.688587 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/dbe43d42-0886-402c-bc38-080acd180803-node-exporter-wtmp\") pod \"node-exporter-r9pzz\" (UID: \"dbe43d42-0886-402c-bc38-080acd180803\") " pod="openshift-monitoring/node-exporter-r9pzz" Nov 03 13:09:00.688775 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:09:00.688635 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/dbe43d42-0886-402c-bc38-080acd180803-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-r9pzz\" (UID: \"dbe43d42-0886-402c-bc38-080acd180803\") " pod="openshift-monitoring/node-exporter-r9pzz" Nov 03 13:09:00.688775 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:09:00.688652 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dbe43d42-0886-402c-bc38-080acd180803-metrics-client-ca\") pod \"node-exporter-r9pzz\" (UID: \"dbe43d42-0886-402c-bc38-080acd180803\") " pod="openshift-monitoring/node-exporter-r9pzz" Nov 03 13:09:00.688775 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:09:00.688680 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/dbe43d42-0886-402c-bc38-080acd180803-sys\") pod \"node-exporter-r9pzz\" (UID: \"dbe43d42-0886-402c-bc38-080acd180803\") " pod="openshift-monitoring/node-exporter-r9pzz" Nov 03 13:09:00.688775 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:09:00.688707 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/dbe43d42-0886-402c-bc38-080acd180803-node-exporter-textfile\") pod \"node-exporter-r9pzz\" (UID: \"dbe43d42-0886-402c-bc38-080acd180803\") " pod="openshift-monitoring/node-exporter-r9pzz" Nov 03 13:09:00.688775 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:09:00.688721 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/dbe43d42-0886-402c-bc38-080acd180803-node-exporter-tls\") pod \"node-exporter-r9pzz\" (UID: \"dbe43d42-0886-402c-bc38-080acd180803\") " pod="openshift-monitoring/node-exporter-r9pzz" Nov 03 13:09:00.688775 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:09:00.688750 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-95vg9\" (UniqueName: \"kubernetes.io/projected/dbe43d42-0886-402c-bc38-080acd180803-kube-api-access-95vg9\") pod \"node-exporter-r9pzz\" (UID: \"dbe43d42-0886-402c-bc38-080acd180803\") " pod="openshift-monitoring/node-exporter-r9pzz" Nov 03 13:09:00.688775 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:09:00.688773 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/dbe43d42-0886-402c-bc38-080acd180803-root\") pod \"node-exporter-r9pzz\" (UID: \"dbe43d42-0886-402c-bc38-080acd180803\") " pod="openshift-monitoring/node-exporter-r9pzz" Nov 03 13:09:00.689136 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:09:00.688791 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/dbe43d42-0886-402c-bc38-080acd180803-sys\") pod \"node-exporter-r9pzz\" (UID: \"dbe43d42-0886-402c-bc38-080acd180803\") " pod="openshift-monitoring/node-exporter-r9pzz" Nov 03 13:09:00.689136 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:09:00.688793 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/dbe43d42-0886-402c-bc38-080acd180803-node-exporter-wtmp\") pod \"node-exporter-r9pzz\" (UID: \"dbe43d42-0886-402c-bc38-080acd180803\") " pod="openshift-monitoring/node-exporter-r9pzz" Nov 03 13:09:00.689136 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:09:00.688802 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/dbe43d42-0886-402c-bc38-080acd180803-node-exporter-accelerators-collector-config\") pod \"node-exporter-r9pzz\" (UID: \"dbe43d42-0886-402c-bc38-080acd180803\") " pod="openshift-monitoring/node-exporter-r9pzz" Nov 03 13:09:00.689136 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:09:00.688840 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/dbe43d42-0886-402c-bc38-080acd180803-root\") pod \"node-exporter-r9pzz\" (UID: \"dbe43d42-0886-402c-bc38-080acd180803\") " pod="openshift-monitoring/node-exporter-r9pzz" Nov 03 13:09:00.689319 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:09:00.689224 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/dbe43d42-0886-402c-bc38-080acd180803-node-exporter-textfile\") pod \"node-exporter-r9pzz\" (UID: \"dbe43d42-0886-402c-bc38-080acd180803\") " pod="openshift-monitoring/node-exporter-r9pzz" Nov 03 13:09:00.689479 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:09:00.689450 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dbe43d42-0886-402c-bc38-080acd180803-metrics-client-ca\") pod \"node-exporter-r9pzz\" (UID: \"dbe43d42-0886-402c-bc38-080acd180803\") " pod="openshift-monitoring/node-exporter-r9pzz" Nov 03 13:09:00.689654 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:09:00.689487 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/dbe43d42-0886-402c-bc38-080acd180803-node-exporter-accelerators-collector-config\") pod \"node-exporter-r9pzz\" (UID: \"dbe43d42-0886-402c-bc38-080acd180803\") " pod="openshift-monitoring/node-exporter-r9pzz" Nov 03 13:09:00.691130 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:09:00.691106 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/dbe43d42-0886-402c-bc38-080acd180803-node-exporter-tls\") pod \"node-exporter-r9pzz\" (UID: \"dbe43d42-0886-402c-bc38-080acd180803\") " pod="openshift-monitoring/node-exporter-r9pzz" Nov 03 13:09:00.691210 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:09:00.691140 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/dbe43d42-0886-402c-bc38-080acd180803-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-r9pzz\" (UID: \"dbe43d42-0886-402c-bc38-080acd180803\") " pod="openshift-monitoring/node-exporter-r9pzz" Nov 03 13:09:00.698261 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:09:00.698241 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-95vg9\" (UniqueName: \"kubernetes.io/projected/dbe43d42-0886-402c-bc38-080acd180803-kube-api-access-95vg9\") pod \"node-exporter-r9pzz\" (UID: \"dbe43d42-0886-402c-bc38-080acd180803\") " pod="openshift-monitoring/node-exporter-r9pzz" Nov 03 13:09:00.810478 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:09:00.810394 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-r9pzz" Nov 03 13:09:00.821356 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:09:00.821328 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbe43d42_0886_402c_bc38_080acd180803.slice/crio-d0281f531b4c193996f91b36d729f396bf31c4d58ea90105ef461d578674a7b6 WatchSource:0}: Error finding container d0281f531b4c193996f91b36d729f396bf31c4d58ea90105ef461d578674a7b6: Status 404 returned error can't find the container with id d0281f531b4c193996f91b36d729f396bf31c4d58ea90105ef461d578674a7b6 Nov 03 13:09:01.286747 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:09:01.286715 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-r9pzz" event={"ID":"dbe43d42-0886-402c-bc38-080acd180803","Type":"ContainerStarted","Data":"d0281f531b4c193996f91b36d729f396bf31c4d58ea90105ef461d578674a7b6"} Nov 03 13:09:02.293612 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:09:02.293574 5024 generic.go:356] "Generic (PLEG): container finished" podID="dbe43d42-0886-402c-bc38-080acd180803" containerID="3eceb32ff9a578a6ce379fa01135f43312f6d83e21dec887995e759b2c782a61" exitCode=0 Nov 03 13:09:02.294015 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:09:02.293658 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-r9pzz" event={"ID":"dbe43d42-0886-402c-bc38-080acd180803","Type":"ContainerDied","Data":"3eceb32ff9a578a6ce379fa01135f43312f6d83e21dec887995e759b2c782a61"} Nov 03 13:09:03.300125 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:09:03.300087 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-r9pzz" event={"ID":"dbe43d42-0886-402c-bc38-080acd180803","Type":"ContainerStarted","Data":"11cb263eb7cdc5a5066a2a4421571d9fbc2c556a52e4011c9293e4c3575b2a9e"} Nov 03 13:09:03.300579 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:09:03.300149 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-r9pzz" event={"ID":"dbe43d42-0886-402c-bc38-080acd180803","Type":"ContainerStarted","Data":"f9469615d59815e13936b9d0dd73e2fedcf070d8aa7869a377eed966a304c60f"} Nov 03 13:09:03.326298 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:09:03.326246 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-r9pzz" podStartSLOduration=2.28660207 podStartE2EDuration="3.326232713s" totalImagesPullingTime="974.022861ms" totalInitContainerRuntime="65.607782ms" isStatefulPod=true podCreationTimestamp="2026-11-03 13:09:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:09:03.324206541 +0000 UTC m=+23.332397819" watchObservedRunningTime="2026-11-03 13:09:03.326232713 +0000 UTC m=+23.334423991" Nov 03 13:09:03.425129 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:09:03.425086 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-555f8976d-mz5ct" Nov 03 13:09:04.870572 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:09:04.870528 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-86b957855b-5k7lz"] Nov 03 13:09:04.873431 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:09:04.873410 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-86b957855b-5k7lz" Nov 03 13:09:04.880032 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:09:04.880009 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Nov 03 13:09:04.881719 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:09:04.881565 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-86b957855b-5k7lz"] Nov 03 13:09:04.882115 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:09:04.882096 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Nov 03 13:09:04.911636 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:09:04.911607 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Nov 03 13:09:04.911808 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:09:04.911684 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Nov 03 13:09:04.911808 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:09:04.911728 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-pnrx8\"" Nov 03 13:09:04.911808 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:09:04.911764 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-9ve803o20m5de\"" Nov 03 13:09:05.029587 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:09:05.029545 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/bddb955a-a11f-4cb4-b25e-ef2ed77c98ba-audit-log\") pod \"metrics-server-86b957855b-5k7lz\" (UID: \"bddb955a-a11f-4cb4-b25e-ef2ed77c98ba\") " pod="openshift-monitoring/metrics-server-86b957855b-5k7lz" Nov 03 13:09:05.029756 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:09:05.029620 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/bddb955a-a11f-4cb4-b25e-ef2ed77c98ba-secret-metrics-server-tls\") pod \"metrics-server-86b957855b-5k7lz\" (UID: \"bddb955a-a11f-4cb4-b25e-ef2ed77c98ba\") " pod="openshift-monitoring/metrics-server-86b957855b-5k7lz" Nov 03 13:09:05.029756 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:09:05.029710 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/bddb955a-a11f-4cb4-b25e-ef2ed77c98ba-secret-metrics-server-client-certs\") pod \"metrics-server-86b957855b-5k7lz\" (UID: \"bddb955a-a11f-4cb4-b25e-ef2ed77c98ba\") " pod="openshift-monitoring/metrics-server-86b957855b-5k7lz" Nov 03 13:09:05.029756 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:09:05.029742 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bddb955a-a11f-4cb4-b25e-ef2ed77c98ba-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-86b957855b-5k7lz\" (UID: \"bddb955a-a11f-4cb4-b25e-ef2ed77c98ba\") " pod="openshift-monitoring/metrics-server-86b957855b-5k7lz" Nov 03 13:09:05.029946 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:09:05.029776 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/bddb955a-a11f-4cb4-b25e-ef2ed77c98ba-metrics-server-audit-profiles\") pod \"metrics-server-86b957855b-5k7lz\" (UID: \"bddb955a-a11f-4cb4-b25e-ef2ed77c98ba\") " pod="openshift-monitoring/metrics-server-86b957855b-5k7lz" Nov 03 13:09:05.029946 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:09:05.029821 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/bddb955a-a11f-4cb4-b25e-ef2ed77c98ba-client-ca-bundle\") pod \"metrics-server-86b957855b-5k7lz\" (UID: \"bddb955a-a11f-4cb4-b25e-ef2ed77c98ba\") " pod="openshift-monitoring/metrics-server-86b957855b-5k7lz" Nov 03 13:09:05.029946 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:09:05.029861 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xnq4\" (UniqueName: \"kubernetes.io/projected/bddb955a-a11f-4cb4-b25e-ef2ed77c98ba-kube-api-access-6xnq4\") pod \"metrics-server-86b957855b-5k7lz\" (UID: \"bddb955a-a11f-4cb4-b25e-ef2ed77c98ba\") " pod="openshift-monitoring/metrics-server-86b957855b-5k7lz" Nov 03 13:09:05.130507 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:09:05.130435 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/bddb955a-a11f-4cb4-b25e-ef2ed77c98ba-audit-log\") pod \"metrics-server-86b957855b-5k7lz\" (UID: \"bddb955a-a11f-4cb4-b25e-ef2ed77c98ba\") " pod="openshift-monitoring/metrics-server-86b957855b-5k7lz" Nov 03 13:09:05.130507 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:09:05.130485 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/bddb955a-a11f-4cb4-b25e-ef2ed77c98ba-secret-metrics-server-tls\") pod \"metrics-server-86b957855b-5k7lz\" (UID: \"bddb955a-a11f-4cb4-b25e-ef2ed77c98ba\") " pod="openshift-monitoring/metrics-server-86b957855b-5k7lz" Nov 03 13:09:05.130697 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:09:05.130604 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/bddb955a-a11f-4cb4-b25e-ef2ed77c98ba-secret-metrics-server-client-certs\") pod \"metrics-server-86b957855b-5k7lz\" (UID: \"bddb955a-a11f-4cb4-b25e-ef2ed77c98ba\") " pod="openshift-monitoring/metrics-server-86b957855b-5k7lz" Nov 03 13:09:05.130697 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:09:05.130643 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bddb955a-a11f-4cb4-b25e-ef2ed77c98ba-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-86b957855b-5k7lz\" (UID: \"bddb955a-a11f-4cb4-b25e-ef2ed77c98ba\") " pod="openshift-monitoring/metrics-server-86b957855b-5k7lz" Nov 03 13:09:05.130697 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:09:05.130675 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/bddb955a-a11f-4cb4-b25e-ef2ed77c98ba-metrics-server-audit-profiles\") pod \"metrics-server-86b957855b-5k7lz\" (UID: \"bddb955a-a11f-4cb4-b25e-ef2ed77c98ba\") " pod="openshift-monitoring/metrics-server-86b957855b-5k7lz" Nov 03 13:09:05.130845 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:09:05.130698 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/bddb955a-a11f-4cb4-b25e-ef2ed77c98ba-client-ca-bundle\") pod \"metrics-server-86b957855b-5k7lz\" (UID: \"bddb955a-a11f-4cb4-b25e-ef2ed77c98ba\") " pod="openshift-monitoring/metrics-server-86b957855b-5k7lz" Nov 03 13:09:05.130845 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:09:05.130723 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xnq4\" (UniqueName: \"kubernetes.io/projected/bddb955a-a11f-4cb4-b25e-ef2ed77c98ba-kube-api-access-6xnq4\") pod \"metrics-server-86b957855b-5k7lz\" (UID: \"bddb955a-a11f-4cb4-b25e-ef2ed77c98ba\") " pod="openshift-monitoring/metrics-server-86b957855b-5k7lz" Nov 03 13:09:05.130994 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:09:05.130966 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/bddb955a-a11f-4cb4-b25e-ef2ed77c98ba-audit-log\") pod \"metrics-server-86b957855b-5k7lz\" (UID: \"bddb955a-a11f-4cb4-b25e-ef2ed77c98ba\") " pod="openshift-monitoring/metrics-server-86b957855b-5k7lz" Nov 03 13:09:05.131449 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:09:05.131427 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bddb955a-a11f-4cb4-b25e-ef2ed77c98ba-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-86b957855b-5k7lz\" (UID: \"bddb955a-a11f-4cb4-b25e-ef2ed77c98ba\") " pod="openshift-monitoring/metrics-server-86b957855b-5k7lz" Nov 03 13:09:05.131855 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:09:05.131829 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/bddb955a-a11f-4cb4-b25e-ef2ed77c98ba-metrics-server-audit-profiles\") pod \"metrics-server-86b957855b-5k7lz\" (UID: \"bddb955a-a11f-4cb4-b25e-ef2ed77c98ba\") " pod="openshift-monitoring/metrics-server-86b957855b-5k7lz" Nov 03 13:09:05.133199 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:09:05.133179 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/bddb955a-a11f-4cb4-b25e-ef2ed77c98ba-secret-metrics-server-client-certs\") pod \"metrics-server-86b957855b-5k7lz\" (UID: \"bddb955a-a11f-4cb4-b25e-ef2ed77c98ba\") " pod="openshift-monitoring/metrics-server-86b957855b-5k7lz" Nov 03 13:09:05.133377 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:09:05.133355 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/bddb955a-a11f-4cb4-b25e-ef2ed77c98ba-secret-metrics-server-tls\") pod \"metrics-server-86b957855b-5k7lz\" (UID: \"bddb955a-a11f-4cb4-b25e-ef2ed77c98ba\") " pod="openshift-monitoring/metrics-server-86b957855b-5k7lz" Nov 03 13:09:05.133560 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:09:05.133544 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/bddb955a-a11f-4cb4-b25e-ef2ed77c98ba-client-ca-bundle\") pod \"metrics-server-86b957855b-5k7lz\" (UID: \"bddb955a-a11f-4cb4-b25e-ef2ed77c98ba\") " pod="openshift-monitoring/metrics-server-86b957855b-5k7lz" Nov 03 13:09:05.146128 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:09:05.146105 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xnq4\" (UniqueName: \"kubernetes.io/projected/bddb955a-a11f-4cb4-b25e-ef2ed77c98ba-kube-api-access-6xnq4\") pod \"metrics-server-86b957855b-5k7lz\" (UID: \"bddb955a-a11f-4cb4-b25e-ef2ed77c98ba\") " pod="openshift-monitoring/metrics-server-86b957855b-5k7lz" Nov 03 13:09:05.186071 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:09:05.186028 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-86b957855b-5k7lz" Nov 03 13:09:05.482042 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:09:05.482016 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-86b957855b-5k7lz"] Nov 03 13:09:05.484587 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:09:05.484552 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbddb955a_a11f_4cb4_b25e_ef2ed77c98ba.slice/crio-9292995c81f22896826617f94be4cec27ccdc89f5f34405fc704481327d392d8 WatchSource:0}: Error finding container 9292995c81f22896826617f94be4cec27ccdc89f5f34405fc704481327d392d8: Status 404 returned error can't find the container with id 9292995c81f22896826617f94be4cec27ccdc89f5f34405fc704481327d392d8 Nov 03 13:09:06.225622 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:09:06.225590 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-7dbb67bd57-74qfj" Nov 03 13:09:06.314414 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:09:06.314377 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-86b957855b-5k7lz" event={"ID":"bddb955a-a11f-4cb4-b25e-ef2ed77c98ba","Type":"ContainerStarted","Data":"9292995c81f22896826617f94be4cec27ccdc89f5f34405fc704481327d392d8"} Nov 03 13:09:07.320697 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:09:07.320652 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-86b957855b-5k7lz" event={"ID":"bddb955a-a11f-4cb4-b25e-ef2ed77c98ba","Type":"ContainerStarted","Data":"d0ada87f73d694bc8a0a68817ce09d16faa7346c7ba2a0ca04b15d9963a938f7"} Nov 03 13:09:07.342238 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:09:07.342185 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-86b957855b-5k7lz" podStartSLOduration=2.030842457 podStartE2EDuration="3.3421716s" totalImagesPullingTime="1.311329143s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:09:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:09:07.341346018 +0000 UTC m=+27.349537294" watchObservedRunningTime="2026-11-03 13:09:07.3421716 +0000 UTC m=+27.350362877" Nov 03 13:09:09.506322 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:09:09.506216 5024 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d7mwz" containerName="extractor" Nov 03 13:09:10.807938 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:09:10.807897 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-bdg96" Nov 03 13:09:15.509916 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:09:15.509145 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7dbb67bd57-74qfj"] Nov 03 13:09:18.440467 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:09:18.440393 5024 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-555f8976d-mz5ct" podUID="cc8ef30e-6d4f-4913-8dc8-b09e5aea2f82" containerName="registry" containerID="cri-o://7c251d5c4ec97f54c433212e0c2983af902326bb7d9b279efbdbcae45f1e0acb" gracePeriod=30 Nov 03 13:09:18.714264 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:09:18.714236 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-555f8976d-mz5ct" Nov 03 13:09:18.744592 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:09:18.744558 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cc8ef30e-6d4f-4913-8dc8-b09e5aea2f82-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/cc8ef30e-6d4f-4913-8dc8-b09e5aea2f82-image-registry-private-configuration\") pod \"cc8ef30e-6d4f-4913-8dc8-b09e5aea2f82\" (UID: \"cc8ef30e-6d4f-4913-8dc8-b09e5aea2f82\") " Nov 03 13:09:18.744764 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:09:18.744601 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc8ef30e-6d4f-4913-8dc8-b09e5aea2f82-kube-api-access-h82mn\" (UniqueName: \"kubernetes.io/projected/cc8ef30e-6d4f-4913-8dc8-b09e5aea2f82-kube-api-access-h82mn\") pod \"cc8ef30e-6d4f-4913-8dc8-b09e5aea2f82\" (UID: \"cc8ef30e-6d4f-4913-8dc8-b09e5aea2f82\") " Nov 03 13:09:18.744764 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:09:18.744642 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc8ef30e-6d4f-4913-8dc8-b09e5aea2f82-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/cc8ef30e-6d4f-4913-8dc8-b09e5aea2f82-registry-certificates\") pod \"cc8ef30e-6d4f-4913-8dc8-b09e5aea2f82\" (UID: \"cc8ef30e-6d4f-4913-8dc8-b09e5aea2f82\") " Nov 03 13:09:18.744881 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:09:18.744752 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cc8ef30e-6d4f-4913-8dc8-b09e5aea2f82-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/cc8ef30e-6d4f-4913-8dc8-b09e5aea2f82-installation-pull-secrets\") pod \"cc8ef30e-6d4f-4913-8dc8-b09e5aea2f82\" (UID: \"cc8ef30e-6d4f-4913-8dc8-b09e5aea2f82\") " Nov 03 13:09:18.744881 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:09:18.744830 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc8ef30e-6d4f-4913-8dc8-b09e5aea2f82-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/cc8ef30e-6d4f-4913-8dc8-b09e5aea2f82-ca-trust-extracted\") pod \"cc8ef30e-6d4f-4913-8dc8-b09e5aea2f82\" (UID: \"cc8ef30e-6d4f-4913-8dc8-b09e5aea2f82\") " Nov 03 13:09:18.744881 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:09:18.744873 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc8ef30e-6d4f-4913-8dc8-b09e5aea2f82-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/cc8ef30e-6d4f-4913-8dc8-b09e5aea2f82-trusted-ca\") pod \"cc8ef30e-6d4f-4913-8dc8-b09e5aea2f82\" (UID: \"cc8ef30e-6d4f-4913-8dc8-b09e5aea2f82\") " Nov 03 13:09:18.745159 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:09:18.744925 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc8ef30e-6d4f-4913-8dc8-b09e5aea2f82-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cc8ef30e-6d4f-4913-8dc8-b09e5aea2f82-bound-sa-token\") pod \"cc8ef30e-6d4f-4913-8dc8-b09e5aea2f82\" (UID: \"cc8ef30e-6d4f-4913-8dc8-b09e5aea2f82\") " Nov 03 13:09:18.745159 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:09:18.744976 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc8ef30e-6d4f-4913-8dc8-b09e5aea2f82-registry-tls\" (UniqueName: \"kubernetes.io/projected/cc8ef30e-6d4f-4913-8dc8-b09e5aea2f82-registry-tls\") pod \"cc8ef30e-6d4f-4913-8dc8-b09e5aea2f82\" (UID: \"cc8ef30e-6d4f-4913-8dc8-b09e5aea2f82\") " Nov 03 13:09:18.745159 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:09:18.745079 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc8ef30e-6d4f-4913-8dc8-b09e5aea2f82-registry-certificates" pod "cc8ef30e-6d4f-4913-8dc8-b09e5aea2f82" (UID: "cc8ef30e-6d4f-4913-8dc8-b09e5aea2f82"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:09:18.745431 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:09:18.745404 5024 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/cc8ef30e-6d4f-4913-8dc8-b09e5aea2f82-registry-certificates\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:09:18.745545 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:09:18.745449 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc8ef30e-6d4f-4913-8dc8-b09e5aea2f82-trusted-ca" pod "cc8ef30e-6d4f-4913-8dc8-b09e5aea2f82" (UID: "cc8ef30e-6d4f-4913-8dc8-b09e5aea2f82"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:09:18.747947 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:09:18.747919 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc8ef30e-6d4f-4913-8dc8-b09e5aea2f82-registry-tls" pod "cc8ef30e-6d4f-4913-8dc8-b09e5aea2f82" (UID: "cc8ef30e-6d4f-4913-8dc8-b09e5aea2f82"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:09:18.748126 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:09:18.748107 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc8ef30e-6d4f-4913-8dc8-b09e5aea2f82-image-registry-private-configuration" pod "cc8ef30e-6d4f-4913-8dc8-b09e5aea2f82" (UID: "cc8ef30e-6d4f-4913-8dc8-b09e5aea2f82"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 13:09:18.748126 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:09:18.748107 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc8ef30e-6d4f-4913-8dc8-b09e5aea2f82-kube-api-access-h82mn" pod "cc8ef30e-6d4f-4913-8dc8-b09e5aea2f82" (UID: "cc8ef30e-6d4f-4913-8dc8-b09e5aea2f82"). InnerVolumeSpecName "kube-api-access-h82mn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:09:18.748248 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:09:18.748154 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc8ef30e-6d4f-4913-8dc8-b09e5aea2f82-bound-sa-token" pod "cc8ef30e-6d4f-4913-8dc8-b09e5aea2f82" (UID: "cc8ef30e-6d4f-4913-8dc8-b09e5aea2f82"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:09:18.749915 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:09:18.749882 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc8ef30e-6d4f-4913-8dc8-b09e5aea2f82-installation-pull-secrets" pod "cc8ef30e-6d4f-4913-8dc8-b09e5aea2f82" (UID: "cc8ef30e-6d4f-4913-8dc8-b09e5aea2f82"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 13:09:18.753019 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:09:18.752991 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc8ef30e-6d4f-4913-8dc8-b09e5aea2f82-ca-trust-extracted" pod "cc8ef30e-6d4f-4913-8dc8-b09e5aea2f82" (UID: "cc8ef30e-6d4f-4913-8dc8-b09e5aea2f82"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:09:18.846415 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:09:18.846380 5024 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/cc8ef30e-6d4f-4913-8dc8-b09e5aea2f82-image-registry-private-configuration\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:09:18.846415 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:09:18.846413 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h82mn\" (UniqueName: \"kubernetes.io/projected/cc8ef30e-6d4f-4913-8dc8-b09e5aea2f82-kube-api-access-h82mn\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:09:18.846613 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:09:18.846426 5024 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/cc8ef30e-6d4f-4913-8dc8-b09e5aea2f82-installation-pull-secrets\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:09:18.846613 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:09:18.846435 5024 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/cc8ef30e-6d4f-4913-8dc8-b09e5aea2f82-ca-trust-extracted\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:09:18.846613 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:09:18.846446 5024 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/cc8ef30e-6d4f-4913-8dc8-b09e5aea2f82-trusted-ca\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:09:18.846613 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:09:18.846455 5024 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cc8ef30e-6d4f-4913-8dc8-b09e5aea2f82-bound-sa-token\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:09:18.846613 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:09:18.846463 5024 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cc8ef30e-6d4f-4913-8dc8-b09e5aea2f82-registry-tls\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:09:19.369894 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:09:19.369855 5024 generic.go:356] "Generic (PLEG): container finished" podID="cc8ef30e-6d4f-4913-8dc8-b09e5aea2f82" containerID="7c251d5c4ec97f54c433212e0c2983af902326bb7d9b279efbdbcae45f1e0acb" exitCode=0 Nov 03 13:09:19.370093 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:09:19.369902 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-555f8976d-mz5ct" Nov 03 13:09:19.370093 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:09:19.369905 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-555f8976d-mz5ct" event={"ID":"cc8ef30e-6d4f-4913-8dc8-b09e5aea2f82","Type":"ContainerDied","Data":"7c251d5c4ec97f54c433212e0c2983af902326bb7d9b279efbdbcae45f1e0acb"} Nov 03 13:09:19.370093 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:09:19.369958 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-555f8976d-mz5ct" event={"ID":"cc8ef30e-6d4f-4913-8dc8-b09e5aea2f82","Type":"ContainerDied","Data":"4278f1b6983e7acc4d4d3d30127f25a11e647c906157a6127809999b409890a2"} Nov 03 13:09:19.370093 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:09:19.369983 5024 scope.go:122] "RemoveContainer" containerID="7c251d5c4ec97f54c433212e0c2983af902326bb7d9b279efbdbcae45f1e0acb" Nov 03 13:09:19.379002 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:09:19.378978 5024 scope.go:122] "RemoveContainer" containerID="7c251d5c4ec97f54c433212e0c2983af902326bb7d9b279efbdbcae45f1e0acb" Nov 03 13:09:19.379249 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:09:19.379226 5024 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c251d5c4ec97f54c433212e0c2983af902326bb7d9b279efbdbcae45f1e0acb\": container with ID starting with 7c251d5c4ec97f54c433212e0c2983af902326bb7d9b279efbdbcae45f1e0acb not found: ID does not exist" containerID="7c251d5c4ec97f54c433212e0c2983af902326bb7d9b279efbdbcae45f1e0acb" Nov 03 13:09:19.379342 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:09:19.379258 5024 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c251d5c4ec97f54c433212e0c2983af902326bb7d9b279efbdbcae45f1e0acb"} err="failed to get container status \"7c251d5c4ec97f54c433212e0c2983af902326bb7d9b279efbdbcae45f1e0acb\": rpc error: code = NotFound desc = could not find container \"7c251d5c4ec97f54c433212e0c2983af902326bb7d9b279efbdbcae45f1e0acb\": container with ID starting with 7c251d5c4ec97f54c433212e0c2983af902326bb7d9b279efbdbcae45f1e0acb not found: ID does not exist" Nov 03 13:09:19.393030 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:09:19.393005 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-555f8976d-mz5ct"] Nov 03 13:09:19.396557 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:09:19.396534 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-555f8976d-mz5ct"] Nov 03 13:09:20.105725 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:09:20.105690 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc8ef30e-6d4f-4913-8dc8-b09e5aea2f82" path="/var/lib/kubelet/pods/cc8ef30e-6d4f-4913-8dc8-b09e5aea2f82/volumes" Nov 03 13:09:25.186355 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:09:25.186313 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-86b957855b-5k7lz" Nov 03 13:09:25.186759 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:09:25.186397 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-86b957855b-5k7lz" Nov 03 13:09:39.506817 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:09:39.506777 5024 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d7mwz" containerName="extractor" Nov 03 13:09:40.531089 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:09:40.531017 5024 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-7dbb67bd57-74qfj" podUID="0a892eb3-ecc4-4312-9f4d-73eeba6b6eee" containerName="registry" containerID="cri-o://e05f253dbd0d764b33d094ec44e84d449c11ff3d57a84848cc2970460a1f1c35" gracePeriod=30 Nov 03 13:09:40.774734 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:09:40.774707 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7dbb67bd57-74qfj" Nov 03 13:09:40.830761 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:09:40.830689 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a892eb3-ecc4-4312-9f4d-73eeba6b6eee-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0a892eb3-ecc4-4312-9f4d-73eeba6b6eee-bound-sa-token\") pod \"0a892eb3-ecc4-4312-9f4d-73eeba6b6eee\" (UID: \"0a892eb3-ecc4-4312-9f4d-73eeba6b6eee\") " Nov 03 13:09:40.830761 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:09:40.830732 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a892eb3-ecc4-4312-9f4d-73eeba6b6eee-registry-tls\" (UniqueName: \"kubernetes.io/projected/0a892eb3-ecc4-4312-9f4d-73eeba6b6eee-registry-tls\") pod \"0a892eb3-ecc4-4312-9f4d-73eeba6b6eee\" (UID: \"0a892eb3-ecc4-4312-9f4d-73eeba6b6eee\") " Nov 03 13:09:40.830978 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:09:40.830763 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a892eb3-ecc4-4312-9f4d-73eeba6b6eee-kube-api-access-xmbd7\" (UniqueName: \"kubernetes.io/projected/0a892eb3-ecc4-4312-9f4d-73eeba6b6eee-kube-api-access-xmbd7\") pod \"0a892eb3-ecc4-4312-9f4d-73eeba6b6eee\" (UID: \"0a892eb3-ecc4-4312-9f4d-73eeba6b6eee\") " Nov 03 13:09:40.830978 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:09:40.830797 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0a892eb3-ecc4-4312-9f4d-73eeba6b6eee-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0a892eb3-ecc4-4312-9f4d-73eeba6b6eee-image-registry-private-configuration\") pod \"0a892eb3-ecc4-4312-9f4d-73eeba6b6eee\" (UID: \"0a892eb3-ecc4-4312-9f4d-73eeba6b6eee\") " Nov 03 13:09:40.830978 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:09:40.830831 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a892eb3-ecc4-4312-9f4d-73eeba6b6eee-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0a892eb3-ecc4-4312-9f4d-73eeba6b6eee-trusted-ca\") pod \"0a892eb3-ecc4-4312-9f4d-73eeba6b6eee\" (UID: \"0a892eb3-ecc4-4312-9f4d-73eeba6b6eee\") " Nov 03 13:09:40.830978 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:09:40.830873 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a892eb3-ecc4-4312-9f4d-73eeba6b6eee-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0a892eb3-ecc4-4312-9f4d-73eeba6b6eee-registry-certificates\") pod \"0a892eb3-ecc4-4312-9f4d-73eeba6b6eee\" (UID: \"0a892eb3-ecc4-4312-9f4d-73eeba6b6eee\") " Nov 03 13:09:40.830978 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:09:40.830944 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0a892eb3-ecc4-4312-9f4d-73eeba6b6eee-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0a892eb3-ecc4-4312-9f4d-73eeba6b6eee-installation-pull-secrets\") pod \"0a892eb3-ecc4-4312-9f4d-73eeba6b6eee\" (UID: \"0a892eb3-ecc4-4312-9f4d-73eeba6b6eee\") " Nov 03 13:09:40.831233 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:09:40.830993 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a892eb3-ecc4-4312-9f4d-73eeba6b6eee-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0a892eb3-ecc4-4312-9f4d-73eeba6b6eee-ca-trust-extracted\") pod \"0a892eb3-ecc4-4312-9f4d-73eeba6b6eee\" (UID: \"0a892eb3-ecc4-4312-9f4d-73eeba6b6eee\") " Nov 03 13:09:40.831572 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:09:40.831491 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a892eb3-ecc4-4312-9f4d-73eeba6b6eee-registry-certificates" pod "0a892eb3-ecc4-4312-9f4d-73eeba6b6eee" (UID: "0a892eb3-ecc4-4312-9f4d-73eeba6b6eee"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:09:40.832248 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:09:40.832220 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a892eb3-ecc4-4312-9f4d-73eeba6b6eee-trusted-ca" pod "0a892eb3-ecc4-4312-9f4d-73eeba6b6eee" (UID: "0a892eb3-ecc4-4312-9f4d-73eeba6b6eee"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:09:40.833757 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:09:40.833645 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a892eb3-ecc4-4312-9f4d-73eeba6b6eee-kube-api-access-xmbd7" pod "0a892eb3-ecc4-4312-9f4d-73eeba6b6eee" (UID: "0a892eb3-ecc4-4312-9f4d-73eeba6b6eee"). InnerVolumeSpecName "kube-api-access-xmbd7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:09:40.833757 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:09:40.833670 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a892eb3-ecc4-4312-9f4d-73eeba6b6eee-image-registry-private-configuration" pod "0a892eb3-ecc4-4312-9f4d-73eeba6b6eee" (UID: "0a892eb3-ecc4-4312-9f4d-73eeba6b6eee"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 13:09:40.833757 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:09:40.833694 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a892eb3-ecc4-4312-9f4d-73eeba6b6eee-bound-sa-token" pod "0a892eb3-ecc4-4312-9f4d-73eeba6b6eee" (UID: "0a892eb3-ecc4-4312-9f4d-73eeba6b6eee"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:09:40.833757 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:09:40.833723 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a892eb3-ecc4-4312-9f4d-73eeba6b6eee-registry-tls" pod "0a892eb3-ecc4-4312-9f4d-73eeba6b6eee" (UID: "0a892eb3-ecc4-4312-9f4d-73eeba6b6eee"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:09:40.833757 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:09:40.833734 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a892eb3-ecc4-4312-9f4d-73eeba6b6eee-installation-pull-secrets" pod "0a892eb3-ecc4-4312-9f4d-73eeba6b6eee" (UID: "0a892eb3-ecc4-4312-9f4d-73eeba6b6eee"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 13:09:40.838319 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:09:40.838257 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a892eb3-ecc4-4312-9f4d-73eeba6b6eee-ca-trust-extracted" pod "0a892eb3-ecc4-4312-9f4d-73eeba6b6eee" (UID: "0a892eb3-ecc4-4312-9f4d-73eeba6b6eee"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:09:40.932169 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:09:40.932136 5024 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0a892eb3-ecc4-4312-9f4d-73eeba6b6eee-ca-trust-extracted\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:09:40.932169 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:09:40.932166 5024 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0a892eb3-ecc4-4312-9f4d-73eeba6b6eee-bound-sa-token\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:09:40.932169 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:09:40.932175 5024 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0a892eb3-ecc4-4312-9f4d-73eeba6b6eee-registry-tls\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:09:40.932410 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:09:40.932185 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xmbd7\" (UniqueName: \"kubernetes.io/projected/0a892eb3-ecc4-4312-9f4d-73eeba6b6eee-kube-api-access-xmbd7\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:09:40.932410 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:09:40.932199 5024 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0a892eb3-ecc4-4312-9f4d-73eeba6b6eee-image-registry-private-configuration\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:09:40.932410 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:09:40.932208 5024 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0a892eb3-ecc4-4312-9f4d-73eeba6b6eee-trusted-ca\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:09:40.932410 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:09:40.932217 5024 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0a892eb3-ecc4-4312-9f4d-73eeba6b6eee-registry-certificates\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:09:40.932410 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:09:40.932226 5024 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0a892eb3-ecc4-4312-9f4d-73eeba6b6eee-installation-pull-secrets\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:09:41.459263 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:09:41.459225 5024 generic.go:356] "Generic (PLEG): container finished" podID="0a892eb3-ecc4-4312-9f4d-73eeba6b6eee" containerID="e05f253dbd0d764b33d094ec44e84d449c11ff3d57a84848cc2970460a1f1c35" exitCode=0 Nov 03 13:09:41.459460 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:09:41.459276 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7dbb67bd57-74qfj" Nov 03 13:09:41.459460 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:09:41.459312 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7dbb67bd57-74qfj" event={"ID":"0a892eb3-ecc4-4312-9f4d-73eeba6b6eee","Type":"ContainerDied","Data":"e05f253dbd0d764b33d094ec44e84d449c11ff3d57a84848cc2970460a1f1c35"} Nov 03 13:09:41.459460 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:09:41.459363 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7dbb67bd57-74qfj" event={"ID":"0a892eb3-ecc4-4312-9f4d-73eeba6b6eee","Type":"ContainerDied","Data":"b13ded60514581be37c7f4965199d2ba2c9988e1bc8866f5a6ad43fe344af28b"} Nov 03 13:09:41.459460 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:09:41.459383 5024 scope.go:122] "RemoveContainer" containerID="e05f253dbd0d764b33d094ec44e84d449c11ff3d57a84848cc2970460a1f1c35" Nov 03 13:09:41.468148 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:09:41.468130 5024 scope.go:122] "RemoveContainer" containerID="e05f253dbd0d764b33d094ec44e84d449c11ff3d57a84848cc2970460a1f1c35" Nov 03 13:09:41.468435 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:09:41.468406 5024 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e05f253dbd0d764b33d094ec44e84d449c11ff3d57a84848cc2970460a1f1c35\": container with ID starting with e05f253dbd0d764b33d094ec44e84d449c11ff3d57a84848cc2970460a1f1c35 not found: ID does not exist" containerID="e05f253dbd0d764b33d094ec44e84d449c11ff3d57a84848cc2970460a1f1c35" Nov 03 13:09:41.468497 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:09:41.468446 5024 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e05f253dbd0d764b33d094ec44e84d449c11ff3d57a84848cc2970460a1f1c35"} err="failed to get container status \"e05f253dbd0d764b33d094ec44e84d449c11ff3d57a84848cc2970460a1f1c35\": rpc error: code = NotFound desc = could not find container \"e05f253dbd0d764b33d094ec44e84d449c11ff3d57a84848cc2970460a1f1c35\": container with ID starting with e05f253dbd0d764b33d094ec44e84d449c11ff3d57a84848cc2970460a1f1c35 not found: ID does not exist" Nov 03 13:09:41.483487 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:09:41.483461 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7dbb67bd57-74qfj"] Nov 03 13:09:41.488232 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:09:41.488210 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-7dbb67bd57-74qfj"] Nov 03 13:09:42.107304 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:09:42.107241 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a892eb3-ecc4-4312-9f4d-73eeba6b6eee" path="/var/lib/kubelet/pods/0a892eb3-ecc4-4312-9f4d-73eeba6b6eee/volumes" Nov 03 13:09:45.191593 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:09:45.191560 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-86b957855b-5k7lz" Nov 03 13:09:45.195377 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:09:45.195356 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-86b957855b-5k7lz" Nov 03 13:09:47.485521 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:09:47.485488 5024 generic.go:356] "Generic (PLEG): container finished" podID="9f20ddda-41fb-4aba-b257-7ebc59dbe678" containerID="c9f958c9f711ed0fab0f4057527197d9dfabc75f606d1cebfdabe78993cd43ca" exitCode=0 Nov 03 13:09:47.485909 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:09:47.485542 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-cqvdv" event={"ID":"9f20ddda-41fb-4aba-b257-7ebc59dbe678","Type":"ContainerDied","Data":"c9f958c9f711ed0fab0f4057527197d9dfabc75f606d1cebfdabe78993cd43ca"} Nov 03 13:09:47.485909 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:09:47.485856 5024 scope.go:122] "RemoveContainer" containerID="c9f958c9f711ed0fab0f4057527197d9dfabc75f606d1cebfdabe78993cd43ca" Nov 03 13:09:48.491135 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:09:48.491098 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-cqvdv" event={"ID":"9f20ddda-41fb-4aba-b257-7ebc59dbe678","Type":"ContainerStarted","Data":"1aa3e97351356fb4e7f90e22003f8d41eb092a089ea1baf281d7b657b830594a"} Nov 03 13:10:09.506309 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:10:09.506253 5024 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d7mwz" containerName="extractor" Nov 03 13:10:39.505456 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:10:39.505372 5024 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d7mwz" containerName="extractor" Nov 03 13:11:09.506294 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:11:09.506238 5024 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d7mwz" containerName="extractor" Nov 03 13:11:39.506132 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:11:39.506090 5024 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d7mwz" containerName="extractor" Nov 03 13:12:09.506812 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:12:09.506730 5024 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d7mwz" containerName="extractor" Nov 03 13:12:39.506847 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:12:39.506802 5024 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d7mwz" containerName="extractor" Nov 03 13:13:00.844817 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:13:00.844781 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-xjs5w"] Nov 03 13:13:00.847305 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:13:00.845065 5024 cpu_manager.go:414] "containerMap: removing container" podUID="0a892eb3-ecc4-4312-9f4d-73eeba6b6eee" containerName="registry" Nov 03 13:13:00.847305 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:13:00.845076 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a892eb3-ecc4-4312-9f4d-73eeba6b6eee" containerName="registry" Nov 03 13:13:00.847305 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:13:00.845098 5024 cpu_manager.go:414] "containerMap: removing container" podUID="cc8ef30e-6d4f-4913-8dc8-b09e5aea2f82" containerName="registry" Nov 03 13:13:00.847305 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:13:00.845103 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc8ef30e-6d4f-4913-8dc8-b09e5aea2f82" containerName="registry" Nov 03 13:13:00.847305 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:13:00.845163 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc8ef30e-6d4f-4913-8dc8-b09e5aea2f82" containerName="registry" Nov 03 13:13:00.847305 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:13:00.845175 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a892eb3-ecc4-4312-9f4d-73eeba6b6eee" containerName="registry" Nov 03 13:13:00.848211 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:13:00.848193 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-xjs5w" Nov 03 13:13:00.853305 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:13:00.853263 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Nov 03 13:13:00.853409 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:13:00.853388 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-8strr\"" Nov 03 13:13:00.857168 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:13:00.857151 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Nov 03 13:13:00.858025 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:13:00.858010 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Nov 03 13:13:00.867827 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:13:00.867802 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-xjs5w"] Nov 03 13:13:00.887125 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:13:00.887103 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Nov 03 13:13:01.032750 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:13:01.032715 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ade54df0-04b2-46bd-abc6-d66757e5edf5-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-xjs5w\" (UID: \"ade54df0-04b2-46bd-abc6-d66757e5edf5\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-xjs5w" Nov 03 13:13:01.032922 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:13:01.032759 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/ade54df0-04b2-46bd-abc6-d66757e5edf5-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-xjs5w\" (UID: \"ade54df0-04b2-46bd-abc6-d66757e5edf5\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-xjs5w" Nov 03 13:13:01.032922 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:13:01.032780 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ade54df0-04b2-46bd-abc6-d66757e5edf5-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-xjs5w\" (UID: \"ade54df0-04b2-46bd-abc6-d66757e5edf5\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-xjs5w" Nov 03 13:13:01.032922 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:13:01.032909 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/ade54df0-04b2-46bd-abc6-d66757e5edf5-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-xjs5w\" (UID: \"ade54df0-04b2-46bd-abc6-d66757e5edf5\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-xjs5w" Nov 03 13:13:01.033034 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:13:01.032942 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9dxk4\" (UniqueName: \"kubernetes.io/projected/ade54df0-04b2-46bd-abc6-d66757e5edf5-kube-api-access-9dxk4\") pod \"istiod-openshift-gateway-7876bfcb74-xjs5w\" (UID: \"ade54df0-04b2-46bd-abc6-d66757e5edf5\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-xjs5w" Nov 03 13:13:01.033034 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:13:01.032967 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/ade54df0-04b2-46bd-abc6-d66757e5edf5-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-xjs5w\" (UID: \"ade54df0-04b2-46bd-abc6-d66757e5edf5\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-xjs5w" Nov 03 13:13:01.033111 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:13:01.033070 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/ade54df0-04b2-46bd-abc6-d66757e5edf5-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-xjs5w\" (UID: \"ade54df0-04b2-46bd-abc6-d66757e5edf5\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-xjs5w" Nov 03 13:13:01.134516 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:13:01.134462 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/ade54df0-04b2-46bd-abc6-d66757e5edf5-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-xjs5w\" (UID: \"ade54df0-04b2-46bd-abc6-d66757e5edf5\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-xjs5w" Nov 03 13:13:01.134700 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:13:01.134558 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/ade54df0-04b2-46bd-abc6-d66757e5edf5-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-xjs5w\" (UID: \"ade54df0-04b2-46bd-abc6-d66757e5edf5\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-xjs5w" Nov 03 13:13:01.134700 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:13:01.134590 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ade54df0-04b2-46bd-abc6-d66757e5edf5-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-xjs5w\" (UID: \"ade54df0-04b2-46bd-abc6-d66757e5edf5\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-xjs5w" Nov 03 13:13:01.134700 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:13:01.134616 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/ade54df0-04b2-46bd-abc6-d66757e5edf5-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-xjs5w\" (UID: \"ade54df0-04b2-46bd-abc6-d66757e5edf5\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-xjs5w" Nov 03 13:13:01.134700 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:13:01.134633 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ade54df0-04b2-46bd-abc6-d66757e5edf5-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-xjs5w\" (UID: \"ade54df0-04b2-46bd-abc6-d66757e5edf5\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-xjs5w" Nov 03 13:13:01.134700 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:13:01.134671 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/ade54df0-04b2-46bd-abc6-d66757e5edf5-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-xjs5w\" (UID: \"ade54df0-04b2-46bd-abc6-d66757e5edf5\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-xjs5w" Nov 03 13:13:01.134951 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:13:01.134701 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9dxk4\" (UniqueName: \"kubernetes.io/projected/ade54df0-04b2-46bd-abc6-d66757e5edf5-kube-api-access-9dxk4\") pod \"istiod-openshift-gateway-7876bfcb74-xjs5w\" (UID: \"ade54df0-04b2-46bd-abc6-d66757e5edf5\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-xjs5w" Nov 03 13:13:01.135246 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:13:01.135212 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/ade54df0-04b2-46bd-abc6-d66757e5edf5-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-xjs5w\" (UID: \"ade54df0-04b2-46bd-abc6-d66757e5edf5\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-xjs5w" Nov 03 13:13:01.137096 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:13:01.137061 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/ade54df0-04b2-46bd-abc6-d66757e5edf5-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-xjs5w\" (UID: \"ade54df0-04b2-46bd-abc6-d66757e5edf5\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-xjs5w" Nov 03 13:13:01.137096 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:13:01.137086 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/ade54df0-04b2-46bd-abc6-d66757e5edf5-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-xjs5w\" (UID: \"ade54df0-04b2-46bd-abc6-d66757e5edf5\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-xjs5w" Nov 03 13:13:01.137228 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:13:01.137198 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ade54df0-04b2-46bd-abc6-d66757e5edf5-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-xjs5w\" (UID: \"ade54df0-04b2-46bd-abc6-d66757e5edf5\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-xjs5w" Nov 03 13:13:01.137263 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:13:01.137198 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/ade54df0-04b2-46bd-abc6-d66757e5edf5-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-xjs5w\" (UID: \"ade54df0-04b2-46bd-abc6-d66757e5edf5\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-xjs5w" Nov 03 13:13:01.144466 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:13:01.144439 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ade54df0-04b2-46bd-abc6-d66757e5edf5-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-xjs5w\" (UID: \"ade54df0-04b2-46bd-abc6-d66757e5edf5\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-xjs5w" Nov 03 13:13:01.145025 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:13:01.144998 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9dxk4\" (UniqueName: \"kubernetes.io/projected/ade54df0-04b2-46bd-abc6-d66757e5edf5-kube-api-access-9dxk4\") pod \"istiod-openshift-gateway-7876bfcb74-xjs5w\" (UID: \"ade54df0-04b2-46bd-abc6-d66757e5edf5\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-xjs5w" Nov 03 13:13:01.159746 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:13:01.159713 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-xjs5w" Nov 03 13:13:01.438182 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:13:01.438111 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-xjs5w"] Nov 03 13:13:01.440896 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:13:01.440865 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podade54df0_04b2_46bd_abc6_d66757e5edf5.slice/crio-c98becef7d53f88420df5765661f6abea411b00d6361fe35de251f8c09208035 WatchSource:0}: Error finding container c98becef7d53f88420df5765661f6abea411b00d6361fe35de251f8c09208035: Status 404 returned error can't find the container with id c98becef7d53f88420df5765661f6abea411b00d6361fe35de251f8c09208035 Nov 03 13:13:02.213402 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:13:02.213362 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-xjs5w" event={"ID":"ade54df0-04b2-46bd-abc6-d66757e5edf5","Type":"ContainerStarted","Data":"c98becef7d53f88420df5765661f6abea411b00d6361fe35de251f8c09208035"} Nov 03 13:13:07.473126 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:13:07.473094 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:13:07.473404 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:13:07.473165 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:13:08.238184 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:13:08.238143 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-xjs5w" event={"ID":"ade54df0-04b2-46bd-abc6-d66757e5edf5","Type":"ContainerStarted","Data":"51529a6186b1a33f0cb68bb4524178d9557c45a40b3748a7812d9b822c9174c0"} Nov 03 13:13:08.238423 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:13:08.238335 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-xjs5w" Nov 03 13:13:08.239945 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:13:08.239912 5024 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7876bfcb74-xjs5w container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 13:13:08.240036 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:13:08.239969 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-xjs5w" podUID="ade54df0-04b2-46bd-abc6-d66757e5edf5" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 13:13:08.267897 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:13:08.267845 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-xjs5w" podStartSLOduration=2.2384378 podStartE2EDuration="8.267832757s" totalImagesPullingTime="6.029394957s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:13:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:13:08.265698254 +0000 UTC m=+268.273889532" watchObservedRunningTime="2026-11-03 13:13:08.267832757 +0000 UTC m=+268.276024034" Nov 03 13:13:09.242904 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:13:09.242868 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-xjs5w" Nov 03 13:13:09.507685 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:13:09.507600 5024 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d7mwz" containerName="extractor" Nov 03 13:13:31.796836 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:13:31.796799 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-5965948766-qhxm9"] Nov 03 13:13:31.806653 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:13:31.806630 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5965948766-qhxm9" Nov 03 13:13:31.831011 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:13:31.830979 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-5965948766-qhxm9"] Nov 03 13:13:31.872879 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:13:31.872846 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dcc2e112-88be-4a51-b8d0-de8be20b84e0-tmp\") pod \"opendatahub-operator-controller-manager-5965948766-qhxm9\" (UID: \"dcc2e112-88be-4a51-b8d0-de8be20b84e0\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5965948766-qhxm9" Nov 03 13:13:31.873040 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:13:31.872894 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/dcc2e112-88be-4a51-b8d0-de8be20b84e0-webhook-cert\") pod \"opendatahub-operator-controller-manager-5965948766-qhxm9\" (UID: \"dcc2e112-88be-4a51-b8d0-de8be20b84e0\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5965948766-qhxm9" Nov 03 13:13:31.873040 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:13:31.872951 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wvxl7\" (UniqueName: \"kubernetes.io/projected/dcc2e112-88be-4a51-b8d0-de8be20b84e0-kube-api-access-wvxl7\") pod \"opendatahub-operator-controller-manager-5965948766-qhxm9\" (UID: \"dcc2e112-88be-4a51-b8d0-de8be20b84e0\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5965948766-qhxm9" Nov 03 13:13:31.873040 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:13:31.872999 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/dcc2e112-88be-4a51-b8d0-de8be20b84e0-metrics-cert\") pod \"opendatahub-operator-controller-manager-5965948766-qhxm9\" (UID: \"dcc2e112-88be-4a51-b8d0-de8be20b84e0\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5965948766-qhxm9" Nov 03 13:13:31.873142 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:13:31.873097 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/dcc2e112-88be-4a51-b8d0-de8be20b84e0-manifests\") pod \"opendatahub-operator-controller-manager-5965948766-qhxm9\" (UID: \"dcc2e112-88be-4a51-b8d0-de8be20b84e0\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5965948766-qhxm9" Nov 03 13:13:31.873176 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:13:31.873158 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/dcc2e112-88be-4a51-b8d0-de8be20b84e0-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5965948766-qhxm9\" (UID: \"dcc2e112-88be-4a51-b8d0-de8be20b84e0\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5965948766-qhxm9" Nov 03 13:13:31.874179 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:13:31.874162 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Nov 03 13:13:31.874229 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:13:31.874173 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Nov 03 13:13:31.877674 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:13:31.877649 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Nov 03 13:13:31.877836 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:13:31.877706 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Nov 03 13:13:31.877901 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:13:31.877725 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-dvqfb\"" Nov 03 13:13:31.877901 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:13:31.877795 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Nov 03 13:13:31.974106 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:13:31.974069 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/dcc2e112-88be-4a51-b8d0-de8be20b84e0-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5965948766-qhxm9\" (UID: \"dcc2e112-88be-4a51-b8d0-de8be20b84e0\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5965948766-qhxm9" Nov 03 13:13:31.974278 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:13:31.974115 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dcc2e112-88be-4a51-b8d0-de8be20b84e0-tmp\") pod \"opendatahub-operator-controller-manager-5965948766-qhxm9\" (UID: \"dcc2e112-88be-4a51-b8d0-de8be20b84e0\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5965948766-qhxm9" Nov 03 13:13:31.974278 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:13:31.974160 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/dcc2e112-88be-4a51-b8d0-de8be20b84e0-webhook-cert\") pod \"opendatahub-operator-controller-manager-5965948766-qhxm9\" (UID: \"dcc2e112-88be-4a51-b8d0-de8be20b84e0\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5965948766-qhxm9" Nov 03 13:13:31.974278 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:13:31.974183 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvxl7\" (UniqueName: \"kubernetes.io/projected/dcc2e112-88be-4a51-b8d0-de8be20b84e0-kube-api-access-wvxl7\") pod \"opendatahub-operator-controller-manager-5965948766-qhxm9\" (UID: \"dcc2e112-88be-4a51-b8d0-de8be20b84e0\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5965948766-qhxm9" Nov 03 13:13:31.974278 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:13:31.974211 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/dcc2e112-88be-4a51-b8d0-de8be20b84e0-metrics-cert\") pod \"opendatahub-operator-controller-manager-5965948766-qhxm9\" (UID: \"dcc2e112-88be-4a51-b8d0-de8be20b84e0\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5965948766-qhxm9" Nov 03 13:13:31.974278 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:13:31.974258 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/dcc2e112-88be-4a51-b8d0-de8be20b84e0-manifests\") pod \"opendatahub-operator-controller-manager-5965948766-qhxm9\" (UID: \"dcc2e112-88be-4a51-b8d0-de8be20b84e0\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5965948766-qhxm9" Nov 03 13:13:31.974767 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:13:31.974733 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/dcc2e112-88be-4a51-b8d0-de8be20b84e0-manifests\") pod \"opendatahub-operator-controller-manager-5965948766-qhxm9\" (UID: \"dcc2e112-88be-4a51-b8d0-de8be20b84e0\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5965948766-qhxm9" Nov 03 13:13:31.976856 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:13:31.976824 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dcc2e112-88be-4a51-b8d0-de8be20b84e0-tmp\") pod \"opendatahub-operator-controller-manager-5965948766-qhxm9\" (UID: \"dcc2e112-88be-4a51-b8d0-de8be20b84e0\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5965948766-qhxm9" Nov 03 13:13:31.976955 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:13:31.976857 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/dcc2e112-88be-4a51-b8d0-de8be20b84e0-metrics-cert\") pod \"opendatahub-operator-controller-manager-5965948766-qhxm9\" (UID: \"dcc2e112-88be-4a51-b8d0-de8be20b84e0\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5965948766-qhxm9" Nov 03 13:13:31.976955 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:13:31.976862 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/dcc2e112-88be-4a51-b8d0-de8be20b84e0-webhook-cert\") pod \"opendatahub-operator-controller-manager-5965948766-qhxm9\" (UID: \"dcc2e112-88be-4a51-b8d0-de8be20b84e0\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5965948766-qhxm9" Nov 03 13:13:31.977026 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:13:31.976951 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/dcc2e112-88be-4a51-b8d0-de8be20b84e0-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5965948766-qhxm9\" (UID: \"dcc2e112-88be-4a51-b8d0-de8be20b84e0\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5965948766-qhxm9" Nov 03 13:13:31.987033 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:13:31.987009 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wvxl7\" (UniqueName: \"kubernetes.io/projected/dcc2e112-88be-4a51-b8d0-de8be20b84e0-kube-api-access-wvxl7\") pod \"opendatahub-operator-controller-manager-5965948766-qhxm9\" (UID: \"dcc2e112-88be-4a51-b8d0-de8be20b84e0\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5965948766-qhxm9" Nov 03 13:13:32.122410 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:13:32.122374 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5965948766-qhxm9" Nov 03 13:13:32.419942 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:13:32.419918 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-5965948766-qhxm9"] Nov 03 13:13:32.422380 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:13:32.422354 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddcc2e112_88be_4a51_b8d0_de8be20b84e0.slice/crio-badc2a801b7716fe54755f237a1591719bfd61098911a8d094b237965d15eeee WatchSource:0}: Error finding container badc2a801b7716fe54755f237a1591719bfd61098911a8d094b237965d15eeee: Status 404 returned error can't find the container with id badc2a801b7716fe54755f237a1591719bfd61098911a8d094b237965d15eeee Nov 03 13:13:33.339325 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:13:33.339271 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5965948766-qhxm9" event={"ID":"dcc2e112-88be-4a51-b8d0-de8be20b84e0","Type":"ContainerStarted","Data":"badc2a801b7716fe54755f237a1591719bfd61098911a8d094b237965d15eeee"} Nov 03 13:13:35.348905 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:13:35.348864 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5965948766-qhxm9" event={"ID":"dcc2e112-88be-4a51-b8d0-de8be20b84e0","Type":"ContainerStarted","Data":"ce2659b482ea33a9c428c0902f1553757f31f4bc4cf8eb9aece5f39a8107ddbe"} Nov 03 13:13:36.354810 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:13:36.354780 5024 generic.go:356] "Generic (PLEG): container finished" podID="dcc2e112-88be-4a51-b8d0-de8be20b84e0" containerID="ce2659b482ea33a9c428c0902f1553757f31f4bc4cf8eb9aece5f39a8107ddbe" exitCode=0 Nov 03 13:13:36.355194 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:13:36.354864 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5965948766-qhxm9" event={"ID":"dcc2e112-88be-4a51-b8d0-de8be20b84e0","Type":"ContainerDied","Data":"ce2659b482ea33a9c428c0902f1553757f31f4bc4cf8eb9aece5f39a8107ddbe"} Nov 03 13:13:37.360674 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:13:37.360635 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5965948766-qhxm9" event={"ID":"dcc2e112-88be-4a51-b8d0-de8be20b84e0","Type":"ContainerStarted","Data":"b25af9c1efc8200baa9cf0e64cf4e5e36740e2f91e5264f36330de9455e49b45"} Nov 03 13:13:37.361117 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:13:37.360763 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5965948766-qhxm9" Nov 03 13:13:37.400591 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:13:37.400505 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5965948766-qhxm9" podStartSLOduration=2.616390766 podStartE2EDuration="6.400486178s" totalImagesPullingTime="3.232478873s" totalInitContainerRuntime="551.616539ms" isStatefulPod=false podCreationTimestamp="2026-11-03 13:13:31 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:13:37.397705891 +0000 UTC m=+297.405897181" watchObservedRunningTime="2026-11-03 13:13:37.400486178 +0000 UTC m=+297.408677455" Nov 03 13:13:39.507306 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:13:39.507253 5024 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d7mwz" containerName="extractor" Nov 03 13:13:40.129724 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:13:40.129690 5024 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6htht_cd91de08-084e-4667-b13f-2e07b3cff624/console-operator/1.log" Nov 03 13:13:40.130177 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:13:40.130151 5024 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6htht_cd91de08-084e-4667-b13f-2e07b3cff624/console-operator/1.log" Nov 03 13:13:48.366931 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:13:48.366901 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5965948766-qhxm9" Nov 03 13:14:08.555755 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:14:08.555720 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-84d8c65c8f-j592j"] Nov 03 13:14:08.566736 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:14:08.566711 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-84d8c65c8f-j592j" Nov 03 13:14:08.571077 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:14:08.571047 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-84d8c65c8f-j592j"] Nov 03 13:14:08.658390 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:14:08.658348 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Nov 03 13:14:08.658563 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:14:08.658402 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Nov 03 13:14:08.658563 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:14:08.658457 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-8xtzk\"" Nov 03 13:14:08.697593 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:14:08.697557 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zl6s4\" (UniqueName: \"kubernetes.io/projected/24d19d80-a3b9-468e-9aff-035a62a578fe-kube-api-access-zl6s4\") pod \"kube-auth-proxy-84d8c65c8f-j592j\" (UID: \"24d19d80-a3b9-468e-9aff-035a62a578fe\") " pod="openshift-ingress/kube-auth-proxy-84d8c65c8f-j592j" Nov 03 13:14:08.697742 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:14:08.697603 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/24d19d80-a3b9-468e-9aff-035a62a578fe-tmp\") pod \"kube-auth-proxy-84d8c65c8f-j592j\" (UID: \"24d19d80-a3b9-468e-9aff-035a62a578fe\") " pod="openshift-ingress/kube-auth-proxy-84d8c65c8f-j592j" Nov 03 13:14:08.697742 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:14:08.697722 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/24d19d80-a3b9-468e-9aff-035a62a578fe-tls-certs\") pod \"kube-auth-proxy-84d8c65c8f-j592j\" (UID: \"24d19d80-a3b9-468e-9aff-035a62a578fe\") " pod="openshift-ingress/kube-auth-proxy-84d8c65c8f-j592j" Nov 03 13:14:08.799199 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:14:08.799155 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/24d19d80-a3b9-468e-9aff-035a62a578fe-tls-certs\") pod \"kube-auth-proxy-84d8c65c8f-j592j\" (UID: \"24d19d80-a3b9-468e-9aff-035a62a578fe\") " pod="openshift-ingress/kube-auth-proxy-84d8c65c8f-j592j" Nov 03 13:14:08.799391 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:14:08.799265 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zl6s4\" (UniqueName: \"kubernetes.io/projected/24d19d80-a3b9-468e-9aff-035a62a578fe-kube-api-access-zl6s4\") pod \"kube-auth-proxy-84d8c65c8f-j592j\" (UID: \"24d19d80-a3b9-468e-9aff-035a62a578fe\") " pod="openshift-ingress/kube-auth-proxy-84d8c65c8f-j592j" Nov 03 13:14:08.799391 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:14:08.799327 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/24d19d80-a3b9-468e-9aff-035a62a578fe-tmp\") pod \"kube-auth-proxy-84d8c65c8f-j592j\" (UID: \"24d19d80-a3b9-468e-9aff-035a62a578fe\") " pod="openshift-ingress/kube-auth-proxy-84d8c65c8f-j592j" Nov 03 13:14:08.801675 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:14:08.801646 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/24d19d80-a3b9-468e-9aff-035a62a578fe-tmp\") pod \"kube-auth-proxy-84d8c65c8f-j592j\" (UID: \"24d19d80-a3b9-468e-9aff-035a62a578fe\") " pod="openshift-ingress/kube-auth-proxy-84d8c65c8f-j592j" Nov 03 13:14:08.801868 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:14:08.801705 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/24d19d80-a3b9-468e-9aff-035a62a578fe-tls-certs\") pod \"kube-auth-proxy-84d8c65c8f-j592j\" (UID: \"24d19d80-a3b9-468e-9aff-035a62a578fe\") " pod="openshift-ingress/kube-auth-proxy-84d8c65c8f-j592j" Nov 03 13:14:08.811497 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:14:08.811442 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zl6s4\" (UniqueName: \"kubernetes.io/projected/24d19d80-a3b9-468e-9aff-035a62a578fe-kube-api-access-zl6s4\") pod \"kube-auth-proxy-84d8c65c8f-j592j\" (UID: \"24d19d80-a3b9-468e-9aff-035a62a578fe\") " pod="openshift-ingress/kube-auth-proxy-84d8c65c8f-j592j" Nov 03 13:14:08.882240 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:14:08.882202 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-84d8c65c8f-j592j" Nov 03 13:14:09.200856 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:14:09.200823 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-84d8c65c8f-j592j"] Nov 03 13:14:09.203829 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:14:09.203790 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24d19d80_a3b9_468e_9aff_035a62a578fe.slice/crio-d392fb3b8a607b0001850fb68c142472ea199af15f52ceb7138d5bc2d3a1d718 WatchSource:0}: Error finding container d392fb3b8a607b0001850fb68c142472ea199af15f52ceb7138d5bc2d3a1d718: Status 404 returned error can't find the container with id d392fb3b8a607b0001850fb68c142472ea199af15f52ceb7138d5bc2d3a1d718 Nov 03 13:14:09.205874 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:14:09.205854 5024 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 13:14:09.490377 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:14:09.490298 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-84d8c65c8f-j592j" event={"ID":"24d19d80-a3b9-468e-9aff-035a62a578fe","Type":"ContainerStarted","Data":"d392fb3b8a607b0001850fb68c142472ea199af15f52ceb7138d5bc2d3a1d718"} Nov 03 13:14:09.506909 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:14:09.506871 5024 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d7mwz" containerName="extractor" Nov 03 13:14:11.775877 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:14:11.775846 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-nx2g7"] Nov 03 13:14:11.779217 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:14:11.779198 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-nx2g7" Nov 03 13:14:11.786410 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:14:11.786380 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-nx2g7"] Nov 03 13:14:11.794038 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:14:11.794006 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-kk72q\"" Nov 03 13:14:11.824370 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:14:11.824306 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Nov 03 13:14:11.824370 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:14:11.824312 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Nov 03 13:14:11.824532 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:14:11.824508 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f5pjn\" (UniqueName: \"kubernetes.io/projected/3a0dbb8e-6cf2-4228-822f-1bcaf3b747d5-kube-api-access-f5pjn\") pod \"authorino-56c6c9ddd5-nx2g7\" (UID: \"3a0dbb8e-6cf2-4228-822f-1bcaf3b747d5\") " pod="kuadrant-system/authorino-56c6c9ddd5-nx2g7" Nov 03 13:14:11.925178 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:14:11.925124 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f5pjn\" (UniqueName: \"kubernetes.io/projected/3a0dbb8e-6cf2-4228-822f-1bcaf3b747d5-kube-api-access-f5pjn\") pod \"authorino-56c6c9ddd5-nx2g7\" (UID: \"3a0dbb8e-6cf2-4228-822f-1bcaf3b747d5\") " pod="kuadrant-system/authorino-56c6c9ddd5-nx2g7" Nov 03 13:14:11.934626 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:14:11.934600 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f5pjn\" (UniqueName: \"kubernetes.io/projected/3a0dbb8e-6cf2-4228-822f-1bcaf3b747d5-kube-api-access-f5pjn\") pod \"authorino-56c6c9ddd5-nx2g7\" (UID: \"3a0dbb8e-6cf2-4228-822f-1bcaf3b747d5\") " pod="kuadrant-system/authorino-56c6c9ddd5-nx2g7" Nov 03 13:14:12.098714 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:14:12.098626 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-nx2g7" Nov 03 13:14:12.372686 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:14:12.372653 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-nx2g7"] Nov 03 13:14:12.375498 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:14:12.375469 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3a0dbb8e_6cf2_4228_822f_1bcaf3b747d5.slice/crio-3f4f9bfb7352b638d3220f018bec0f83eae58f5b55f4a179fff3ab1a083778ed WatchSource:0}: Error finding container 3f4f9bfb7352b638d3220f018bec0f83eae58f5b55f4a179fff3ab1a083778ed: Status 404 returned error can't find the container with id 3f4f9bfb7352b638d3220f018bec0f83eae58f5b55f4a179fff3ab1a083778ed Nov 03 13:14:12.505927 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:14:12.505887 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-84d8c65c8f-j592j" event={"ID":"24d19d80-a3b9-468e-9aff-035a62a578fe","Type":"ContainerStarted","Data":"5e77ff1bde69f379167577c29072303ca44f1d0e99e743f2dd1a6dde83a7ce0e"} Nov 03 13:14:12.507101 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:14:12.507075 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-nx2g7" event={"ID":"3a0dbb8e-6cf2-4228-822f-1bcaf3b747d5","Type":"ContainerStarted","Data":"3f4f9bfb7352b638d3220f018bec0f83eae58f5b55f4a179fff3ab1a083778ed"} Nov 03 13:14:12.524504 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:14:12.524454 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-84d8c65c8f-j592j" podStartSLOduration=2.167030588 podStartE2EDuration="4.524440474s" totalImagesPullingTime="2.357409886s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:14:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:14:12.523543928 +0000 UTC m=+332.531735206" watchObservedRunningTime="2026-11-03 13:14:12.524440474 +0000 UTC m=+332.532631750" Nov 03 13:14:15.523725 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:14:15.523686 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-nx2g7" event={"ID":"3a0dbb8e-6cf2-4228-822f-1bcaf3b747d5","Type":"ContainerStarted","Data":"48cf845a5c4e753502b90b1d6ec4a5f1b3b8fedfa403daeb80f70678c59e84dc"} Nov 03 13:14:15.540714 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:14:15.540663 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-56c6c9ddd5-nx2g7" podStartSLOduration=1.923051858 podStartE2EDuration="4.540648758s" totalImagesPullingTime="2.6175969s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:14:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:14:15.539953351 +0000 UTC m=+335.548144629" watchObservedRunningTime="2026-11-03 13:14:15.540648758 +0000 UTC m=+335.548840035" Nov 03 13:14:19.526724 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:14:19.526686 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-bsvp9"] Nov 03 13:14:19.530155 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:14:19.530137 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-bsvp9" Nov 03 13:14:19.541701 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:14:19.541642 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-5qr2g\"" Nov 03 13:14:19.545103 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:14:19.545076 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-bsvp9"] Nov 03 13:14:19.584925 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:14:19.584880 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8f111884-464c-4369-88ce-d3aed661014d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-bsvp9\" (UID: \"8f111884-464c-4369-88ce-d3aed661014d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-bsvp9" Nov 03 13:14:19.584925 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:14:19.584927 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qvcc4\" (UniqueName: \"kubernetes.io/projected/8f111884-464c-4369-88ce-d3aed661014d-kube-api-access-qvcc4\") pod \"kuadrant-operator-controller-manager-678d87766f-bsvp9\" (UID: \"8f111884-464c-4369-88ce-d3aed661014d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-bsvp9" Nov 03 13:14:19.685906 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:14:19.685865 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8f111884-464c-4369-88ce-d3aed661014d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-bsvp9\" (UID: \"8f111884-464c-4369-88ce-d3aed661014d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-bsvp9" Nov 03 13:14:19.685906 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:14:19.685908 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvcc4\" (UniqueName: \"kubernetes.io/projected/8f111884-464c-4369-88ce-d3aed661014d-kube-api-access-qvcc4\") pod \"kuadrant-operator-controller-manager-678d87766f-bsvp9\" (UID: \"8f111884-464c-4369-88ce-d3aed661014d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-bsvp9" Nov 03 13:14:19.686348 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:14:19.686321 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8f111884-464c-4369-88ce-d3aed661014d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-bsvp9\" (UID: \"8f111884-464c-4369-88ce-d3aed661014d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-bsvp9" Nov 03 13:14:19.699749 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:14:19.699715 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qvcc4\" (UniqueName: \"kubernetes.io/projected/8f111884-464c-4369-88ce-d3aed661014d-kube-api-access-qvcc4\") pod \"kuadrant-operator-controller-manager-678d87766f-bsvp9\" (UID: \"8f111884-464c-4369-88ce-d3aed661014d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-bsvp9" Nov 03 13:14:19.843560 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:14:19.843459 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-bsvp9" Nov 03 13:14:20.133668 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:14:20.133637 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-bsvp9"] Nov 03 13:14:20.135172 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:14:20.135141 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f111884_464c_4369_88ce_d3aed661014d.slice/crio-b584e9db6edc1062407abb81d6beb8f2950e4b8a0918d96f85f43c0744fc3a43 WatchSource:0}: Error finding container b584e9db6edc1062407abb81d6beb8f2950e4b8a0918d96f85f43c0744fc3a43: Status 404 returned error can't find the container with id b584e9db6edc1062407abb81d6beb8f2950e4b8a0918d96f85f43c0744fc3a43 Nov 03 13:14:20.545353 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:14:20.545247 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-bsvp9" event={"ID":"8f111884-464c-4369-88ce-d3aed661014d","Type":"ContainerStarted","Data":"b584e9db6edc1062407abb81d6beb8f2950e4b8a0918d96f85f43c0744fc3a43"} Nov 03 13:14:24.563675 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:14:24.563581 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-bsvp9" event={"ID":"8f111884-464c-4369-88ce-d3aed661014d","Type":"ContainerStarted","Data":"c7c61e5bb5a7db43d92df0fb35b1660b3d05d089a68ad3508eb55866317de7b3"} Nov 03 13:14:24.564043 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:14:24.563711 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-bsvp9" Nov 03 13:14:24.591669 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:14:24.591615 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-bsvp9" podStartSLOduration=1.56912097 podStartE2EDuration="5.59160298s" totalImagesPullingTime="4.02248201s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:14:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:14:24.588648388 +0000 UTC m=+344.596839690" watchObservedRunningTime="2026-11-03 13:14:24.59160298 +0000 UTC m=+344.599794254" Nov 03 13:14:35.569772 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:14:35.569739 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-bsvp9" Nov 03 13:14:39.506334 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:14:39.506279 5024 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d7mwz" containerName="extractor" Nov 03 13:15:05.498328 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:15:05.498275 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b9tb7"] Nov 03 13:15:05.504072 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:15:05.504052 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b9tb7" Nov 03 13:15:05.518765 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:15:05.518736 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b9tb7"] Nov 03 13:15:05.685244 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:15:05.685205 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22e78255-1b4a-430e-ae67-617d02d15b51-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-b9tb7\" (UID: \"22e78255-1b4a-430e-ae67-617d02d15b51\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b9tb7" Nov 03 13:15:05.685244 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:15:05.685250 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/22e78255-1b4a-430e-ae67-617d02d15b51-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-b9tb7\" (UID: \"22e78255-1b4a-430e-ae67-617d02d15b51\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b9tb7" Nov 03 13:15:05.685511 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:15:05.685371 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/22e78255-1b4a-430e-ae67-617d02d15b51-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-b9tb7\" (UID: \"22e78255-1b4a-430e-ae67-617d02d15b51\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b9tb7" Nov 03 13:15:05.685511 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:15:05.685408 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/22e78255-1b4a-430e-ae67-617d02d15b51-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-b9tb7\" (UID: \"22e78255-1b4a-430e-ae67-617d02d15b51\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b9tb7" Nov 03 13:15:05.685511 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:15:05.685427 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zfsn9\" (UniqueName: \"kubernetes.io/projected/22e78255-1b4a-430e-ae67-617d02d15b51-kube-api-access-zfsn9\") pod \"istiod-openshift-gateway-7ddd9cfdc8-b9tb7\" (UID: \"22e78255-1b4a-430e-ae67-617d02d15b51\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b9tb7" Nov 03 13:15:05.685625 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:15:05.685538 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/22e78255-1b4a-430e-ae67-617d02d15b51-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-b9tb7\" (UID: \"22e78255-1b4a-430e-ae67-617d02d15b51\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b9tb7" Nov 03 13:15:05.685625 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:15:05.685580 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/22e78255-1b4a-430e-ae67-617d02d15b51-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-b9tb7\" (UID: \"22e78255-1b4a-430e-ae67-617d02d15b51\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b9tb7" Nov 03 13:15:05.786507 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:15:05.786413 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/22e78255-1b4a-430e-ae67-617d02d15b51-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-b9tb7\" (UID: \"22e78255-1b4a-430e-ae67-617d02d15b51\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b9tb7" Nov 03 13:15:05.786507 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:15:05.786468 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/22e78255-1b4a-430e-ae67-617d02d15b51-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-b9tb7\" (UID: \"22e78255-1b4a-430e-ae67-617d02d15b51\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b9tb7" Nov 03 13:15:05.786740 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:15:05.786512 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22e78255-1b4a-430e-ae67-617d02d15b51-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-b9tb7\" (UID: \"22e78255-1b4a-430e-ae67-617d02d15b51\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b9tb7" Nov 03 13:15:05.786740 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:15:05.786530 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/22e78255-1b4a-430e-ae67-617d02d15b51-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-b9tb7\" (UID: \"22e78255-1b4a-430e-ae67-617d02d15b51\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b9tb7" Nov 03 13:15:05.786740 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:15:05.786560 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/22e78255-1b4a-430e-ae67-617d02d15b51-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-b9tb7\" (UID: \"22e78255-1b4a-430e-ae67-617d02d15b51\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b9tb7" Nov 03 13:15:05.786740 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:15:05.786585 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/22e78255-1b4a-430e-ae67-617d02d15b51-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-b9tb7\" (UID: \"22e78255-1b4a-430e-ae67-617d02d15b51\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b9tb7" Nov 03 13:15:05.786740 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:15:05.786608 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zfsn9\" (UniqueName: \"kubernetes.io/projected/22e78255-1b4a-430e-ae67-617d02d15b51-kube-api-access-zfsn9\") pod \"istiod-openshift-gateway-7ddd9cfdc8-b9tb7\" (UID: \"22e78255-1b4a-430e-ae67-617d02d15b51\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b9tb7" Nov 03 13:15:05.787275 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:15:05.787241 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/22e78255-1b4a-430e-ae67-617d02d15b51-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-b9tb7\" (UID: \"22e78255-1b4a-430e-ae67-617d02d15b51\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b9tb7" Nov 03 13:15:05.788958 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:15:05.788934 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/22e78255-1b4a-430e-ae67-617d02d15b51-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-b9tb7\" (UID: \"22e78255-1b4a-430e-ae67-617d02d15b51\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b9tb7" Nov 03 13:15:05.789088 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:15:05.788983 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/22e78255-1b4a-430e-ae67-617d02d15b51-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-b9tb7\" (UID: \"22e78255-1b4a-430e-ae67-617d02d15b51\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b9tb7" Nov 03 13:15:05.789179 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:15:05.789159 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/22e78255-1b4a-430e-ae67-617d02d15b51-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-b9tb7\" (UID: \"22e78255-1b4a-430e-ae67-617d02d15b51\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b9tb7" Nov 03 13:15:05.789233 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:15:05.789213 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/22e78255-1b4a-430e-ae67-617d02d15b51-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-b9tb7\" (UID: \"22e78255-1b4a-430e-ae67-617d02d15b51\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b9tb7" Nov 03 13:15:05.801499 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:15:05.801472 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22e78255-1b4a-430e-ae67-617d02d15b51-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-b9tb7\" (UID: \"22e78255-1b4a-430e-ae67-617d02d15b51\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b9tb7" Nov 03 13:15:05.802683 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:15:05.802663 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zfsn9\" (UniqueName: \"kubernetes.io/projected/22e78255-1b4a-430e-ae67-617d02d15b51-kube-api-access-zfsn9\") pod \"istiod-openshift-gateway-7ddd9cfdc8-b9tb7\" (UID: \"22e78255-1b4a-430e-ae67-617d02d15b51\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b9tb7" Nov 03 13:15:05.814488 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:15:05.814467 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b9tb7" Nov 03 13:15:06.091102 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:15:06.091019 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b9tb7"] Nov 03 13:15:06.095975 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:15:06.095952 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22e78255_1b4a_430e_ae67_617d02d15b51.slice/crio-8eaa933fd5d93058e1c4e406761d096c8771ef17c2c0fc5e84943032b0f303e4 WatchSource:0}: Error finding container 8eaa933fd5d93058e1c4e406761d096c8771ef17c2c0fc5e84943032b0f303e4: Status 404 returned error can't find the container with id 8eaa933fd5d93058e1c4e406761d096c8771ef17c2c0fc5e84943032b0f303e4 Nov 03 13:15:06.098052 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:15:06.098017 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:15:06.098152 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:15:06.098092 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:15:06.736179 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:15:06.736144 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b9tb7" event={"ID":"22e78255-1b4a-430e-ae67-617d02d15b51","Type":"ContainerStarted","Data":"50b79ccfe3e11fdd16b41b0b2bfcd27c750965754deb3e9370af345870d8720e"} Nov 03 13:15:06.736680 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:15:06.736192 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b9tb7" event={"ID":"22e78255-1b4a-430e-ae67-617d02d15b51","Type":"ContainerStarted","Data":"8eaa933fd5d93058e1c4e406761d096c8771ef17c2c0fc5e84943032b0f303e4"} Nov 03 13:15:06.736680 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:15:06.736303 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b9tb7" Nov 03 13:15:06.765916 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:15:06.765853 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b9tb7" podStartSLOduration=1.765841069 podStartE2EDuration="1.765841069s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:15:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:15:06.763172831 +0000 UTC m=+386.771364114" watchObservedRunningTime="2026-11-03 13:15:06.765841069 +0000 UTC m=+386.774032346" Nov 03 13:15:07.741954 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:15:07.741916 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b9tb7" Nov 03 13:15:07.791767 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:15:07.791697 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-xjs5w"] Nov 03 13:15:07.791951 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:15:07.791916 5024 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-xjs5w" podUID="ade54df0-04b2-46bd-abc6-d66757e5edf5" containerName="discovery" containerID="cri-o://51529a6186b1a33f0cb68bb4524178d9557c45a40b3748a7812d9b822c9174c0" gracePeriod=30 Nov 03 13:15:08.048416 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:15:08.048389 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-xjs5w" Nov 03 13:15:08.208189 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:15:08.208080 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ade54df0-04b2-46bd-abc6-d66757e5edf5-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/ade54df0-04b2-46bd-abc6-d66757e5edf5-istio-csr-ca-configmap\") pod \"ade54df0-04b2-46bd-abc6-d66757e5edf5\" (UID: \"ade54df0-04b2-46bd-abc6-d66757e5edf5\") " Nov 03 13:15:08.208189 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:15:08.208166 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ade54df0-04b2-46bd-abc6-d66757e5edf5-kube-api-access-9dxk4\" (UniqueName: \"kubernetes.io/projected/ade54df0-04b2-46bd-abc6-d66757e5edf5-kube-api-access-9dxk4\") pod \"ade54df0-04b2-46bd-abc6-d66757e5edf5\" (UID: \"ade54df0-04b2-46bd-abc6-d66757e5edf5\") " Nov 03 13:15:08.208530 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:15:08.208265 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ade54df0-04b2-46bd-abc6-d66757e5edf5-cacerts\" (UniqueName: \"kubernetes.io/secret/ade54df0-04b2-46bd-abc6-d66757e5edf5-cacerts\") pod \"ade54df0-04b2-46bd-abc6-d66757e5edf5\" (UID: \"ade54df0-04b2-46bd-abc6-d66757e5edf5\") " Nov 03 13:15:08.208530 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:15:08.208335 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ade54df0-04b2-46bd-abc6-d66757e5edf5-istio-token\" (UniqueName: \"kubernetes.io/projected/ade54df0-04b2-46bd-abc6-d66757e5edf5-istio-token\") pod \"ade54df0-04b2-46bd-abc6-d66757e5edf5\" (UID: \"ade54df0-04b2-46bd-abc6-d66757e5edf5\") " Nov 03 13:15:08.208530 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:15:08.208355 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ade54df0-04b2-46bd-abc6-d66757e5edf5-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ade54df0-04b2-46bd-abc6-d66757e5edf5-istio-kubeconfig\") pod \"ade54df0-04b2-46bd-abc6-d66757e5edf5\" (UID: \"ade54df0-04b2-46bd-abc6-d66757e5edf5\") " Nov 03 13:15:08.208530 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:15:08.208378 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ade54df0-04b2-46bd-abc6-d66757e5edf5-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/ade54df0-04b2-46bd-abc6-d66757e5edf5-istio-csr-dns-cert\") pod \"ade54df0-04b2-46bd-abc6-d66757e5edf5\" (UID: \"ade54df0-04b2-46bd-abc6-d66757e5edf5\") " Nov 03 13:15:08.208530 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:15:08.208440 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ade54df0-04b2-46bd-abc6-d66757e5edf5-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/ade54df0-04b2-46bd-abc6-d66757e5edf5-local-certs\") pod \"ade54df0-04b2-46bd-abc6-d66757e5edf5\" (UID: \"ade54df0-04b2-46bd-abc6-d66757e5edf5\") " Nov 03 13:15:08.208803 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:15:08.208545 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ade54df0-04b2-46bd-abc6-d66757e5edf5-istio-csr-ca-configmap" pod "ade54df0-04b2-46bd-abc6-d66757e5edf5" (UID: "ade54df0-04b2-46bd-abc6-d66757e5edf5"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:15:08.208803 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:15:08.208729 5024 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/ade54df0-04b2-46bd-abc6-d66757e5edf5-istio-csr-ca-configmap\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:15:08.210799 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:15:08.210771 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ade54df0-04b2-46bd-abc6-d66757e5edf5-cacerts" pod "ade54df0-04b2-46bd-abc6-d66757e5edf5" (UID: "ade54df0-04b2-46bd-abc6-d66757e5edf5"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 13:15:08.210913 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:15:08.210880 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ade54df0-04b2-46bd-abc6-d66757e5edf5-istio-kubeconfig" pod "ade54df0-04b2-46bd-abc6-d66757e5edf5" (UID: "ade54df0-04b2-46bd-abc6-d66757e5edf5"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 13:15:08.210973 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:15:08.210933 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ade54df0-04b2-46bd-abc6-d66757e5edf5-istio-token" pod "ade54df0-04b2-46bd-abc6-d66757e5edf5" (UID: "ade54df0-04b2-46bd-abc6-d66757e5edf5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:15:08.210973 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:15:08.210962 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ade54df0-04b2-46bd-abc6-d66757e5edf5-istio-csr-dns-cert" pod "ade54df0-04b2-46bd-abc6-d66757e5edf5" (UID: "ade54df0-04b2-46bd-abc6-d66757e5edf5"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 13:15:08.211075 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:15:08.210967 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ade54df0-04b2-46bd-abc6-d66757e5edf5-kube-api-access-9dxk4" pod "ade54df0-04b2-46bd-abc6-d66757e5edf5" (UID: "ade54df0-04b2-46bd-abc6-d66757e5edf5"). InnerVolumeSpecName "kube-api-access-9dxk4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:15:08.211075 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:15:08.210995 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ade54df0-04b2-46bd-abc6-d66757e5edf5-local-certs" pod "ade54df0-04b2-46bd-abc6-d66757e5edf5" (UID: "ade54df0-04b2-46bd-abc6-d66757e5edf5"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:15:08.309391 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:15:08.309258 5024 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/ade54df0-04b2-46bd-abc6-d66757e5edf5-local-certs\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:15:08.309391 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:15:08.309334 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9dxk4\" (UniqueName: \"kubernetes.io/projected/ade54df0-04b2-46bd-abc6-d66757e5edf5-kube-api-access-9dxk4\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:15:08.309391 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:15:08.309349 5024 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/ade54df0-04b2-46bd-abc6-d66757e5edf5-cacerts\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:15:08.309391 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:15:08.309359 5024 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ade54df0-04b2-46bd-abc6-d66757e5edf5-istio-token\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:15:08.309391 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:15:08.309368 5024 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ade54df0-04b2-46bd-abc6-d66757e5edf5-istio-kubeconfig\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:15:08.309391 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:15:08.309377 5024 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/ade54df0-04b2-46bd-abc6-d66757e5edf5-istio-csr-dns-cert\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:15:08.745627 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:15:08.745595 5024 generic.go:356] "Generic (PLEG): container finished" podID="ade54df0-04b2-46bd-abc6-d66757e5edf5" containerID="51529a6186b1a33f0cb68bb4524178d9557c45a40b3748a7812d9b822c9174c0" exitCode=0 Nov 03 13:15:08.746074 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:15:08.745644 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-xjs5w" Nov 03 13:15:08.746074 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:15:08.745690 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-xjs5w" event={"ID":"ade54df0-04b2-46bd-abc6-d66757e5edf5","Type":"ContainerDied","Data":"51529a6186b1a33f0cb68bb4524178d9557c45a40b3748a7812d9b822c9174c0"} Nov 03 13:15:08.746074 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:15:08.745746 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-xjs5w" event={"ID":"ade54df0-04b2-46bd-abc6-d66757e5edf5","Type":"ContainerDied","Data":"c98becef7d53f88420df5765661f6abea411b00d6361fe35de251f8c09208035"} Nov 03 13:15:08.746074 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:15:08.745773 5024 scope.go:122] "RemoveContainer" containerID="51529a6186b1a33f0cb68bb4524178d9557c45a40b3748a7812d9b822c9174c0" Nov 03 13:15:08.755464 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:15:08.755440 5024 scope.go:122] "RemoveContainer" containerID="51529a6186b1a33f0cb68bb4524178d9557c45a40b3748a7812d9b822c9174c0" Nov 03 13:15:08.755763 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:15:08.755741 5024 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"51529a6186b1a33f0cb68bb4524178d9557c45a40b3748a7812d9b822c9174c0\": container with ID starting with 51529a6186b1a33f0cb68bb4524178d9557c45a40b3748a7812d9b822c9174c0 not found: ID does not exist" containerID="51529a6186b1a33f0cb68bb4524178d9557c45a40b3748a7812d9b822c9174c0" Nov 03 13:15:08.755805 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:15:08.755774 5024 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51529a6186b1a33f0cb68bb4524178d9557c45a40b3748a7812d9b822c9174c0"} err="failed to get container status \"51529a6186b1a33f0cb68bb4524178d9557c45a40b3748a7812d9b822c9174c0\": rpc error: code = NotFound desc = could not find container \"51529a6186b1a33f0cb68bb4524178d9557c45a40b3748a7812d9b822c9174c0\": container with ID starting with 51529a6186b1a33f0cb68bb4524178d9557c45a40b3748a7812d9b822c9174c0 not found: ID does not exist" Nov 03 13:15:08.770370 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:15:08.770346 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-xjs5w"] Nov 03 13:15:08.774713 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:15:08.774689 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-xjs5w"] Nov 03 13:15:09.505817 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:15:09.505777 5024 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d7mwz" containerName="extractor" Nov 03 13:15:10.106568 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:15:10.106535 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ade54df0-04b2-46bd-abc6-d66757e5edf5" path="/var/lib/kubelet/pods/ade54df0-04b2-46bd-abc6-d66757e5edf5/volumes" Nov 03 13:15:24.984574 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:15:24.984539 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-7b59d85bd-fhrvt"] Nov 03 13:15:24.984939 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:15:24.984882 5024 cpu_manager.go:414] "containerMap: removing container" podUID="ade54df0-04b2-46bd-abc6-d66757e5edf5" containerName="discovery" Nov 03 13:15:24.984939 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:15:24.984894 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ade54df0-04b2-46bd-abc6-d66757e5edf5" containerName="discovery" Nov 03 13:15:24.985007 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:15:24.984970 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="ade54df0-04b2-46bd-abc6-d66757e5edf5" containerName="discovery" Nov 03 13:15:24.988200 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:15:24.988182 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7b59d85bd-fhrvt" Nov 03 13:15:25.000321 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:15:25.000264 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 03 13:15:25.000321 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:15:25.000274 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 13:15:25.000321 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:15:25.000312 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 13:15:25.000515 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:15:25.000332 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Nov 03 13:15:25.000515 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:15:25.000465 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-p5qd5\"" Nov 03 13:15:25.001696 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:15:25.001673 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7b59d85bd-fhrvt"] Nov 03 13:15:25.049433 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:15:25.049398 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0b35ab55-eda2-4db5-b97a-f61bf59c87ce-metrics-certs\") pod \"maas-api-7b59d85bd-fhrvt\" (UID: \"0b35ab55-eda2-4db5-b97a-f61bf59c87ce\") " pod="odh-ai-gateway-infra/maas-api-7b59d85bd-fhrvt" Nov 03 13:15:25.049433 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:15:25.049436 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b35ab55-eda2-4db5-b97a-f61bf59c87ce-openshift-service-ca\") pod \"maas-api-7b59d85bd-fhrvt\" (UID: \"0b35ab55-eda2-4db5-b97a-f61bf59c87ce\") " pod="odh-ai-gateway-infra/maas-api-7b59d85bd-fhrvt" Nov 03 13:15:25.049670 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:15:25.049472 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0b35ab55-eda2-4db5-b97a-f61bf59c87ce-maas-api-tls\") pod \"maas-api-7b59d85bd-fhrvt\" (UID: \"0b35ab55-eda2-4db5-b97a-f61bf59c87ce\") " pod="odh-ai-gateway-infra/maas-api-7b59d85bd-fhrvt" Nov 03 13:15:25.049670 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:15:25.049638 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zzjzj\" (UniqueName: \"kubernetes.io/projected/0b35ab55-eda2-4db5-b97a-f61bf59c87ce-kube-api-access-zzjzj\") pod \"maas-api-7b59d85bd-fhrvt\" (UID: \"0b35ab55-eda2-4db5-b97a-f61bf59c87ce\") " pod="odh-ai-gateway-infra/maas-api-7b59d85bd-fhrvt" Nov 03 13:15:25.150538 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:15:25.150494 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zzjzj\" (UniqueName: \"kubernetes.io/projected/0b35ab55-eda2-4db5-b97a-f61bf59c87ce-kube-api-access-zzjzj\") pod \"maas-api-7b59d85bd-fhrvt\" (UID: \"0b35ab55-eda2-4db5-b97a-f61bf59c87ce\") " pod="odh-ai-gateway-infra/maas-api-7b59d85bd-fhrvt" Nov 03 13:15:25.150747 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:15:25.150544 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0b35ab55-eda2-4db5-b97a-f61bf59c87ce-metrics-certs\") pod \"maas-api-7b59d85bd-fhrvt\" (UID: \"0b35ab55-eda2-4db5-b97a-f61bf59c87ce\") " pod="odh-ai-gateway-infra/maas-api-7b59d85bd-fhrvt" Nov 03 13:15:25.150747 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:15:25.150578 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b35ab55-eda2-4db5-b97a-f61bf59c87ce-openshift-service-ca\") pod \"maas-api-7b59d85bd-fhrvt\" (UID: \"0b35ab55-eda2-4db5-b97a-f61bf59c87ce\") " pod="odh-ai-gateway-infra/maas-api-7b59d85bd-fhrvt" Nov 03 13:15:25.150747 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:15:25.150627 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0b35ab55-eda2-4db5-b97a-f61bf59c87ce-maas-api-tls\") pod \"maas-api-7b59d85bd-fhrvt\" (UID: \"0b35ab55-eda2-4db5-b97a-f61bf59c87ce\") " pod="odh-ai-gateway-infra/maas-api-7b59d85bd-fhrvt" Nov 03 13:15:25.151395 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:15:25.151369 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b35ab55-eda2-4db5-b97a-f61bf59c87ce-openshift-service-ca\") pod \"maas-api-7b59d85bd-fhrvt\" (UID: \"0b35ab55-eda2-4db5-b97a-f61bf59c87ce\") " pod="odh-ai-gateway-infra/maas-api-7b59d85bd-fhrvt" Nov 03 13:15:25.153125 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:15:25.153103 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0b35ab55-eda2-4db5-b97a-f61bf59c87ce-metrics-certs\") pod \"maas-api-7b59d85bd-fhrvt\" (UID: \"0b35ab55-eda2-4db5-b97a-f61bf59c87ce\") " pod="odh-ai-gateway-infra/maas-api-7b59d85bd-fhrvt" Nov 03 13:15:25.153227 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:15:25.153160 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0b35ab55-eda2-4db5-b97a-f61bf59c87ce-maas-api-tls\") pod \"maas-api-7b59d85bd-fhrvt\" (UID: \"0b35ab55-eda2-4db5-b97a-f61bf59c87ce\") " pod="odh-ai-gateway-infra/maas-api-7b59d85bd-fhrvt" Nov 03 13:15:25.160795 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:15:25.160770 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zzjzj\" (UniqueName: \"kubernetes.io/projected/0b35ab55-eda2-4db5-b97a-f61bf59c87ce-kube-api-access-zzjzj\") pod \"maas-api-7b59d85bd-fhrvt\" (UID: \"0b35ab55-eda2-4db5-b97a-f61bf59c87ce\") " pod="odh-ai-gateway-infra/maas-api-7b59d85bd-fhrvt" Nov 03 13:15:25.301544 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:15:25.301453 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7b59d85bd-fhrvt" Nov 03 13:15:25.574433 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:15:25.574405 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7b59d85bd-fhrvt"] Nov 03 13:15:25.576982 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:15:25.576953 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b35ab55_eda2_4db5_b97a_f61bf59c87ce.slice/crio-caaa77e5a6cf1c40bcff7b9faa0d9e2a5e7ab7b4f21ec1971ed291514d4a804e WatchSource:0}: Error finding container caaa77e5a6cf1c40bcff7b9faa0d9e2a5e7ab7b4f21ec1971ed291514d4a804e: Status 404 returned error can't find the container with id caaa77e5a6cf1c40bcff7b9faa0d9e2a5e7ab7b4f21ec1971ed291514d4a804e Nov 03 13:15:25.817947 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:15:25.817908 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7b59d85bd-fhrvt" event={"ID":"0b35ab55-eda2-4db5-b97a-f61bf59c87ce","Type":"ContainerStarted","Data":"caaa77e5a6cf1c40bcff7b9faa0d9e2a5e7ab7b4f21ec1971ed291514d4a804e"} Nov 03 13:15:30.841299 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:15:30.841238 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7b59d85bd-fhrvt" event={"ID":"0b35ab55-eda2-4db5-b97a-f61bf59c87ce","Type":"ContainerStarted","Data":"1499c46220c550b477fee32b9ba76c47f7dec49ba5e0c4716f96d7bdd165fd99"} Nov 03 13:15:30.841702 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:15:30.841324 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-7b59d85bd-fhrvt" Nov 03 13:15:30.861138 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:15:30.861082 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-7b59d85bd-fhrvt" podStartSLOduration=2.601983893 podStartE2EDuration="6.861067588s" totalImagesPullingTime="4.259083695s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:15:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:15:30.860368082 +0000 UTC m=+410.868559359" watchObservedRunningTime="2026-11-03 13:15:30.861067588 +0000 UTC m=+410.869258856" Nov 03 13:15:36.851427 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:15:36.851398 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-7b59d85bd-fhrvt" Nov 03 13:15:38.336517 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:15:38.336476 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-p4bbv"] Nov 03 13:15:38.340195 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:15:38.340178 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-p4bbv" Nov 03 13:15:38.347332 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:15:38.347262 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-p4bbv"] Nov 03 13:15:38.347510 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:15:38.347462 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Nov 03 13:15:38.465143 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:15:38.465102 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4jhqt\" (UniqueName: \"kubernetes.io/projected/8af62367-4b99-4ca0-afe0-e223fc9dd805-kube-api-access-4jhqt\") pod \"authorino-6889cbd8c4-p4bbv\" (UID: \"8af62367-4b99-4ca0-afe0-e223fc9dd805\") " pod="kuadrant-system/authorino-6889cbd8c4-p4bbv" Nov 03 13:15:38.465336 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:15:38.465154 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/8af62367-4b99-4ca0-afe0-e223fc9dd805-tls-cert\") pod \"authorino-6889cbd8c4-p4bbv\" (UID: \"8af62367-4b99-4ca0-afe0-e223fc9dd805\") " pod="kuadrant-system/authorino-6889cbd8c4-p4bbv" Nov 03 13:15:38.565875 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:15:38.565840 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jhqt\" (UniqueName: \"kubernetes.io/projected/8af62367-4b99-4ca0-afe0-e223fc9dd805-kube-api-access-4jhqt\") pod \"authorino-6889cbd8c4-p4bbv\" (UID: \"8af62367-4b99-4ca0-afe0-e223fc9dd805\") " pod="kuadrant-system/authorino-6889cbd8c4-p4bbv" Nov 03 13:15:38.565875 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:15:38.565881 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/8af62367-4b99-4ca0-afe0-e223fc9dd805-tls-cert\") pod \"authorino-6889cbd8c4-p4bbv\" (UID: \"8af62367-4b99-4ca0-afe0-e223fc9dd805\") " pod="kuadrant-system/authorino-6889cbd8c4-p4bbv" Nov 03 13:15:38.568324 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:15:38.568295 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/8af62367-4b99-4ca0-afe0-e223fc9dd805-tls-cert\") pod \"authorino-6889cbd8c4-p4bbv\" (UID: \"8af62367-4b99-4ca0-afe0-e223fc9dd805\") " pod="kuadrant-system/authorino-6889cbd8c4-p4bbv" Nov 03 13:15:38.574950 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:15:38.574927 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jhqt\" (UniqueName: \"kubernetes.io/projected/8af62367-4b99-4ca0-afe0-e223fc9dd805-kube-api-access-4jhqt\") pod \"authorino-6889cbd8c4-p4bbv\" (UID: \"8af62367-4b99-4ca0-afe0-e223fc9dd805\") " pod="kuadrant-system/authorino-6889cbd8c4-p4bbv" Nov 03 13:15:38.654393 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:15:38.654361 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-p4bbv" Nov 03 13:15:38.667537 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:15:38.667510 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-p4bbv"] Nov 03 13:15:38.718099 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:15:38.717715 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5588f885fc-qvxj7"] Nov 03 13:15:38.722412 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:15:38.722388 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5588f885fc-qvxj7" Nov 03 13:15:38.740028 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:15:38.740006 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5588f885fc-qvxj7"] Nov 03 13:15:38.867956 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:15:38.867925 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/0f2b79da-fca5-41bf-832c-34dce95f1ca5-tls-cert\") pod \"authorino-5588f885fc-qvxj7\" (UID: \"0f2b79da-fca5-41bf-832c-34dce95f1ca5\") " pod="kuadrant-system/authorino-5588f885fc-qvxj7" Nov 03 13:15:38.868107 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:15:38.867975 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hs7kc\" (UniqueName: \"kubernetes.io/projected/0f2b79da-fca5-41bf-832c-34dce95f1ca5-kube-api-access-hs7kc\") pod \"authorino-5588f885fc-qvxj7\" (UID: \"0f2b79da-fca5-41bf-832c-34dce95f1ca5\") " pod="kuadrant-system/authorino-5588f885fc-qvxj7" Nov 03 13:15:38.923074 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:15:38.922915 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-p4bbv"] Nov 03 13:15:38.925557 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:15:38.925532 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8af62367_4b99_4ca0_afe0_e223fc9dd805.slice/crio-8507a226fee322c2d29559869a20f6187d1e64546b7b22bf1aa4a77bf794f93b WatchSource:0}: Error finding container 8507a226fee322c2d29559869a20f6187d1e64546b7b22bf1aa4a77bf794f93b: Status 404 returned error can't find the container with id 8507a226fee322c2d29559869a20f6187d1e64546b7b22bf1aa4a77bf794f93b Nov 03 13:15:38.968636 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:15:38.968600 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/0f2b79da-fca5-41bf-832c-34dce95f1ca5-tls-cert\") pod \"authorino-5588f885fc-qvxj7\" (UID: \"0f2b79da-fca5-41bf-832c-34dce95f1ca5\") " pod="kuadrant-system/authorino-5588f885fc-qvxj7" Nov 03 13:15:38.968811 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:15:38.968662 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hs7kc\" (UniqueName: \"kubernetes.io/projected/0f2b79da-fca5-41bf-832c-34dce95f1ca5-kube-api-access-hs7kc\") pod \"authorino-5588f885fc-qvxj7\" (UID: \"0f2b79da-fca5-41bf-832c-34dce95f1ca5\") " pod="kuadrant-system/authorino-5588f885fc-qvxj7" Nov 03 13:15:38.971074 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:15:38.971040 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/0f2b79da-fca5-41bf-832c-34dce95f1ca5-tls-cert\") pod \"authorino-5588f885fc-qvxj7\" (UID: \"0f2b79da-fca5-41bf-832c-34dce95f1ca5\") " pod="kuadrant-system/authorino-5588f885fc-qvxj7" Nov 03 13:15:38.977233 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:15:38.977208 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hs7kc\" (UniqueName: \"kubernetes.io/projected/0f2b79da-fca5-41bf-832c-34dce95f1ca5-kube-api-access-hs7kc\") pod \"authorino-5588f885fc-qvxj7\" (UID: \"0f2b79da-fca5-41bf-832c-34dce95f1ca5\") " pod="kuadrant-system/authorino-5588f885fc-qvxj7" Nov 03 13:15:39.035788 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:15:39.035758 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5588f885fc-qvxj7" Nov 03 13:15:39.306572 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:15:39.306540 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5588f885fc-qvxj7"] Nov 03 13:15:39.308968 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:15:39.308939 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0f2b79da_fca5_41bf_832c_34dce95f1ca5.slice/crio-83ac0992502a97a875804966a31028a1726b3a251e189ec9cb0b5da7cefb0f06 WatchSource:0}: Error finding container 83ac0992502a97a875804966a31028a1726b3a251e189ec9cb0b5da7cefb0f06: Status 404 returned error can't find the container with id 83ac0992502a97a875804966a31028a1726b3a251e189ec9cb0b5da7cefb0f06 Nov 03 13:15:39.506320 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:15:39.506272 5024 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d7mwz" containerName="extractor" Nov 03 13:15:39.885271 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:15:39.885224 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6889cbd8c4-p4bbv" event={"ID":"8af62367-4b99-4ca0-afe0-e223fc9dd805","Type":"ContainerStarted","Data":"28d45448d091c59d15c15084459ca71186cabaf98fab44d41281c57b474ba37f"} Nov 03 13:15:39.885473 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:15:39.885294 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6889cbd8c4-p4bbv" event={"ID":"8af62367-4b99-4ca0-afe0-e223fc9dd805","Type":"ContainerStarted","Data":"8507a226fee322c2d29559869a20f6187d1e64546b7b22bf1aa4a77bf794f93b"} Nov 03 13:15:39.885473 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:15:39.885258 5024 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-6889cbd8c4-p4bbv" podUID="8af62367-4b99-4ca0-afe0-e223fc9dd805" containerName="authorino" containerID="cri-o://28d45448d091c59d15c15084459ca71186cabaf98fab44d41281c57b474ba37f" gracePeriod=30 Nov 03 13:15:39.886568 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:15:39.886540 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5588f885fc-qvxj7" event={"ID":"0f2b79da-fca5-41bf-832c-34dce95f1ca5","Type":"ContainerStarted","Data":"83ac0992502a97a875804966a31028a1726b3a251e189ec9cb0b5da7cefb0f06"} Nov 03 13:15:39.909978 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:15:39.909932 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-6889cbd8c4-p4bbv" podStartSLOduration=1.109955345 podStartE2EDuration="1.909917969s" totalImagesPullingTime="799.962624ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:15:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:15:39.907098536 +0000 UTC m=+419.915289814" watchObservedRunningTime="2026-11-03 13:15:39.909917969 +0000 UTC m=+419.918109244" Nov 03 13:15:40.122887 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:15:40.122859 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-p4bbv" Nov 03 13:15:40.179054 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:15:40.179017 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8af62367-4b99-4ca0-afe0-e223fc9dd805-tls-cert\" (UniqueName: \"kubernetes.io/secret/8af62367-4b99-4ca0-afe0-e223fc9dd805-tls-cert\") pod \"8af62367-4b99-4ca0-afe0-e223fc9dd805\" (UID: \"8af62367-4b99-4ca0-afe0-e223fc9dd805\") " Nov 03 13:15:40.179246 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:15:40.179069 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8af62367-4b99-4ca0-afe0-e223fc9dd805-kube-api-access-4jhqt\" (UniqueName: \"kubernetes.io/projected/8af62367-4b99-4ca0-afe0-e223fc9dd805-kube-api-access-4jhqt\") pod \"8af62367-4b99-4ca0-afe0-e223fc9dd805\" (UID: \"8af62367-4b99-4ca0-afe0-e223fc9dd805\") " Nov 03 13:15:40.181246 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:15:40.181214 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8af62367-4b99-4ca0-afe0-e223fc9dd805-kube-api-access-4jhqt" pod "8af62367-4b99-4ca0-afe0-e223fc9dd805" (UID: "8af62367-4b99-4ca0-afe0-e223fc9dd805"). InnerVolumeSpecName "kube-api-access-4jhqt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:15:40.190928 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:15:40.190895 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8af62367-4b99-4ca0-afe0-e223fc9dd805-tls-cert" pod "8af62367-4b99-4ca0-afe0-e223fc9dd805" (UID: "8af62367-4b99-4ca0-afe0-e223fc9dd805"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 13:15:40.280668 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:15:40.280567 5024 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/8af62367-4b99-4ca0-afe0-e223fc9dd805-tls-cert\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:15:40.280911 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:15:40.280889 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4jhqt\" (UniqueName: \"kubernetes.io/projected/8af62367-4b99-4ca0-afe0-e223fc9dd805-kube-api-access-4jhqt\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:15:40.891690 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:15:40.891654 5024 generic.go:356] "Generic (PLEG): container finished" podID="8af62367-4b99-4ca0-afe0-e223fc9dd805" containerID="28d45448d091c59d15c15084459ca71186cabaf98fab44d41281c57b474ba37f" exitCode=0 Nov 03 13:15:40.892184 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:15:40.891713 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-p4bbv" Nov 03 13:15:40.892184 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:15:40.891740 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6889cbd8c4-p4bbv" event={"ID":"8af62367-4b99-4ca0-afe0-e223fc9dd805","Type":"ContainerDied","Data":"28d45448d091c59d15c15084459ca71186cabaf98fab44d41281c57b474ba37f"} Nov 03 13:15:40.892184 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:15:40.891791 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6889cbd8c4-p4bbv" event={"ID":"8af62367-4b99-4ca0-afe0-e223fc9dd805","Type":"ContainerDied","Data":"8507a226fee322c2d29559869a20f6187d1e64546b7b22bf1aa4a77bf794f93b"} Nov 03 13:15:40.892184 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:15:40.891816 5024 scope.go:122] "RemoveContainer" containerID="28d45448d091c59d15c15084459ca71186cabaf98fab44d41281c57b474ba37f" Nov 03 13:15:40.893428 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:15:40.893406 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5588f885fc-qvxj7" event={"ID":"0f2b79da-fca5-41bf-832c-34dce95f1ca5","Type":"ContainerStarted","Data":"868302f875c2aa5b9233e49183bfe0bc38998e0046d1191dbbfec63f7645bd15"} Nov 03 13:15:40.900188 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:15:40.900166 5024 scope.go:122] "RemoveContainer" containerID="28d45448d091c59d15c15084459ca71186cabaf98fab44d41281c57b474ba37f" Nov 03 13:15:40.900492 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:15:40.900471 5024 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"28d45448d091c59d15c15084459ca71186cabaf98fab44d41281c57b474ba37f\": container with ID starting with 28d45448d091c59d15c15084459ca71186cabaf98fab44d41281c57b474ba37f not found: ID does not exist" containerID="28d45448d091c59d15c15084459ca71186cabaf98fab44d41281c57b474ba37f" Nov 03 13:15:40.900564 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:15:40.900501 5024 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28d45448d091c59d15c15084459ca71186cabaf98fab44d41281c57b474ba37f"} err="failed to get container status \"28d45448d091c59d15c15084459ca71186cabaf98fab44d41281c57b474ba37f\": rpc error: code = NotFound desc = could not find container \"28d45448d091c59d15c15084459ca71186cabaf98fab44d41281c57b474ba37f\": container with ID starting with 28d45448d091c59d15c15084459ca71186cabaf98fab44d41281c57b474ba37f not found: ID does not exist" Nov 03 13:15:40.915869 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:15:40.915822 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-5588f885fc-qvxj7" podStartSLOduration=2.3176789859999998 podStartE2EDuration="2.915812027s" totalImagesPullingTime="598.133041ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:15:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:15:40.91352352 +0000 UTC m=+420.921714800" watchObservedRunningTime="2026-11-03 13:15:40.915812027 +0000 UTC m=+420.924003299" Nov 03 13:15:40.928891 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:15:40.928861 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-p4bbv"] Nov 03 13:15:40.932217 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:15:40.932191 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-p4bbv"] Nov 03 13:15:40.937927 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:15:40.937897 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-nx2g7"] Nov 03 13:15:40.938145 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:15:40.938095 5024 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-56c6c9ddd5-nx2g7" podUID="3a0dbb8e-6cf2-4228-822f-1bcaf3b747d5" containerName="authorino" containerID="cri-o://48cf845a5c4e753502b90b1d6ec4a5f1b3b8fedfa403daeb80f70678c59e84dc" gracePeriod=30 Nov 03 13:15:41.194091 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:15:41.194064 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-nx2g7" Nov 03 13:15:41.291385 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:15:41.291348 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a0dbb8e-6cf2-4228-822f-1bcaf3b747d5-kube-api-access-f5pjn\" (UniqueName: \"kubernetes.io/projected/3a0dbb8e-6cf2-4228-822f-1bcaf3b747d5-kube-api-access-f5pjn\") pod \"3a0dbb8e-6cf2-4228-822f-1bcaf3b747d5\" (UID: \"3a0dbb8e-6cf2-4228-822f-1bcaf3b747d5\") " Nov 03 13:15:41.293468 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:15:41.293438 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a0dbb8e-6cf2-4228-822f-1bcaf3b747d5-kube-api-access-f5pjn" pod "3a0dbb8e-6cf2-4228-822f-1bcaf3b747d5" (UID: "3a0dbb8e-6cf2-4228-822f-1bcaf3b747d5"). InnerVolumeSpecName "kube-api-access-f5pjn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:15:41.392186 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:15:41.392149 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f5pjn\" (UniqueName: \"kubernetes.io/projected/3a0dbb8e-6cf2-4228-822f-1bcaf3b747d5-kube-api-access-f5pjn\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:15:41.898922 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:15:41.898886 5024 generic.go:356] "Generic (PLEG): container finished" podID="3a0dbb8e-6cf2-4228-822f-1bcaf3b747d5" containerID="48cf845a5c4e753502b90b1d6ec4a5f1b3b8fedfa403daeb80f70678c59e84dc" exitCode=0 Nov 03 13:15:41.899406 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:15:41.898926 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-nx2g7" Nov 03 13:15:41.899406 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:15:41.898970 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-nx2g7" event={"ID":"3a0dbb8e-6cf2-4228-822f-1bcaf3b747d5","Type":"ContainerDied","Data":"48cf845a5c4e753502b90b1d6ec4a5f1b3b8fedfa403daeb80f70678c59e84dc"} Nov 03 13:15:41.899406 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:15:41.899014 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-nx2g7" event={"ID":"3a0dbb8e-6cf2-4228-822f-1bcaf3b747d5","Type":"ContainerDied","Data":"3f4f9bfb7352b638d3220f018bec0f83eae58f5b55f4a179fff3ab1a083778ed"} Nov 03 13:15:41.899406 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:15:41.899031 5024 scope.go:122] "RemoveContainer" containerID="48cf845a5c4e753502b90b1d6ec4a5f1b3b8fedfa403daeb80f70678c59e84dc" Nov 03 13:15:41.907769 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:15:41.907752 5024 scope.go:122] "RemoveContainer" containerID="48cf845a5c4e753502b90b1d6ec4a5f1b3b8fedfa403daeb80f70678c59e84dc" Nov 03 13:15:41.908034 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:15:41.908017 5024 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"48cf845a5c4e753502b90b1d6ec4a5f1b3b8fedfa403daeb80f70678c59e84dc\": container with ID starting with 48cf845a5c4e753502b90b1d6ec4a5f1b3b8fedfa403daeb80f70678c59e84dc not found: ID does not exist" containerID="48cf845a5c4e753502b90b1d6ec4a5f1b3b8fedfa403daeb80f70678c59e84dc" Nov 03 13:15:41.908085 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:15:41.908042 5024 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48cf845a5c4e753502b90b1d6ec4a5f1b3b8fedfa403daeb80f70678c59e84dc"} err="failed to get container status \"48cf845a5c4e753502b90b1d6ec4a5f1b3b8fedfa403daeb80f70678c59e84dc\": rpc error: code = NotFound desc = could not find container \"48cf845a5c4e753502b90b1d6ec4a5f1b3b8fedfa403daeb80f70678c59e84dc\": container with ID starting with 48cf845a5c4e753502b90b1d6ec4a5f1b3b8fedfa403daeb80f70678c59e84dc not found: ID does not exist" Nov 03 13:15:41.920363 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:15:41.920340 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-nx2g7"] Nov 03 13:15:41.924935 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:15:41.924912 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-nx2g7"] Nov 03 13:15:42.106024 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:15:42.105993 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a0dbb8e-6cf2-4228-822f-1bcaf3b747d5" path="/var/lib/kubelet/pods/3a0dbb8e-6cf2-4228-822f-1bcaf3b747d5/volumes" Nov 03 13:15:42.106375 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:15:42.106361 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8af62367-4b99-4ca0-afe0-e223fc9dd805" path="/var/lib/kubelet/pods/8af62367-4b99-4ca0-afe0-e223fc9dd805/volumes" Nov 03 13:15:47.622102 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:15:47.622062 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-7b59d85bd-fhrvt"] Nov 03 13:15:47.622678 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:15:47.622278 5024 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-7b59d85bd-fhrvt" podUID="0b35ab55-eda2-4db5-b97a-f61bf59c87ce" containerName="maas-api" containerID="cri-o://1499c46220c550b477fee32b9ba76c47f7dec49ba5e0c4716f96d7bdd165fd99" gracePeriod=30 Nov 03 13:15:47.863021 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:15:47.862996 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7b59d85bd-fhrvt" Nov 03 13:15:47.933435 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:15:47.933345 5024 generic.go:356] "Generic (PLEG): container finished" podID="0b35ab55-eda2-4db5-b97a-f61bf59c87ce" containerID="1499c46220c550b477fee32b9ba76c47f7dec49ba5e0c4716f96d7bdd165fd99" exitCode=0 Nov 03 13:15:47.933435 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:15:47.933392 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7b59d85bd-fhrvt" Nov 03 13:15:47.933435 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:15:47.933424 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7b59d85bd-fhrvt" event={"ID":"0b35ab55-eda2-4db5-b97a-f61bf59c87ce","Type":"ContainerDied","Data":"1499c46220c550b477fee32b9ba76c47f7dec49ba5e0c4716f96d7bdd165fd99"} Nov 03 13:15:47.933634 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:15:47.933479 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7b59d85bd-fhrvt" event={"ID":"0b35ab55-eda2-4db5-b97a-f61bf59c87ce","Type":"ContainerDied","Data":"caaa77e5a6cf1c40bcff7b9faa0d9e2a5e7ab7b4f21ec1971ed291514d4a804e"} Nov 03 13:15:47.933634 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:15:47.933502 5024 scope.go:122] "RemoveContainer" containerID="1499c46220c550b477fee32b9ba76c47f7dec49ba5e0c4716f96d7bdd165fd99" Nov 03 13:15:47.942320 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:15:47.942278 5024 scope.go:122] "RemoveContainer" containerID="1499c46220c550b477fee32b9ba76c47f7dec49ba5e0c4716f96d7bdd165fd99" Nov 03 13:15:47.942586 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:15:47.942567 5024 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1499c46220c550b477fee32b9ba76c47f7dec49ba5e0c4716f96d7bdd165fd99\": container with ID starting with 1499c46220c550b477fee32b9ba76c47f7dec49ba5e0c4716f96d7bdd165fd99 not found: ID does not exist" containerID="1499c46220c550b477fee32b9ba76c47f7dec49ba5e0c4716f96d7bdd165fd99" Nov 03 13:15:47.942648 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:15:47.942602 5024 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1499c46220c550b477fee32b9ba76c47f7dec49ba5e0c4716f96d7bdd165fd99"} err="failed to get container status \"1499c46220c550b477fee32b9ba76c47f7dec49ba5e0c4716f96d7bdd165fd99\": rpc error: code = NotFound desc = could not find container \"1499c46220c550b477fee32b9ba76c47f7dec49ba5e0c4716f96d7bdd165fd99\": container with ID starting with 1499c46220c550b477fee32b9ba76c47f7dec49ba5e0c4716f96d7bdd165fd99 not found: ID does not exist" Nov 03 13:15:47.953935 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:15:47.953913 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0b35ab55-eda2-4db5-b97a-f61bf59c87ce-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b35ab55-eda2-4db5-b97a-f61bf59c87ce-openshift-service-ca\") pod \"0b35ab55-eda2-4db5-b97a-f61bf59c87ce\" (UID: \"0b35ab55-eda2-4db5-b97a-f61bf59c87ce\") " Nov 03 13:15:47.954018 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:15:47.953969 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0b35ab55-eda2-4db5-b97a-f61bf59c87ce-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0b35ab55-eda2-4db5-b97a-f61bf59c87ce-maas-api-tls\") pod \"0b35ab55-eda2-4db5-b97a-f61bf59c87ce\" (UID: \"0b35ab55-eda2-4db5-b97a-f61bf59c87ce\") " Nov 03 13:15:47.954061 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:15:47.954025 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0b35ab55-eda2-4db5-b97a-f61bf59c87ce-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0b35ab55-eda2-4db5-b97a-f61bf59c87ce-metrics-certs\") pod \"0b35ab55-eda2-4db5-b97a-f61bf59c87ce\" (UID: \"0b35ab55-eda2-4db5-b97a-f61bf59c87ce\") " Nov 03 13:15:47.954061 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:15:47.954049 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b35ab55-eda2-4db5-b97a-f61bf59c87ce-kube-api-access-zzjzj\" (UniqueName: \"kubernetes.io/projected/0b35ab55-eda2-4db5-b97a-f61bf59c87ce-kube-api-access-zzjzj\") pod \"0b35ab55-eda2-4db5-b97a-f61bf59c87ce\" (UID: \"0b35ab55-eda2-4db5-b97a-f61bf59c87ce\") " Nov 03 13:15:47.954312 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:15:47.954261 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b35ab55-eda2-4db5-b97a-f61bf59c87ce-openshift-service-ca" pod "0b35ab55-eda2-4db5-b97a-f61bf59c87ce" (UID: "0b35ab55-eda2-4db5-b97a-f61bf59c87ce"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:15:47.956188 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:15:47.956162 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b35ab55-eda2-4db5-b97a-f61bf59c87ce-maas-api-tls" pod "0b35ab55-eda2-4db5-b97a-f61bf59c87ce" (UID: "0b35ab55-eda2-4db5-b97a-f61bf59c87ce"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 13:15:47.956263 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:15:47.956241 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b35ab55-eda2-4db5-b97a-f61bf59c87ce-kube-api-access-zzjzj" pod "0b35ab55-eda2-4db5-b97a-f61bf59c87ce" (UID: "0b35ab55-eda2-4db5-b97a-f61bf59c87ce"). InnerVolumeSpecName "kube-api-access-zzjzj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:15:47.956362 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:15:47.956262 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b35ab55-eda2-4db5-b97a-f61bf59c87ce-metrics-certs" pod "0b35ab55-eda2-4db5-b97a-f61bf59c87ce" (UID: "0b35ab55-eda2-4db5-b97a-f61bf59c87ce"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 13:15:48.055550 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:15:48.055515 5024 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0b35ab55-eda2-4db5-b97a-f61bf59c87ce-maas-api-tls\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:15:48.055550 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:15:48.055545 5024 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0b35ab55-eda2-4db5-b97a-f61bf59c87ce-metrics-certs\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:15:48.055779 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:15:48.055557 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zzjzj\" (UniqueName: \"kubernetes.io/projected/0b35ab55-eda2-4db5-b97a-f61bf59c87ce-kube-api-access-zzjzj\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:15:48.055779 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:15:48.055568 5024 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b35ab55-eda2-4db5-b97a-f61bf59c87ce-openshift-service-ca\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:15:48.253077 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:15:48.252999 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-7b59d85bd-fhrvt"] Nov 03 13:15:48.259331 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:15:48.259304 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-7b59d85bd-fhrvt"] Nov 03 13:15:50.106228 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:15:50.106195 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b35ab55-eda2-4db5-b97a-f61bf59c87ce" path="/var/lib/kubelet/pods/0b35ab55-eda2-4db5-b97a-f61bf59c87ce/volumes" Nov 03 13:16:09.507069 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:16:09.507025 5024 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d7mwz" containerName="extractor" Nov 03 13:16:36.506881 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:16:36.506835 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-5d8dddb676-vts47"] Nov 03 13:16:36.507408 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:16:36.507249 5024 cpu_manager.go:414] "containerMap: removing container" podUID="0b35ab55-eda2-4db5-b97a-f61bf59c87ce" containerName="maas-api" Nov 03 13:16:36.507408 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:16:36.507269 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b35ab55-eda2-4db5-b97a-f61bf59c87ce" containerName="maas-api" Nov 03 13:16:36.507408 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:16:36.507314 5024 cpu_manager.go:414] "containerMap: removing container" podUID="8af62367-4b99-4ca0-afe0-e223fc9dd805" containerName="authorino" Nov 03 13:16:36.507408 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:16:36.507324 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8af62367-4b99-4ca0-afe0-e223fc9dd805" containerName="authorino" Nov 03 13:16:36.507408 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:16:36.507347 5024 cpu_manager.go:414] "containerMap: removing container" podUID="3a0dbb8e-6cf2-4228-822f-1bcaf3b747d5" containerName="authorino" Nov 03 13:16:36.507408 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:16:36.507355 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a0dbb8e-6cf2-4228-822f-1bcaf3b747d5" containerName="authorino" Nov 03 13:16:36.507727 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:16:36.507425 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a0dbb8e-6cf2-4228-822f-1bcaf3b747d5" containerName="authorino" Nov 03 13:16:36.507727 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:16:36.507436 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b35ab55-eda2-4db5-b97a-f61bf59c87ce" containerName="maas-api" Nov 03 13:16:36.507727 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:16:36.507445 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="8af62367-4b99-4ca0-afe0-e223fc9dd805" containerName="authorino" Nov 03 13:16:36.512009 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:16:36.511624 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-5d8dddb676-vts47" Nov 03 13:16:36.518994 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:16:36.518957 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-dockercfg-5ljqz\"" Nov 03 13:16:36.519184 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:16:36.518981 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Nov 03 13:16:36.519184 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:16:36.519041 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-metrics-tls\"" Nov 03 13:16:36.519390 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:16:36.519056 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Nov 03 13:16:36.519390 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:16:36.519104 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"ai-gateway-controller-parameters\"" Nov 03 13:16:36.521156 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:16:36.521125 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-5d8dddb676-vts47"] Nov 03 13:16:36.663913 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:16:36.663870 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rjndz\" (UniqueName: \"kubernetes.io/projected/135dd450-92b6-4e25-a00b-a28d587f03ea-kube-api-access-rjndz\") pod \"ai-gateway-controller-5d8dddb676-vts47\" (UID: \"135dd450-92b6-4e25-a00b-a28d587f03ea\") " pod="opendatahub/ai-gateway-controller-5d8dddb676-vts47" Nov 03 13:16:36.664096 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:16:36.664013 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/135dd450-92b6-4e25-a00b-a28d587f03ea-metrics-certs\") pod \"ai-gateway-controller-5d8dddb676-vts47\" (UID: \"135dd450-92b6-4e25-a00b-a28d587f03ea\") " pod="opendatahub/ai-gateway-controller-5d8dddb676-vts47" Nov 03 13:16:36.765006 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:16:36.764925 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/135dd450-92b6-4e25-a00b-a28d587f03ea-metrics-certs\") pod \"ai-gateway-controller-5d8dddb676-vts47\" (UID: \"135dd450-92b6-4e25-a00b-a28d587f03ea\") " pod="opendatahub/ai-gateway-controller-5d8dddb676-vts47" Nov 03 13:16:36.765006 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:16:36.764974 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rjndz\" (UniqueName: \"kubernetes.io/projected/135dd450-92b6-4e25-a00b-a28d587f03ea-kube-api-access-rjndz\") pod \"ai-gateway-controller-5d8dddb676-vts47\" (UID: \"135dd450-92b6-4e25-a00b-a28d587f03ea\") " pod="opendatahub/ai-gateway-controller-5d8dddb676-vts47" Nov 03 13:16:36.767411 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:16:36.767380 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/135dd450-92b6-4e25-a00b-a28d587f03ea-metrics-certs\") pod \"ai-gateway-controller-5d8dddb676-vts47\" (UID: \"135dd450-92b6-4e25-a00b-a28d587f03ea\") " pod="opendatahub/ai-gateway-controller-5d8dddb676-vts47" Nov 03 13:16:36.774776 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:16:36.774746 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rjndz\" (UniqueName: \"kubernetes.io/projected/135dd450-92b6-4e25-a00b-a28d587f03ea-kube-api-access-rjndz\") pod \"ai-gateway-controller-5d8dddb676-vts47\" (UID: \"135dd450-92b6-4e25-a00b-a28d587f03ea\") " pod="opendatahub/ai-gateway-controller-5d8dddb676-vts47" Nov 03 13:16:36.826794 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:16:36.826750 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-5d8dddb676-vts47" Nov 03 13:16:37.097804 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:16:37.097776 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-5d8dddb676-vts47"] Nov 03 13:16:37.100582 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:16:37.100550 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod135dd450_92b6_4e25_a00b_a28d587f03ea.slice/crio-d8218490b2ca25af3c2cdb1b5955f2dd569de6475346c90cf52e664268a9c9af WatchSource:0}: Error finding container d8218490b2ca25af3c2cdb1b5955f2dd569de6475346c90cf52e664268a9c9af: Status 404 returned error can't find the container with id d8218490b2ca25af3c2cdb1b5955f2dd569de6475346c90cf52e664268a9c9af Nov 03 13:16:37.136936 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:16:37.136904 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-5d8dddb676-vts47" event={"ID":"135dd450-92b6-4e25-a00b-a28d587f03ea","Type":"ContainerStarted","Data":"d8218490b2ca25af3c2cdb1b5955f2dd569de6475346c90cf52e664268a9c9af"} Nov 03 13:16:39.147980 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:16:39.147945 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-5d8dddb676-vts47" event={"ID":"135dd450-92b6-4e25-a00b-a28d587f03ea","Type":"ContainerStarted","Data":"88f9bcf70fe0276c3f490ed7276597ba4a8a8033ecb639ed1dec34eecf2d8a65"} Nov 03 13:16:39.148499 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:16:39.148012 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-5d8dddb676-vts47" Nov 03 13:16:39.168395 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:16:39.168347 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-5d8dddb676-vts47" podStartSLOduration=1.771181836 podStartE2EDuration="3.168332213s" totalImagesPullingTime="1.397150377s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:16:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:16:39.16721176 +0000 UTC m=+479.175403039" watchObservedRunningTime="2026-11-03 13:16:39.168332213 +0000 UTC m=+479.176523489" Nov 03 13:16:39.507270 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:16:39.507185 5024 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d7mwz" containerName="extractor" Nov 03 13:16:40.321871 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:16:40.321841 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-p9ckq"] Nov 03 13:16:40.326837 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:16:40.326815 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-p9ckq" Nov 03 13:16:40.334476 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:16:40.333712 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Nov 03 13:16:40.334476 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:16:40.333908 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-ls9nf\"" Nov 03 13:16:40.335930 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:16:40.335910 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-p9ckq"] Nov 03 13:16:40.376014 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:16:40.375977 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-khw76"] Nov 03 13:16:40.379368 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:16:40.379349 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-khw76" Nov 03 13:16:40.386905 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:16:40.386879 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-khw76"] Nov 03 13:16:40.499200 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:16:40.499104 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gn5vr\" (UniqueName: \"kubernetes.io/projected/793b4955-68a1-4bf4-bcc7-7f0556c6783c-kube-api-access-gn5vr\") pod \"payload-pre-processing-7b98bb678c-p9ckq\" (UID: \"793b4955-68a1-4bf4-bcc7-7f0556c6783c\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-p9ckq" Nov 03 13:16:40.499200 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:16:40.499146 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rhmc8\" (UniqueName: \"kubernetes.io/projected/36118654-3637-489e-b048-af4aa366f34d-kube-api-access-rhmc8\") pod \"payload-processing-55fbbbf9dc-khw76\" (UID: \"36118654-3637-489e-b048-af4aa366f34d\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-khw76" Nov 03 13:16:40.499419 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:16:40.499316 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36118654-3637-489e-b048-af4aa366f34d-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-khw76\" (UID: \"36118654-3637-489e-b048-af4aa366f34d\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-khw76" Nov 03 13:16:40.499419 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:16:40.499349 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/793b4955-68a1-4bf4-bcc7-7f0556c6783c-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-p9ckq\" (UID: \"793b4955-68a1-4bf4-bcc7-7f0556c6783c\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-p9ckq" Nov 03 13:16:40.600806 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:16:40.600770 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36118654-3637-489e-b048-af4aa366f34d-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-khw76\" (UID: \"36118654-3637-489e-b048-af4aa366f34d\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-khw76" Nov 03 13:16:40.601013 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:16:40.600814 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/793b4955-68a1-4bf4-bcc7-7f0556c6783c-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-p9ckq\" (UID: \"793b4955-68a1-4bf4-bcc7-7f0556c6783c\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-p9ckq" Nov 03 13:16:40.601013 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:16:40.600861 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gn5vr\" (UniqueName: \"kubernetes.io/projected/793b4955-68a1-4bf4-bcc7-7f0556c6783c-kube-api-access-gn5vr\") pod \"payload-pre-processing-7b98bb678c-p9ckq\" (UID: \"793b4955-68a1-4bf4-bcc7-7f0556c6783c\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-p9ckq" Nov 03 13:16:40.601013 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:16:40.600888 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhmc8\" (UniqueName: \"kubernetes.io/projected/36118654-3637-489e-b048-af4aa366f34d-kube-api-access-rhmc8\") pod \"payload-processing-55fbbbf9dc-khw76\" (UID: \"36118654-3637-489e-b048-af4aa366f34d\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-khw76" Nov 03 13:16:40.601656 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:16:40.601628 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36118654-3637-489e-b048-af4aa366f34d-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-khw76\" (UID: \"36118654-3637-489e-b048-af4aa366f34d\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-khw76" Nov 03 13:16:40.601656 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:16:40.601646 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/793b4955-68a1-4bf4-bcc7-7f0556c6783c-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-p9ckq\" (UID: \"793b4955-68a1-4bf4-bcc7-7f0556c6783c\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-p9ckq" Nov 03 13:16:40.609567 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:16:40.609539 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rhmc8\" (UniqueName: \"kubernetes.io/projected/36118654-3637-489e-b048-af4aa366f34d-kube-api-access-rhmc8\") pod \"payload-processing-55fbbbf9dc-khw76\" (UID: \"36118654-3637-489e-b048-af4aa366f34d\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-khw76" Nov 03 13:16:40.609666 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:16:40.609581 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gn5vr\" (UniqueName: \"kubernetes.io/projected/793b4955-68a1-4bf4-bcc7-7f0556c6783c-kube-api-access-gn5vr\") pod \"payload-pre-processing-7b98bb678c-p9ckq\" (UID: \"793b4955-68a1-4bf4-bcc7-7f0556c6783c\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-p9ckq" Nov 03 13:16:40.640210 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:16:40.640180 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-p9ckq" Nov 03 13:16:40.693593 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:16:40.693558 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-khw76" Nov 03 13:16:40.915473 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:16:40.915447 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-p9ckq"] Nov 03 13:16:40.917461 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:16:40.917433 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod793b4955_68a1_4bf4_bcc7_7f0556c6783c.slice/crio-3f592c5b43e686dab328ab678ea8d3edba2ce2d37f323fffb25f9e7f0a80215a WatchSource:0}: Error finding container 3f592c5b43e686dab328ab678ea8d3edba2ce2d37f323fffb25f9e7f0a80215a: Status 404 returned error can't find the container with id 3f592c5b43e686dab328ab678ea8d3edba2ce2d37f323fffb25f9e7f0a80215a Nov 03 13:16:40.976482 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:16:40.976416 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-khw76"] Nov 03 13:16:40.978677 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:16:40.978649 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36118654_3637_489e_b048_af4aa366f34d.slice/crio-9449ddb9ff43f0e40f6783dec1afa165f7620367c34ce51fc455eb398d279e4d WatchSource:0}: Error finding container 9449ddb9ff43f0e40f6783dec1afa165f7620367c34ce51fc455eb398d279e4d: Status 404 returned error can't find the container with id 9449ddb9ff43f0e40f6783dec1afa165f7620367c34ce51fc455eb398d279e4d Nov 03 13:16:41.157888 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:16:41.157848 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-khw76" event={"ID":"36118654-3637-489e-b048-af4aa366f34d","Type":"ContainerStarted","Data":"9449ddb9ff43f0e40f6783dec1afa165f7620367c34ce51fc455eb398d279e4d"} Nov 03 13:16:41.159063 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:16:41.159029 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-p9ckq" event={"ID":"793b4955-68a1-4bf4-bcc7-7f0556c6783c","Type":"ContainerStarted","Data":"3f592c5b43e686dab328ab678ea8d3edba2ce2d37f323fffb25f9e7f0a80215a"} Nov 03 13:16:44.172593 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:16:44.172549 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-khw76" event={"ID":"36118654-3637-489e-b048-af4aa366f34d","Type":"ContainerStarted","Data":"79e8003191720eef732cc991efc1dc83c996233a833c5121a47c0a85cbc0f09a"} Nov 03 13:16:44.173059 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:16:44.172708 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-55fbbbf9dc-khw76" Nov 03 13:16:44.174005 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:16:44.173977 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-p9ckq" event={"ID":"793b4955-68a1-4bf4-bcc7-7f0556c6783c","Type":"ContainerStarted","Data":"4a7e4a6fe1ecb74061d898007b6b280233b8fc1b2886d36e99e6a90ed4ee6a8a"} Nov 03 13:16:44.174145 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:16:44.174126 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-p9ckq" Nov 03 13:16:44.190742 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:16:44.190692 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-55fbbbf9dc-khw76" podStartSLOduration=1.977502069 podStartE2EDuration="4.190681099s" totalImagesPullingTime="2.21317903s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:16:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:16:44.189870072 +0000 UTC m=+484.198061349" watchObservedRunningTime="2026-11-03 13:16:44.190681099 +0000 UTC m=+484.198872424" Nov 03 13:16:44.210176 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:16:44.210129 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7b98bb678c-p9ckq" podStartSLOduration=1.9868909430000001 podStartE2EDuration="4.210116003s" totalImagesPullingTime="2.22322506s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:16:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:16:44.20946453 +0000 UTC m=+484.217655806" watchObservedRunningTime="2026-11-03 13:16:44.210116003 +0000 UTC m=+484.218307279" Nov 03 13:16:50.153739 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:16:50.153705 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-5d8dddb676-vts47" Nov 03 13:16:50.438992 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:16:50.438915 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-6f5b4db54-wxw9w"] Nov 03 13:16:50.442417 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:16:50.442400 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-6f5b4db54-wxw9w" Nov 03 13:16:50.471266 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:16:50.471238 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-6f5b4db54-wxw9w"] Nov 03 13:16:50.482855 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:16:50.482827 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h7sm6\" (UniqueName: \"kubernetes.io/projected/9af794f7-e3d1-4049-9828-499e3a49a290-kube-api-access-h7sm6\") pod \"ai-gateway-controller-6f5b4db54-wxw9w\" (UID: \"9af794f7-e3d1-4049-9828-499e3a49a290\") " pod="opendatahub/ai-gateway-controller-6f5b4db54-wxw9w" Nov 03 13:16:50.482989 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:16:50.482859 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9af794f7-e3d1-4049-9828-499e3a49a290-metrics-certs\") pod \"ai-gateway-controller-6f5b4db54-wxw9w\" (UID: \"9af794f7-e3d1-4049-9828-499e3a49a290\") " pod="opendatahub/ai-gateway-controller-6f5b4db54-wxw9w" Nov 03 13:16:50.583359 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:16:50.583323 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h7sm6\" (UniqueName: \"kubernetes.io/projected/9af794f7-e3d1-4049-9828-499e3a49a290-kube-api-access-h7sm6\") pod \"ai-gateway-controller-6f5b4db54-wxw9w\" (UID: \"9af794f7-e3d1-4049-9828-499e3a49a290\") " pod="opendatahub/ai-gateway-controller-6f5b4db54-wxw9w" Nov 03 13:16:50.583359 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:16:50.583364 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9af794f7-e3d1-4049-9828-499e3a49a290-metrics-certs\") pod \"ai-gateway-controller-6f5b4db54-wxw9w\" (UID: \"9af794f7-e3d1-4049-9828-499e3a49a290\") " pod="opendatahub/ai-gateway-controller-6f5b4db54-wxw9w" Nov 03 13:16:50.585841 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:16:50.585816 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9af794f7-e3d1-4049-9828-499e3a49a290-metrics-certs\") pod \"ai-gateway-controller-6f5b4db54-wxw9w\" (UID: \"9af794f7-e3d1-4049-9828-499e3a49a290\") " pod="opendatahub/ai-gateway-controller-6f5b4db54-wxw9w" Nov 03 13:16:50.594840 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:16:50.594811 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h7sm6\" (UniqueName: \"kubernetes.io/projected/9af794f7-e3d1-4049-9828-499e3a49a290-kube-api-access-h7sm6\") pod \"ai-gateway-controller-6f5b4db54-wxw9w\" (UID: \"9af794f7-e3d1-4049-9828-499e3a49a290\") " pod="opendatahub/ai-gateway-controller-6f5b4db54-wxw9w" Nov 03 13:16:50.754664 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:16:50.754570 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-6f5b4db54-wxw9w" Nov 03 13:16:51.026903 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:16:51.026873 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-6f5b4db54-wxw9w"] Nov 03 13:16:51.028363 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:16:51.028331 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9af794f7_e3d1_4049_9828_499e3a49a290.slice/crio-bd8d8cb57b0348d693b08925f29bae49d953190929dc6cb43c81a17af90b1006 WatchSource:0}: Error finding container bd8d8cb57b0348d693b08925f29bae49d953190929dc6cb43c81a17af90b1006: Status 404 returned error can't find the container with id bd8d8cb57b0348d693b08925f29bae49d953190929dc6cb43c81a17af90b1006 Nov 03 13:16:51.203299 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:16:51.203250 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-6f5b4db54-wxw9w" event={"ID":"9af794f7-e3d1-4049-9828-499e3a49a290","Type":"ContainerStarted","Data":"bd8d8cb57b0348d693b08925f29bae49d953190929dc6cb43c81a17af90b1006"} Nov 03 13:16:52.208622 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:16:52.208586 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-6f5b4db54-wxw9w" event={"ID":"9af794f7-e3d1-4049-9828-499e3a49a290","Type":"ContainerStarted","Data":"cc127dd26437d51ce4224d56a11f5f2c4a6c18712d8f13d89fa0125901efce7c"} Nov 03 13:16:52.209009 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:16:52.208730 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-6f5b4db54-wxw9w" Nov 03 13:16:52.227743 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:16:52.227692 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-6f5b4db54-wxw9w" podStartSLOduration=1.822765592 podStartE2EDuration="2.227679431s" totalImagesPullingTime="404.913839ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:16:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:16:52.227228686 +0000 UTC m=+492.235419988" watchObservedRunningTime="2026-11-03 13:16:52.227679431 +0000 UTC m=+492.235870749" Nov 03 13:16:55.181587 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:16:55.181557 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-p9ckq" Nov 03 13:16:55.181968 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:16:55.181719 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-55fbbbf9dc-khw76" Nov 03 13:17:03.215279 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:03.215247 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-6f5b4db54-wxw9w" Nov 03 13:17:03.261301 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:03.261247 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-5d8dddb676-vts47"] Nov 03 13:17:03.261563 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:03.261518 5024 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/ai-gateway-controller-5d8dddb676-vts47" podUID="135dd450-92b6-4e25-a00b-a28d587f03ea" containerName="manager" containerID="cri-o://88f9bcf70fe0276c3f490ed7276597ba4a8a8033ecb639ed1dec34eecf2d8a65" gracePeriod=30 Nov 03 13:17:03.507605 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:03.507576 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-5d8dddb676-vts47" Nov 03 13:17:03.589223 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:03.589190 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/135dd450-92b6-4e25-a00b-a28d587f03ea-metrics-certs\" (UniqueName: \"kubernetes.io/secret/135dd450-92b6-4e25-a00b-a28d587f03ea-metrics-certs\") pod \"135dd450-92b6-4e25-a00b-a28d587f03ea\" (UID: \"135dd450-92b6-4e25-a00b-a28d587f03ea\") " Nov 03 13:17:03.595912 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:03.589636 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/135dd450-92b6-4e25-a00b-a28d587f03ea-kube-api-access-rjndz\" (UniqueName: \"kubernetes.io/projected/135dd450-92b6-4e25-a00b-a28d587f03ea-kube-api-access-rjndz\") pod \"135dd450-92b6-4e25-a00b-a28d587f03ea\" (UID: \"135dd450-92b6-4e25-a00b-a28d587f03ea\") " Nov 03 13:17:03.596473 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:03.596440 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/135dd450-92b6-4e25-a00b-a28d587f03ea-metrics-certs" pod "135dd450-92b6-4e25-a00b-a28d587f03ea" (UID: "135dd450-92b6-4e25-a00b-a28d587f03ea"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 13:17:03.596578 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:03.596479 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/135dd450-92b6-4e25-a00b-a28d587f03ea-kube-api-access-rjndz" pod "135dd450-92b6-4e25-a00b-a28d587f03ea" (UID: "135dd450-92b6-4e25-a00b-a28d587f03ea"). InnerVolumeSpecName "kube-api-access-rjndz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:17:03.691103 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:03.691060 5024 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/135dd450-92b6-4e25-a00b-a28d587f03ea-metrics-certs\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:17:03.691103 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:03.691097 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rjndz\" (UniqueName: \"kubernetes.io/projected/135dd450-92b6-4e25-a00b-a28d587f03ea-kube-api-access-rjndz\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:17:04.259129 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:04.259091 5024 generic.go:356] "Generic (PLEG): container finished" podID="135dd450-92b6-4e25-a00b-a28d587f03ea" containerID="88f9bcf70fe0276c3f490ed7276597ba4a8a8033ecb639ed1dec34eecf2d8a65" exitCode=0 Nov 03 13:17:04.259585 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:04.259152 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-5d8dddb676-vts47" event={"ID":"135dd450-92b6-4e25-a00b-a28d587f03ea","Type":"ContainerDied","Data":"88f9bcf70fe0276c3f490ed7276597ba4a8a8033ecb639ed1dec34eecf2d8a65"} Nov 03 13:17:04.259585 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:04.259195 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-5d8dddb676-vts47" event={"ID":"135dd450-92b6-4e25-a00b-a28d587f03ea","Type":"ContainerDied","Data":"d8218490b2ca25af3c2cdb1b5955f2dd569de6475346c90cf52e664268a9c9af"} Nov 03 13:17:04.259585 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:04.259216 5024 scope.go:122] "RemoveContainer" containerID="88f9bcf70fe0276c3f490ed7276597ba4a8a8033ecb639ed1dec34eecf2d8a65" Nov 03 13:17:04.259585 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:04.259155 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-5d8dddb676-vts47" Nov 03 13:17:04.268088 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:04.267903 5024 scope.go:122] "RemoveContainer" containerID="88f9bcf70fe0276c3f490ed7276597ba4a8a8033ecb639ed1dec34eecf2d8a65" Nov 03 13:17:04.268212 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:17:04.268192 5024 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"88f9bcf70fe0276c3f490ed7276597ba4a8a8033ecb639ed1dec34eecf2d8a65\": container with ID starting with 88f9bcf70fe0276c3f490ed7276597ba4a8a8033ecb639ed1dec34eecf2d8a65 not found: ID does not exist" containerID="88f9bcf70fe0276c3f490ed7276597ba4a8a8033ecb639ed1dec34eecf2d8a65" Nov 03 13:17:04.268253 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:04.268220 5024 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88f9bcf70fe0276c3f490ed7276597ba4a8a8033ecb639ed1dec34eecf2d8a65"} err="failed to get container status \"88f9bcf70fe0276c3f490ed7276597ba4a8a8033ecb639ed1dec34eecf2d8a65\": rpc error: code = NotFound desc = could not find container \"88f9bcf70fe0276c3f490ed7276597ba4a8a8033ecb639ed1dec34eecf2d8a65\": container with ID starting with 88f9bcf70fe0276c3f490ed7276597ba4a8a8033ecb639ed1dec34eecf2d8a65 not found: ID does not exist" Nov 03 13:17:04.281941 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:04.281910 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-5d8dddb676-vts47"] Nov 03 13:17:04.284038 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:04.284017 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/ai-gateway-controller-5d8dddb676-vts47"] Nov 03 13:17:06.106989 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:06.106950 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="135dd450-92b6-4e25-a00b-a28d587f03ea" path="/var/lib/kubelet/pods/135dd450-92b6-4e25-a00b-a28d587f03ea/volumes" Nov 03 13:17:09.506928 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:17:09.506887 5024 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d7mwz" containerName="extractor" Nov 03 13:17:11.848872 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:11.848839 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-xpkhb"] Nov 03 13:17:11.849510 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:11.849199 5024 cpu_manager.go:414] "containerMap: removing container" podUID="135dd450-92b6-4e25-a00b-a28d587f03ea" containerName="manager" Nov 03 13:17:11.849510 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:11.849211 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="135dd450-92b6-4e25-a00b-a28d587f03ea" containerName="manager" Nov 03 13:17:11.849510 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:11.849297 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="135dd450-92b6-4e25-a00b-a28d587f03ea" containerName="manager" Nov 03 13:17:11.852030 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:11.852011 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-xpkhb" Nov 03 13:17:11.859443 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:11.859419 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Nov 03 13:17:11.859564 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:11.859448 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Nov 03 13:17:11.859564 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:11.859499 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-l4zxf\"" Nov 03 13:17:11.859564 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:11.859530 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Nov 03 13:17:11.867033 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:11.867000 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-xpkhb"] Nov 03 13:17:11.962969 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:11.962929 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b0c8b366-13df-493b-a519-10797d27e14c-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-xpkhb\" (UID: \"b0c8b366-13df-493b-a519-10797d27e14c\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-xpkhb" Nov 03 13:17:11.962969 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:11.962975 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b0c8b366-13df-493b-a519-10797d27e14c-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-xpkhb\" (UID: \"b0c8b366-13df-493b-a519-10797d27e14c\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-xpkhb" Nov 03 13:17:11.963178 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:11.963061 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b0c8b366-13df-493b-a519-10797d27e14c-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-xpkhb\" (UID: \"b0c8b366-13df-493b-a519-10797d27e14c\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-xpkhb" Nov 03 13:17:11.963178 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:11.963114 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b0c8b366-13df-493b-a519-10797d27e14c-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-xpkhb\" (UID: \"b0c8b366-13df-493b-a519-10797d27e14c\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-xpkhb" Nov 03 13:17:11.963178 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:11.963130 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b0c8b366-13df-493b-a519-10797d27e14c-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-xpkhb\" (UID: \"b0c8b366-13df-493b-a519-10797d27e14c\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-xpkhb" Nov 03 13:17:11.963274 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:11.963186 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pdksj\" (UniqueName: \"kubernetes.io/projected/b0c8b366-13df-493b-a519-10797d27e14c-kube-api-access-pdksj\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-xpkhb\" (UID: \"b0c8b366-13df-493b-a519-10797d27e14c\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-xpkhb" Nov 03 13:17:12.064672 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:12.064612 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b0c8b366-13df-493b-a519-10797d27e14c-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-xpkhb\" (UID: \"b0c8b366-13df-493b-a519-10797d27e14c\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-xpkhb" Nov 03 13:17:12.064672 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:12.064683 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b0c8b366-13df-493b-a519-10797d27e14c-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-xpkhb\" (UID: \"b0c8b366-13df-493b-a519-10797d27e14c\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-xpkhb" Nov 03 13:17:12.064944 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:12.064733 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b0c8b366-13df-493b-a519-10797d27e14c-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-xpkhb\" (UID: \"b0c8b366-13df-493b-a519-10797d27e14c\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-xpkhb" Nov 03 13:17:12.064944 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:12.064784 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b0c8b366-13df-493b-a519-10797d27e14c-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-xpkhb\" (UID: \"b0c8b366-13df-493b-a519-10797d27e14c\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-xpkhb" Nov 03 13:17:12.064944 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:12.064808 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b0c8b366-13df-493b-a519-10797d27e14c-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-xpkhb\" (UID: \"b0c8b366-13df-493b-a519-10797d27e14c\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-xpkhb" Nov 03 13:17:12.064944 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:12.064834 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pdksj\" (UniqueName: \"kubernetes.io/projected/b0c8b366-13df-493b-a519-10797d27e14c-kube-api-access-pdksj\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-xpkhb\" (UID: \"b0c8b366-13df-493b-a519-10797d27e14c\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-xpkhb" Nov 03 13:17:12.065173 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:12.065098 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b0c8b366-13df-493b-a519-10797d27e14c-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-xpkhb\" (UID: \"b0c8b366-13df-493b-a519-10797d27e14c\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-xpkhb" Nov 03 13:17:12.065173 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:12.065148 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b0c8b366-13df-493b-a519-10797d27e14c-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-xpkhb\" (UID: \"b0c8b366-13df-493b-a519-10797d27e14c\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-xpkhb" Nov 03 13:17:12.065312 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:12.065209 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b0c8b366-13df-493b-a519-10797d27e14c-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-xpkhb\" (UID: \"b0c8b366-13df-493b-a519-10797d27e14c\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-xpkhb" Nov 03 13:17:12.067034 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:12.067011 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b0c8b366-13df-493b-a519-10797d27e14c-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-xpkhb\" (UID: \"b0c8b366-13df-493b-a519-10797d27e14c\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-xpkhb" Nov 03 13:17:12.067426 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:12.067406 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b0c8b366-13df-493b-a519-10797d27e14c-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-xpkhb\" (UID: \"b0c8b366-13df-493b-a519-10797d27e14c\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-xpkhb" Nov 03 13:17:12.074470 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:12.074440 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pdksj\" (UniqueName: \"kubernetes.io/projected/b0c8b366-13df-493b-a519-10797d27e14c-kube-api-access-pdksj\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-xpkhb\" (UID: \"b0c8b366-13df-493b-a519-10797d27e14c\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-xpkhb" Nov 03 13:17:12.165869 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:12.165834 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-xpkhb" Nov 03 13:17:12.446142 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:12.446052 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-xpkhb"] Nov 03 13:17:12.451447 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:17:12.451414 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0c8b366_13df_493b_a519_10797d27e14c.slice/crio-77981d7d37d7942cb8f43d4208294d4dcc77b478cce0bffd12c6d7ae0df0cc3a WatchSource:0}: Error finding container 77981d7d37d7942cb8f43d4208294d4dcc77b478cce0bffd12c6d7ae0df0cc3a: Status 404 returned error can't find the container with id 77981d7d37d7942cb8f43d4208294d4dcc77b478cce0bffd12c6d7ae0df0cc3a Nov 03 13:17:13.304155 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:13.304114 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-xpkhb" event={"ID":"b0c8b366-13df-493b-a519-10797d27e14c","Type":"ContainerStarted","Data":"77981d7d37d7942cb8f43d4208294d4dcc77b478cce0bffd12c6d7ae0df0cc3a"} Nov 03 13:17:14.017428 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:14.017393 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-q6cdk"] Nov 03 13:17:14.020839 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:14.020819 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-q6cdk" Nov 03 13:17:14.029447 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:14.029152 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-q6cdk"] Nov 03 13:17:14.029933 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:14.029912 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Nov 03 13:17:14.084228 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:14.084200 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/510862f1-3997-4a93-b4a9-75e3855154dc-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-q6cdk\" (UID: \"510862f1-3997-4a93-b4a9-75e3855154dc\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-q6cdk" Nov 03 13:17:14.084392 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:14.084242 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/510862f1-3997-4a93-b4a9-75e3855154dc-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-q6cdk\" (UID: \"510862f1-3997-4a93-b4a9-75e3855154dc\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-q6cdk" Nov 03 13:17:14.084392 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:14.084274 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/510862f1-3997-4a93-b4a9-75e3855154dc-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-q6cdk\" (UID: \"510862f1-3997-4a93-b4a9-75e3855154dc\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-q6cdk" Nov 03 13:17:14.084392 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:14.084360 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2n9wj\" (UniqueName: \"kubernetes.io/projected/510862f1-3997-4a93-b4a9-75e3855154dc-kube-api-access-2n9wj\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-q6cdk\" (UID: \"510862f1-3997-4a93-b4a9-75e3855154dc\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-q6cdk" Nov 03 13:17:14.084502 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:14.084449 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/510862f1-3997-4a93-b4a9-75e3855154dc-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-q6cdk\" (UID: \"510862f1-3997-4a93-b4a9-75e3855154dc\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-q6cdk" Nov 03 13:17:14.084502 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:14.084473 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/510862f1-3997-4a93-b4a9-75e3855154dc-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-q6cdk\" (UID: \"510862f1-3997-4a93-b4a9-75e3855154dc\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-q6cdk" Nov 03 13:17:14.186074 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:14.185989 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/510862f1-3997-4a93-b4a9-75e3855154dc-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-q6cdk\" (UID: \"510862f1-3997-4a93-b4a9-75e3855154dc\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-q6cdk" Nov 03 13:17:14.186074 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:14.186064 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/510862f1-3997-4a93-b4a9-75e3855154dc-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-q6cdk\" (UID: \"510862f1-3997-4a93-b4a9-75e3855154dc\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-q6cdk" Nov 03 13:17:14.186344 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:14.186086 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/510862f1-3997-4a93-b4a9-75e3855154dc-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-q6cdk\" (UID: \"510862f1-3997-4a93-b4a9-75e3855154dc\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-q6cdk" Nov 03 13:17:14.186344 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:14.186118 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2n9wj\" (UniqueName: \"kubernetes.io/projected/510862f1-3997-4a93-b4a9-75e3855154dc-kube-api-access-2n9wj\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-q6cdk\" (UID: \"510862f1-3997-4a93-b4a9-75e3855154dc\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-q6cdk" Nov 03 13:17:14.186344 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:14.186147 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/510862f1-3997-4a93-b4a9-75e3855154dc-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-q6cdk\" (UID: \"510862f1-3997-4a93-b4a9-75e3855154dc\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-q6cdk" Nov 03 13:17:14.186694 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:14.186652 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/510862f1-3997-4a93-b4a9-75e3855154dc-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-q6cdk\" (UID: \"510862f1-3997-4a93-b4a9-75e3855154dc\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-q6cdk" Nov 03 13:17:14.187061 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:14.187034 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/510862f1-3997-4a93-b4a9-75e3855154dc-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-q6cdk\" (UID: \"510862f1-3997-4a93-b4a9-75e3855154dc\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-q6cdk" Nov 03 13:17:14.187397 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:14.187375 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/510862f1-3997-4a93-b4a9-75e3855154dc-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-q6cdk\" (UID: \"510862f1-3997-4a93-b4a9-75e3855154dc\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-q6cdk" Nov 03 13:17:14.194302 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:14.187627 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/510862f1-3997-4a93-b4a9-75e3855154dc-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-q6cdk\" (UID: \"510862f1-3997-4a93-b4a9-75e3855154dc\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-q6cdk" Nov 03 13:17:14.195242 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:14.195216 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/510862f1-3997-4a93-b4a9-75e3855154dc-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-q6cdk\" (UID: \"510862f1-3997-4a93-b4a9-75e3855154dc\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-q6cdk" Nov 03 13:17:14.196458 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:14.196436 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2n9wj\" (UniqueName: \"kubernetes.io/projected/510862f1-3997-4a93-b4a9-75e3855154dc-kube-api-access-2n9wj\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-q6cdk\" (UID: \"510862f1-3997-4a93-b4a9-75e3855154dc\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-q6cdk" Nov 03 13:17:14.197168 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:14.197151 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/510862f1-3997-4a93-b4a9-75e3855154dc-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-q6cdk\" (UID: \"510862f1-3997-4a93-b4a9-75e3855154dc\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-q6cdk" Nov 03 13:17:14.310453 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:14.310423 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-xpkhb" event={"ID":"b0c8b366-13df-493b-a519-10797d27e14c","Type":"ContainerStarted","Data":"ac017876cc43e6eb55ad2d7ba39826c4009e7b7e6c6bb6d51a2b47c78237543c"} Nov 03 13:17:14.331561 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:14.331513 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-xpkhb" podStartSLOduration=1.9214890470000001 podStartE2EDuration="3.331500446s" totalImagesPullingTime="1.410011399s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:17:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:17:14.329876054 +0000 UTC m=+514.338067333" watchObservedRunningTime="2026-11-03 13:17:14.331500446 +0000 UTC m=+514.339691722" Nov 03 13:17:14.337747 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:14.337727 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-q6cdk" Nov 03 13:17:14.618213 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:14.618184 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-q6cdk"] Nov 03 13:17:14.620564 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:17:14.620534 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod510862f1_3997_4a93_b4a9_75e3855154dc.slice/crio-08f48e3ba31ac33d1fecc223f2f668cae40e600c9989d6629269c7a0c8d5ca7a WatchSource:0}: Error finding container 08f48e3ba31ac33d1fecc223f2f668cae40e600c9989d6629269c7a0c8d5ca7a: Status 404 returned error can't find the container with id 08f48e3ba31ac33d1fecc223f2f668cae40e600c9989d6629269c7a0c8d5ca7a Nov 03 13:17:15.168962 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:15.168812 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-xpkhb" Nov 03 13:17:15.316224 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:15.316131 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-q6cdk" event={"ID":"510862f1-3997-4a93-b4a9-75e3855154dc","Type":"ContainerStarted","Data":"b472b12e57496ef02e289c66c430ac9c31ec576141ff17676c550c19a2203a9f"} Nov 03 13:17:15.316224 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:15.316189 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-q6cdk" event={"ID":"510862f1-3997-4a93-b4a9-75e3855154dc","Type":"ContainerStarted","Data":"08f48e3ba31ac33d1fecc223f2f668cae40e600c9989d6629269c7a0c8d5ca7a"} Nov 03 13:17:15.338617 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:15.338577 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-q6cdk" Nov 03 13:17:15.343875 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:15.343815 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-q6cdk" podStartSLOduration=1.040402451 podStartE2EDuration="1.343797834s" totalImagesPullingTime="303.395383ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:17:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:17:15.343378305 +0000 UTC m=+515.351569582" watchObservedRunningTime="2026-11-03 13:17:15.343797834 +0000 UTC m=+515.351989111" Nov 03 13:17:18.221972 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:18.221939 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8h566d"] Nov 03 13:17:18.225632 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:18.225612 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8h566d" Nov 03 13:17:18.238423 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:18.238400 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8h566d"] Nov 03 13:17:18.239788 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:18.239762 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Nov 03 13:17:18.327588 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:18.327547 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6428a275-67b4-43e6-be9c-a16728516866-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8h566d\" (UID: \"6428a275-67b4-43e6-be9c-a16728516866\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8h566d" Nov 03 13:17:18.327784 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:18.327595 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6428a275-67b4-43e6-be9c-a16728516866-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8h566d\" (UID: \"6428a275-67b4-43e6-be9c-a16728516866\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8h566d" Nov 03 13:17:18.327784 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:18.327622 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6428a275-67b4-43e6-be9c-a16728516866-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8h566d\" (UID: \"6428a275-67b4-43e6-be9c-a16728516866\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8h566d" Nov 03 13:17:18.327784 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:18.327718 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6428a275-67b4-43e6-be9c-a16728516866-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8h566d\" (UID: \"6428a275-67b4-43e6-be9c-a16728516866\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8h566d" Nov 03 13:17:18.327923 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:18.327804 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6428a275-67b4-43e6-be9c-a16728516866-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8h566d\" (UID: \"6428a275-67b4-43e6-be9c-a16728516866\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8h566d" Nov 03 13:17:18.327923 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:18.327862 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-svfpp\" (UniqueName: \"kubernetes.io/projected/6428a275-67b4-43e6-be9c-a16728516866-kube-api-access-svfpp\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8h566d\" (UID: \"6428a275-67b4-43e6-be9c-a16728516866\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8h566d" Nov 03 13:17:18.428775 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:18.428740 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6428a275-67b4-43e6-be9c-a16728516866-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8h566d\" (UID: \"6428a275-67b4-43e6-be9c-a16728516866\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8h566d" Nov 03 13:17:18.428775 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:18.428776 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6428a275-67b4-43e6-be9c-a16728516866-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8h566d\" (UID: \"6428a275-67b4-43e6-be9c-a16728516866\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8h566d" Nov 03 13:17:18.429008 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:18.428793 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6428a275-67b4-43e6-be9c-a16728516866-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8h566d\" (UID: \"6428a275-67b4-43e6-be9c-a16728516866\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8h566d" Nov 03 13:17:18.429008 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:18.428811 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6428a275-67b4-43e6-be9c-a16728516866-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8h566d\" (UID: \"6428a275-67b4-43e6-be9c-a16728516866\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8h566d" Nov 03 13:17:18.429008 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:18.428828 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6428a275-67b4-43e6-be9c-a16728516866-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8h566d\" (UID: \"6428a275-67b4-43e6-be9c-a16728516866\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8h566d" Nov 03 13:17:18.429008 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:18.428855 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svfpp\" (UniqueName: \"kubernetes.io/projected/6428a275-67b4-43e6-be9c-a16728516866-kube-api-access-svfpp\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8h566d\" (UID: \"6428a275-67b4-43e6-be9c-a16728516866\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8h566d" Nov 03 13:17:18.429319 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:18.429276 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6428a275-67b4-43e6-be9c-a16728516866-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8h566d\" (UID: \"6428a275-67b4-43e6-be9c-a16728516866\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8h566d" Nov 03 13:17:18.429319 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:18.429309 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6428a275-67b4-43e6-be9c-a16728516866-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8h566d\" (UID: \"6428a275-67b4-43e6-be9c-a16728516866\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8h566d" Nov 03 13:17:18.429444 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:18.429354 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6428a275-67b4-43e6-be9c-a16728516866-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8h566d\" (UID: \"6428a275-67b4-43e6-be9c-a16728516866\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8h566d" Nov 03 13:17:18.431043 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:18.431019 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6428a275-67b4-43e6-be9c-a16728516866-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8h566d\" (UID: \"6428a275-67b4-43e6-be9c-a16728516866\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8h566d" Nov 03 13:17:18.431352 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:18.431332 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6428a275-67b4-43e6-be9c-a16728516866-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8h566d\" (UID: \"6428a275-67b4-43e6-be9c-a16728516866\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8h566d" Nov 03 13:17:18.442229 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:18.442204 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-svfpp\" (UniqueName: \"kubernetes.io/projected/6428a275-67b4-43e6-be9c-a16728516866-kube-api-access-svfpp\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8h566d\" (UID: \"6428a275-67b4-43e6-be9c-a16728516866\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8h566d" Nov 03 13:17:18.538474 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:18.538393 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8h566d" Nov 03 13:17:18.811631 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:18.811603 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8h566d"] Nov 03 13:17:18.814038 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:17:18.814009 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6428a275_67b4_43e6_be9c_a16728516866.slice/crio-766cbb005553fc69ae5a32e8e401dcd645ce93dc0830662bc985c76503a9b96e WatchSource:0}: Error finding container 766cbb005553fc69ae5a32e8e401dcd645ce93dc0830662bc985c76503a9b96e: Status 404 returned error can't find the container with id 766cbb005553fc69ae5a32e8e401dcd645ce93dc0830662bc985c76503a9b96e Nov 03 13:17:18.961057 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:18.961026 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-ghn7b"] Nov 03 13:17:18.965871 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:18.965843 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-ghn7b" Nov 03 13:17:18.974911 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:18.974351 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"echo-simulator-kserve-self-signed-certs\"" Nov 03 13:17:18.975041 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:18.974934 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-ghn7b"] Nov 03 13:17:19.034727 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:19.034659 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0cf9c918-efa8-490a-8a25-069337ee09b0-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-ghn7b\" (UID: \"0cf9c918-efa8-490a-8a25-069337ee09b0\") " pod="llm/echo-simulator-kserve-64b6f8f96c-ghn7b" Nov 03 13:17:19.034911 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:19.034788 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0cf9c918-efa8-490a-8a25-069337ee09b0-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-ghn7b\" (UID: \"0cf9c918-efa8-490a-8a25-069337ee09b0\") " pod="llm/echo-simulator-kserve-64b6f8f96c-ghn7b" Nov 03 13:17:19.034911 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:19.034826 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7zl8l\" (UniqueName: \"kubernetes.io/projected/0cf9c918-efa8-490a-8a25-069337ee09b0-kube-api-access-7zl8l\") pod \"echo-simulator-kserve-64b6f8f96c-ghn7b\" (UID: \"0cf9c918-efa8-490a-8a25-069337ee09b0\") " pod="llm/echo-simulator-kserve-64b6f8f96c-ghn7b" Nov 03 13:17:19.034911 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:19.034870 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0cf9c918-efa8-490a-8a25-069337ee09b0-home\") pod \"echo-simulator-kserve-64b6f8f96c-ghn7b\" (UID: \"0cf9c918-efa8-490a-8a25-069337ee09b0\") " pod="llm/echo-simulator-kserve-64b6f8f96c-ghn7b" Nov 03 13:17:19.035122 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:19.034927 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0cf9c918-efa8-490a-8a25-069337ee09b0-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-ghn7b\" (UID: \"0cf9c918-efa8-490a-8a25-069337ee09b0\") " pod="llm/echo-simulator-kserve-64b6f8f96c-ghn7b" Nov 03 13:17:19.035122 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:19.034945 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0cf9c918-efa8-490a-8a25-069337ee09b0-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-ghn7b\" (UID: \"0cf9c918-efa8-490a-8a25-069337ee09b0\") " pod="llm/echo-simulator-kserve-64b6f8f96c-ghn7b" Nov 03 13:17:19.135767 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:19.135743 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0cf9c918-efa8-490a-8a25-069337ee09b0-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-ghn7b\" (UID: \"0cf9c918-efa8-490a-8a25-069337ee09b0\") " pod="llm/echo-simulator-kserve-64b6f8f96c-ghn7b" Nov 03 13:17:19.135889 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:19.135779 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0cf9c918-efa8-490a-8a25-069337ee09b0-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-ghn7b\" (UID: \"0cf9c918-efa8-490a-8a25-069337ee09b0\") " pod="llm/echo-simulator-kserve-64b6f8f96c-ghn7b" Nov 03 13:17:19.135979 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:19.135955 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0cf9c918-efa8-490a-8a25-069337ee09b0-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-ghn7b\" (UID: \"0cf9c918-efa8-490a-8a25-069337ee09b0\") " pod="llm/echo-simulator-kserve-64b6f8f96c-ghn7b" Nov 03 13:17:19.136121 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:19.136036 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0cf9c918-efa8-490a-8a25-069337ee09b0-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-ghn7b\" (UID: \"0cf9c918-efa8-490a-8a25-069337ee09b0\") " pod="llm/echo-simulator-kserve-64b6f8f96c-ghn7b" Nov 03 13:17:19.136121 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:19.136072 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zl8l\" (UniqueName: \"kubernetes.io/projected/0cf9c918-efa8-490a-8a25-069337ee09b0-kube-api-access-7zl8l\") pod \"echo-simulator-kserve-64b6f8f96c-ghn7b\" (UID: \"0cf9c918-efa8-490a-8a25-069337ee09b0\") " pod="llm/echo-simulator-kserve-64b6f8f96c-ghn7b" Nov 03 13:17:19.136504 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:19.136150 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0cf9c918-efa8-490a-8a25-069337ee09b0-home\") pod \"echo-simulator-kserve-64b6f8f96c-ghn7b\" (UID: \"0cf9c918-efa8-490a-8a25-069337ee09b0\") " pod="llm/echo-simulator-kserve-64b6f8f96c-ghn7b" Nov 03 13:17:19.136595 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:19.136520 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0cf9c918-efa8-490a-8a25-069337ee09b0-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-ghn7b\" (UID: \"0cf9c918-efa8-490a-8a25-069337ee09b0\") " pod="llm/echo-simulator-kserve-64b6f8f96c-ghn7b" Nov 03 13:17:19.136595 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:19.136374 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0cf9c918-efa8-490a-8a25-069337ee09b0-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-ghn7b\" (UID: \"0cf9c918-efa8-490a-8a25-069337ee09b0\") " pod="llm/echo-simulator-kserve-64b6f8f96c-ghn7b" Nov 03 13:17:19.136595 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:19.136448 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0cf9c918-efa8-490a-8a25-069337ee09b0-home\") pod \"echo-simulator-kserve-64b6f8f96c-ghn7b\" (UID: \"0cf9c918-efa8-490a-8a25-069337ee09b0\") " pod="llm/echo-simulator-kserve-64b6f8f96c-ghn7b" Nov 03 13:17:19.142854 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:19.138621 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0cf9c918-efa8-490a-8a25-069337ee09b0-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-ghn7b\" (UID: \"0cf9c918-efa8-490a-8a25-069337ee09b0\") " pod="llm/echo-simulator-kserve-64b6f8f96c-ghn7b" Nov 03 13:17:19.142854 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:19.139179 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0cf9c918-efa8-490a-8a25-069337ee09b0-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-ghn7b\" (UID: \"0cf9c918-efa8-490a-8a25-069337ee09b0\") " pod="llm/echo-simulator-kserve-64b6f8f96c-ghn7b" Nov 03 13:17:19.146648 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:19.146623 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zl8l\" (UniqueName: \"kubernetes.io/projected/0cf9c918-efa8-490a-8a25-069337ee09b0-kube-api-access-7zl8l\") pod \"echo-simulator-kserve-64b6f8f96c-ghn7b\" (UID: \"0cf9c918-efa8-490a-8a25-069337ee09b0\") " pod="llm/echo-simulator-kserve-64b6f8f96c-ghn7b" Nov 03 13:17:19.282526 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:19.282462 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-ghn7b" Nov 03 13:17:19.349996 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:19.349897 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8h566d" event={"ID":"6428a275-67b4-43e6-be9c-a16728516866","Type":"ContainerStarted","Data":"573fc138c76f9814d6bd616e5049e686eff16211f71f830e612829c5ab073bd2"} Nov 03 13:17:19.349996 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:19.349963 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8h566d" event={"ID":"6428a275-67b4-43e6-be9c-a16728516866","Type":"ContainerStarted","Data":"766cbb005553fc69ae5a32e8e401dcd645ce93dc0830662bc985c76503a9b96e"} Nov 03 13:17:19.375105 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:19.375006 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8h566d" podStartSLOduration=1.0651906819999999 podStartE2EDuration="1.374987957s" totalImagesPullingTime="309.797275ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:17:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:17:19.373107076 +0000 UTC m=+519.381298354" watchObservedRunningTime="2026-11-03 13:17:19.374987957 +0000 UTC m=+519.383179236" Nov 03 13:17:19.539518 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:19.539480 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8h566d" Nov 03 13:17:19.564874 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:19.564850 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-ghn7b"] Nov 03 13:17:19.567004 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:17:19.566976 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0cf9c918_efa8_490a_8a25_069337ee09b0.slice/crio-83dbc70f4932dc75aec93b1b555abe0c44332fc7bf0ebe3d2fee0d91197c66a4 WatchSource:0}: Error finding container 83dbc70f4932dc75aec93b1b555abe0c44332fc7bf0ebe3d2fee0d91197c66a4: Status 404 returned error can't find the container with id 83dbc70f4932dc75aec93b1b555abe0c44332fc7bf0ebe3d2fee0d91197c66a4 Nov 03 13:17:20.357468 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:20.357435 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-ghn7b" event={"ID":"0cf9c918-efa8-490a-8a25-069337ee09b0","Type":"ContainerStarted","Data":"83dbc70f4932dc75aec93b1b555abe0c44332fc7bf0ebe3d2fee0d91197c66a4"} Nov 03 13:17:22.166272 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:22.166238 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-xpkhb" Nov 03 13:17:22.179195 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:22.179168 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-xpkhb" Nov 03 13:17:22.367988 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:22.367947 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-ghn7b" event={"ID":"0cf9c918-efa8-490a-8a25-069337ee09b0","Type":"ContainerStarted","Data":"684a9c9e352a5a6c3b6e2cd4349c065e1589104ad85dc750fce72a17abc2d2d9"} Nov 03 13:17:22.379734 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:22.379703 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-xpkhb" Nov 03 13:17:22.389131 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:22.389085 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/echo-simulator-kserve-64b6f8f96c-ghn7b" podStartSLOduration=2.5635944840000002 podStartE2EDuration="4.389072464s" totalImagesPullingTime="1.82547798s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:17:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:17:22.388407441 +0000 UTC m=+522.396598721" watchObservedRunningTime="2026-11-03 13:17:22.389072464 +0000 UTC m=+522.397263741" Nov 03 13:17:23.282829 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:23.282787 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/echo-simulator-kserve-64b6f8f96c-ghn7b" Nov 03 13:17:24.003072 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:24.003028 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-84587756dc-xtsb8"] Nov 03 13:17:24.006873 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:24.006847 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-xtsb8" Nov 03 13:17:24.018655 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:24.018607 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-nxhjw\"" Nov 03 13:17:24.020585 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:24.020563 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-84587756dc-xtsb8"] Nov 03 13:17:24.087567 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:24.087527 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b46130ce-1911-475e-8ec7-5204881eb5ea-credential-socket\") pod \"maas-default-gateway-openshift-default-84587756dc-xtsb8\" (UID: \"b46130ce-1911-475e-8ec7-5204881eb5ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-xtsb8" Nov 03 13:17:24.087721 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:24.087577 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b46130ce-1911-475e-8ec7-5204881eb5ea-istio-token\") pod \"maas-default-gateway-openshift-default-84587756dc-xtsb8\" (UID: \"b46130ce-1911-475e-8ec7-5204881eb5ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-xtsb8" Nov 03 13:17:24.087721 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:24.087633 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b46130ce-1911-475e-8ec7-5204881eb5ea-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-84587756dc-xtsb8\" (UID: \"b46130ce-1911-475e-8ec7-5204881eb5ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-xtsb8" Nov 03 13:17:24.087721 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:24.087687 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8q4b\" (UniqueName: \"kubernetes.io/projected/b46130ce-1911-475e-8ec7-5204881eb5ea-kube-api-access-q8q4b\") pod \"maas-default-gateway-openshift-default-84587756dc-xtsb8\" (UID: \"b46130ce-1911-475e-8ec7-5204881eb5ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-xtsb8" Nov 03 13:17:24.087873 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:24.087752 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b46130ce-1911-475e-8ec7-5204881eb5ea-workload-certs\") pod \"maas-default-gateway-openshift-default-84587756dc-xtsb8\" (UID: \"b46130ce-1911-475e-8ec7-5204881eb5ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-xtsb8" Nov 03 13:17:24.087873 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:24.087786 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b46130ce-1911-475e-8ec7-5204881eb5ea-istio-data\") pod \"maas-default-gateway-openshift-default-84587756dc-xtsb8\" (UID: \"b46130ce-1911-475e-8ec7-5204881eb5ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-xtsb8" Nov 03 13:17:24.087873 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:24.087821 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b46130ce-1911-475e-8ec7-5204881eb5ea-istio-envoy\") pod \"maas-default-gateway-openshift-default-84587756dc-xtsb8\" (UID: \"b46130ce-1911-475e-8ec7-5204881eb5ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-xtsb8" Nov 03 13:17:24.087978 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:24.087938 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b46130ce-1911-475e-8ec7-5204881eb5ea-workload-socket\") pod \"maas-default-gateway-openshift-default-84587756dc-xtsb8\" (UID: \"b46130ce-1911-475e-8ec7-5204881eb5ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-xtsb8" Nov 03 13:17:24.087978 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:24.087967 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b46130ce-1911-475e-8ec7-5204881eb5ea-istio-podinfo\") pod \"maas-default-gateway-openshift-default-84587756dc-xtsb8\" (UID: \"b46130ce-1911-475e-8ec7-5204881eb5ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-xtsb8" Nov 03 13:17:24.189011 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:24.188974 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b46130ce-1911-475e-8ec7-5204881eb5ea-workload-certs\") pod \"maas-default-gateway-openshift-default-84587756dc-xtsb8\" (UID: \"b46130ce-1911-475e-8ec7-5204881eb5ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-xtsb8" Nov 03 13:17:24.189151 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:24.189019 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b46130ce-1911-475e-8ec7-5204881eb5ea-istio-data\") pod \"maas-default-gateway-openshift-default-84587756dc-xtsb8\" (UID: \"b46130ce-1911-475e-8ec7-5204881eb5ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-xtsb8" Nov 03 13:17:24.189151 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:24.189051 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b46130ce-1911-475e-8ec7-5204881eb5ea-istio-envoy\") pod \"maas-default-gateway-openshift-default-84587756dc-xtsb8\" (UID: \"b46130ce-1911-475e-8ec7-5204881eb5ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-xtsb8" Nov 03 13:17:24.189151 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:24.189088 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b46130ce-1911-475e-8ec7-5204881eb5ea-workload-socket\") pod \"maas-default-gateway-openshift-default-84587756dc-xtsb8\" (UID: \"b46130ce-1911-475e-8ec7-5204881eb5ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-xtsb8" Nov 03 13:17:24.189151 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:24.189124 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b46130ce-1911-475e-8ec7-5204881eb5ea-istio-podinfo\") pod \"maas-default-gateway-openshift-default-84587756dc-xtsb8\" (UID: \"b46130ce-1911-475e-8ec7-5204881eb5ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-xtsb8" Nov 03 13:17:24.189383 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:24.189213 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b46130ce-1911-475e-8ec7-5204881eb5ea-credential-socket\") pod \"maas-default-gateway-openshift-default-84587756dc-xtsb8\" (UID: \"b46130ce-1911-475e-8ec7-5204881eb5ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-xtsb8" Nov 03 13:17:24.189383 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:24.189237 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b46130ce-1911-475e-8ec7-5204881eb5ea-istio-token\") pod \"maas-default-gateway-openshift-default-84587756dc-xtsb8\" (UID: \"b46130ce-1911-475e-8ec7-5204881eb5ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-xtsb8" Nov 03 13:17:24.189383 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:24.189262 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b46130ce-1911-475e-8ec7-5204881eb5ea-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-84587756dc-xtsb8\" (UID: \"b46130ce-1911-475e-8ec7-5204881eb5ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-xtsb8" Nov 03 13:17:24.189383 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:24.189313 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8q4b\" (UniqueName: \"kubernetes.io/projected/b46130ce-1911-475e-8ec7-5204881eb5ea-kube-api-access-q8q4b\") pod \"maas-default-gateway-openshift-default-84587756dc-xtsb8\" (UID: \"b46130ce-1911-475e-8ec7-5204881eb5ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-xtsb8" Nov 03 13:17:24.189564 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:24.189452 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b46130ce-1911-475e-8ec7-5204881eb5ea-workload-certs\") pod \"maas-default-gateway-openshift-default-84587756dc-xtsb8\" (UID: \"b46130ce-1911-475e-8ec7-5204881eb5ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-xtsb8" Nov 03 13:17:24.189564 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:24.189474 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b46130ce-1911-475e-8ec7-5204881eb5ea-istio-data\") pod \"maas-default-gateway-openshift-default-84587756dc-xtsb8\" (UID: \"b46130ce-1911-475e-8ec7-5204881eb5ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-xtsb8" Nov 03 13:17:24.189791 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:24.189761 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b46130ce-1911-475e-8ec7-5204881eb5ea-workload-socket\") pod \"maas-default-gateway-openshift-default-84587756dc-xtsb8\" (UID: \"b46130ce-1911-475e-8ec7-5204881eb5ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-xtsb8" Nov 03 13:17:24.189899 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:24.189818 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b46130ce-1911-475e-8ec7-5204881eb5ea-credential-socket\") pod \"maas-default-gateway-openshift-default-84587756dc-xtsb8\" (UID: \"b46130ce-1911-475e-8ec7-5204881eb5ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-xtsb8" Nov 03 13:17:24.190001 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:24.189978 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b46130ce-1911-475e-8ec7-5204881eb5ea-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-84587756dc-xtsb8\" (UID: \"b46130ce-1911-475e-8ec7-5204881eb5ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-xtsb8" Nov 03 13:17:24.191519 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:24.191501 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b46130ce-1911-475e-8ec7-5204881eb5ea-istio-envoy\") pod \"maas-default-gateway-openshift-default-84587756dc-xtsb8\" (UID: \"b46130ce-1911-475e-8ec7-5204881eb5ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-xtsb8" Nov 03 13:17:24.192076 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:24.192053 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b46130ce-1911-475e-8ec7-5204881eb5ea-istio-podinfo\") pod \"maas-default-gateway-openshift-default-84587756dc-xtsb8\" (UID: \"b46130ce-1911-475e-8ec7-5204881eb5ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-xtsb8" Nov 03 13:17:24.198237 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:24.198209 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b46130ce-1911-475e-8ec7-5204881eb5ea-istio-token\") pod \"maas-default-gateway-openshift-default-84587756dc-xtsb8\" (UID: \"b46130ce-1911-475e-8ec7-5204881eb5ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-xtsb8" Nov 03 13:17:24.198452 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:24.198435 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8q4b\" (UniqueName: \"kubernetes.io/projected/b46130ce-1911-475e-8ec7-5204881eb5ea-kube-api-access-q8q4b\") pod \"maas-default-gateway-openshift-default-84587756dc-xtsb8\" (UID: \"b46130ce-1911-475e-8ec7-5204881eb5ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-xtsb8" Nov 03 13:17:24.319530 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:24.319471 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-xtsb8" Nov 03 13:17:24.338600 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:24.338576 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-q6cdk" Nov 03 13:17:24.353154 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:24.353132 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-q6cdk" Nov 03 13:17:24.400367 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:24.400341 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-q6cdk" Nov 03 13:17:24.601388 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:24.601336 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-84587756dc-xtsb8"] Nov 03 13:17:24.603242 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:17:24.603212 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb46130ce_1911_475e_8ec7_5204881eb5ea.slice/crio-4009da59fe60b3d6bcc8fb1673cd92346a5af2a6272d483784d4af75572ef4c5 WatchSource:0}: Error finding container 4009da59fe60b3d6bcc8fb1673cd92346a5af2a6272d483784d4af75572ef4c5: Status 404 returned error can't find the container with id 4009da59fe60b3d6bcc8fb1673cd92346a5af2a6272d483784d4af75572ef4c5 Nov 03 13:17:25.389781 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:25.389748 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-xtsb8" event={"ID":"b46130ce-1911-475e-8ec7-5204881eb5ea","Type":"ContainerStarted","Data":"4009da59fe60b3d6bcc8fb1673cd92346a5af2a6272d483784d4af75572ef4c5"} Nov 03 13:17:28.538739 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:28.538698 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8h566d" Nov 03 13:17:28.552824 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:28.552791 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8h566d" Nov 03 13:17:29.283423 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:29.283388 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/echo-simulator-kserve-64b6f8f96c-ghn7b" Nov 03 13:17:29.297318 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:29.297270 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/echo-simulator-kserve-64b6f8f96c-ghn7b" Nov 03 13:17:29.423632 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:29.423599 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8h566d" Nov 03 13:17:29.424494 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:29.424469 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/echo-simulator-kserve-64b6f8f96c-ghn7b" Nov 03 13:17:29.824310 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:29.824257 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:17:29.824696 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:29.824351 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:17:29.824696 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:29.824399 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:17:30.417043 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:30.416997 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-xtsb8" event={"ID":"b46130ce-1911-475e-8ec7-5204881eb5ea","Type":"ContainerStarted","Data":"e525aba39eee600786bacf1d3950338c7cc4baaa122bc1eb2d7be06c938f0d0d"} Nov 03 13:17:30.443086 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:30.443030 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-xtsb8" podStartSLOduration=2.225041822 podStartE2EDuration="7.443017195s" totalImagesPullingTime="5.217975373s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:17:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:17:30.440680269 +0000 UTC m=+530.448871547" watchObservedRunningTime="2026-11-03 13:17:30.443017195 +0000 UTC m=+530.451208473" Nov 03 13:17:31.319825 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:31.319780 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-xtsb8" Nov 03 13:17:31.321492 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:31.321461 5024 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-xtsb8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body= Nov 03 13:17:31.321636 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:31.321518 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-xtsb8" podUID="b46130ce-1911-475e-8ec7-5204881eb5ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" Nov 03 13:17:32.320614 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:32.320576 5024 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-xtsb8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body= Nov 03 13:17:32.320999 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:32.320633 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-xtsb8" podUID="b46130ce-1911-475e-8ec7-5204881eb5ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" Nov 03 13:17:33.320239 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:33.320203 5024 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-xtsb8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body= Nov 03 13:17:33.320450 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:33.320259 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-xtsb8" podUID="b46130ce-1911-475e-8ec7-5204881eb5ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" Nov 03 13:17:34.320821 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:34.320780 5024 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-xtsb8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body= Nov 03 13:17:34.321187 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:34.320836 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-xtsb8" podUID="b46130ce-1911-475e-8ec7-5204881eb5ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" Nov 03 13:17:35.320059 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:35.320025 5024 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-xtsb8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body= Nov 03 13:17:35.320236 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:35.320075 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-xtsb8" podUID="b46130ce-1911-475e-8ec7-5204881eb5ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" Nov 03 13:17:36.320082 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:36.320049 5024 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-xtsb8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body= Nov 03 13:17:36.320474 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:36.320098 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-xtsb8" podUID="b46130ce-1911-475e-8ec7-5204881eb5ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" Nov 03 13:17:38.018133 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:38.018083 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-xtsb8" Nov 03 13:17:38.018872 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:38.018843 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-xtsb8" Nov 03 13:17:38.019663 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:38.019637 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-xtsb8" Nov 03 13:17:39.035860 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:39.035827 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-84587756dc-h2ns8"] Nov 03 13:17:39.040968 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:39.040944 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-h2ns8" Nov 03 13:17:39.051699 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:39.051671 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-84587756dc-h2ns8"] Nov 03 13:17:39.053984 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:39.053953 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/155ba865-dcfd-4203-82b9-5399ca58d988-workload-certs\") pod \"maas-default-gateway-openshift-default-84587756dc-h2ns8\" (UID: \"155ba865-dcfd-4203-82b9-5399ca58d988\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-h2ns8" Nov 03 13:17:39.054089 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:39.053997 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/155ba865-dcfd-4203-82b9-5399ca58d988-istio-podinfo\") pod \"maas-default-gateway-openshift-default-84587756dc-h2ns8\" (UID: \"155ba865-dcfd-4203-82b9-5399ca58d988\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-h2ns8" Nov 03 13:17:39.054089 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:39.054032 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/155ba865-dcfd-4203-82b9-5399ca58d988-istio-envoy\") pod \"maas-default-gateway-openshift-default-84587756dc-h2ns8\" (UID: \"155ba865-dcfd-4203-82b9-5399ca58d988\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-h2ns8" Nov 03 13:17:39.054089 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:39.054061 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/155ba865-dcfd-4203-82b9-5399ca58d988-istio-token\") pod \"maas-default-gateway-openshift-default-84587756dc-h2ns8\" (UID: \"155ba865-dcfd-4203-82b9-5399ca58d988\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-h2ns8" Nov 03 13:17:39.054274 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:39.054090 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sqbdt\" (UniqueName: \"kubernetes.io/projected/155ba865-dcfd-4203-82b9-5399ca58d988-kube-api-access-sqbdt\") pod \"maas-default-gateway-openshift-default-84587756dc-h2ns8\" (UID: \"155ba865-dcfd-4203-82b9-5399ca58d988\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-h2ns8" Nov 03 13:17:39.054274 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:39.054121 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/155ba865-dcfd-4203-82b9-5399ca58d988-workload-socket\") pod \"maas-default-gateway-openshift-default-84587756dc-h2ns8\" (UID: \"155ba865-dcfd-4203-82b9-5399ca58d988\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-h2ns8" Nov 03 13:17:39.054274 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:39.054154 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/155ba865-dcfd-4203-82b9-5399ca58d988-istio-data\") pod \"maas-default-gateway-openshift-default-84587756dc-h2ns8\" (UID: \"155ba865-dcfd-4203-82b9-5399ca58d988\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-h2ns8" Nov 03 13:17:39.054274 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:39.054192 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/155ba865-dcfd-4203-82b9-5399ca58d988-credential-socket\") pod \"maas-default-gateway-openshift-default-84587756dc-h2ns8\" (UID: \"155ba865-dcfd-4203-82b9-5399ca58d988\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-h2ns8" Nov 03 13:17:39.054274 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:39.054245 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/155ba865-dcfd-4203-82b9-5399ca58d988-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-84587756dc-h2ns8\" (UID: \"155ba865-dcfd-4203-82b9-5399ca58d988\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-h2ns8" Nov 03 13:17:39.155543 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:39.155505 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/155ba865-dcfd-4203-82b9-5399ca58d988-workload-certs\") pod \"maas-default-gateway-openshift-default-84587756dc-h2ns8\" (UID: \"155ba865-dcfd-4203-82b9-5399ca58d988\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-h2ns8" Nov 03 13:17:39.155543 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:39.155541 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/155ba865-dcfd-4203-82b9-5399ca58d988-istio-podinfo\") pod \"maas-default-gateway-openshift-default-84587756dc-h2ns8\" (UID: \"155ba865-dcfd-4203-82b9-5399ca58d988\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-h2ns8" Nov 03 13:17:39.155782 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:39.155559 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/155ba865-dcfd-4203-82b9-5399ca58d988-istio-envoy\") pod \"maas-default-gateway-openshift-default-84587756dc-h2ns8\" (UID: \"155ba865-dcfd-4203-82b9-5399ca58d988\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-h2ns8" Nov 03 13:17:39.155782 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:39.155579 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/155ba865-dcfd-4203-82b9-5399ca58d988-istio-token\") pod \"maas-default-gateway-openshift-default-84587756dc-h2ns8\" (UID: \"155ba865-dcfd-4203-82b9-5399ca58d988\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-h2ns8" Nov 03 13:17:39.155782 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:39.155607 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqbdt\" (UniqueName: \"kubernetes.io/projected/155ba865-dcfd-4203-82b9-5399ca58d988-kube-api-access-sqbdt\") pod \"maas-default-gateway-openshift-default-84587756dc-h2ns8\" (UID: \"155ba865-dcfd-4203-82b9-5399ca58d988\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-h2ns8" Nov 03 13:17:39.155782 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:39.155634 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/155ba865-dcfd-4203-82b9-5399ca58d988-workload-socket\") pod \"maas-default-gateway-openshift-default-84587756dc-h2ns8\" (UID: \"155ba865-dcfd-4203-82b9-5399ca58d988\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-h2ns8" Nov 03 13:17:39.155782 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:39.155679 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/155ba865-dcfd-4203-82b9-5399ca58d988-istio-data\") pod \"maas-default-gateway-openshift-default-84587756dc-h2ns8\" (UID: \"155ba865-dcfd-4203-82b9-5399ca58d988\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-h2ns8" Nov 03 13:17:39.155782 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:39.155732 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/155ba865-dcfd-4203-82b9-5399ca58d988-credential-socket\") pod \"maas-default-gateway-openshift-default-84587756dc-h2ns8\" (UID: \"155ba865-dcfd-4203-82b9-5399ca58d988\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-h2ns8" Nov 03 13:17:39.156087 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:39.155815 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/155ba865-dcfd-4203-82b9-5399ca58d988-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-84587756dc-h2ns8\" (UID: \"155ba865-dcfd-4203-82b9-5399ca58d988\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-h2ns8" Nov 03 13:17:39.156087 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:39.155926 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/155ba865-dcfd-4203-82b9-5399ca58d988-workload-certs\") pod \"maas-default-gateway-openshift-default-84587756dc-h2ns8\" (UID: \"155ba865-dcfd-4203-82b9-5399ca58d988\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-h2ns8" Nov 03 13:17:39.156442 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:39.156416 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/155ba865-dcfd-4203-82b9-5399ca58d988-istio-data\") pod \"maas-default-gateway-openshift-default-84587756dc-h2ns8\" (UID: \"155ba865-dcfd-4203-82b9-5399ca58d988\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-h2ns8" Nov 03 13:17:39.156562 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:39.156465 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/155ba865-dcfd-4203-82b9-5399ca58d988-workload-socket\") pod \"maas-default-gateway-openshift-default-84587756dc-h2ns8\" (UID: \"155ba865-dcfd-4203-82b9-5399ca58d988\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-h2ns8" Nov 03 13:17:39.156562 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:39.156545 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/155ba865-dcfd-4203-82b9-5399ca58d988-credential-socket\") pod \"maas-default-gateway-openshift-default-84587756dc-h2ns8\" (UID: \"155ba865-dcfd-4203-82b9-5399ca58d988\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-h2ns8" Nov 03 13:17:39.156668 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:39.156615 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/155ba865-dcfd-4203-82b9-5399ca58d988-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-84587756dc-h2ns8\" (UID: \"155ba865-dcfd-4203-82b9-5399ca58d988\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-h2ns8" Nov 03 13:17:39.158478 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:39.158450 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/155ba865-dcfd-4203-82b9-5399ca58d988-istio-envoy\") pod \"maas-default-gateway-openshift-default-84587756dc-h2ns8\" (UID: \"155ba865-dcfd-4203-82b9-5399ca58d988\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-h2ns8" Nov 03 13:17:39.158730 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:39.158713 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/155ba865-dcfd-4203-82b9-5399ca58d988-istio-podinfo\") pod \"maas-default-gateway-openshift-default-84587756dc-h2ns8\" (UID: \"155ba865-dcfd-4203-82b9-5399ca58d988\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-h2ns8" Nov 03 13:17:39.164463 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:39.164434 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/155ba865-dcfd-4203-82b9-5399ca58d988-istio-token\") pod \"maas-default-gateway-openshift-default-84587756dc-h2ns8\" (UID: \"155ba865-dcfd-4203-82b9-5399ca58d988\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-h2ns8" Nov 03 13:17:39.164561 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:39.164507 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqbdt\" (UniqueName: \"kubernetes.io/projected/155ba865-dcfd-4203-82b9-5399ca58d988-kube-api-access-sqbdt\") pod \"maas-default-gateway-openshift-default-84587756dc-h2ns8\" (UID: \"155ba865-dcfd-4203-82b9-5399ca58d988\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-h2ns8" Nov 03 13:17:39.359688 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:39.359654 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-h2ns8" Nov 03 13:17:39.506221 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:17:39.506187 5024 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d7mwz" containerName="extractor" Nov 03 13:17:39.637171 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:39.637145 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-84587756dc-h2ns8"] Nov 03 13:17:39.638939 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:17:39.638907 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod155ba865_dcfd_4203_82b9_5399ca58d988.slice/crio-d83b13849381eef46d6fec8f5fc1bdf4c3aff737f82ce61a1042a2ef26b51f98 WatchSource:0}: Error finding container d83b13849381eef46d6fec8f5fc1bdf4c3aff737f82ce61a1042a2ef26b51f98: Status 404 returned error can't find the container with id d83b13849381eef46d6fec8f5fc1bdf4c3aff737f82ce61a1042a2ef26b51f98 Nov 03 13:17:39.641919 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:39.641883 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:17:39.642014 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:39.641957 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:17:39.642014 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:39.642003 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:17:40.464399 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:40.464308 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-h2ns8" event={"ID":"155ba865-dcfd-4203-82b9-5399ca58d988","Type":"ContainerStarted","Data":"b497e622f8a4363b340c1d0fe74937acbcd77a6120d6e2e07dbfd73c7c50b244"} Nov 03 13:17:40.464399 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:40.464365 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-h2ns8" event={"ID":"155ba865-dcfd-4203-82b9-5399ca58d988","Type":"ContainerStarted","Data":"d83b13849381eef46d6fec8f5fc1bdf4c3aff737f82ce61a1042a2ef26b51f98"} Nov 03 13:17:41.360826 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:41.360778 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-h2ns8" Nov 03 13:17:41.362388 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:41.362361 5024 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-h2ns8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" start-of-body= Nov 03 13:17:41.362527 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:41.362411 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-h2ns8" podUID="155ba865-dcfd-4203-82b9-5399ca58d988" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" Nov 03 13:17:42.360459 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:42.360421 5024 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-h2ns8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" start-of-body= Nov 03 13:17:42.360967 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:42.360478 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-h2ns8" podUID="155ba865-dcfd-4203-82b9-5399ca58d988" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" Nov 03 13:17:43.198247 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:43.198186 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-h2ns8" podStartSLOduration=4.198166458 podStartE2EDuration="4.198166458s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:17:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:17:40.49155188 +0000 UTC m=+540.499743158" watchObservedRunningTime="2026-11-03 13:17:43.198166458 +0000 UTC m=+543.206357734" Nov 03 13:17:43.201094 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:43.201066 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-796d65f566-65br2"] Nov 03 13:17:43.207820 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:43.207798 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-796d65f566-65br2"] Nov 03 13:17:43.207972 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:43.207946 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-796d65f566-65br2" Nov 03 13:17:43.220014 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:43.219976 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f6vsk\" (UniqueName: \"kubernetes.io/projected/795bfdb5-d443-4daf-a0d1-e0bb7567416c-kube-api-access-f6vsk\") pod \"authorino-796d65f566-65br2\" (UID: \"795bfdb5-d443-4daf-a0d1-e0bb7567416c\") " pod="kuadrant-system/authorino-796d65f566-65br2" Nov 03 13:17:43.220136 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:43.220078 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/795bfdb5-d443-4daf-a0d1-e0bb7567416c-tls-cert\") pod \"authorino-796d65f566-65br2\" (UID: \"795bfdb5-d443-4daf-a0d1-e0bb7567416c\") " pod="kuadrant-system/authorino-796d65f566-65br2" Nov 03 13:17:43.320772 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:43.320734 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f6vsk\" (UniqueName: \"kubernetes.io/projected/795bfdb5-d443-4daf-a0d1-e0bb7567416c-kube-api-access-f6vsk\") pod \"authorino-796d65f566-65br2\" (UID: \"795bfdb5-d443-4daf-a0d1-e0bb7567416c\") " pod="kuadrant-system/authorino-796d65f566-65br2" Nov 03 13:17:43.320930 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:43.320792 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/795bfdb5-d443-4daf-a0d1-e0bb7567416c-tls-cert\") pod \"authorino-796d65f566-65br2\" (UID: \"795bfdb5-d443-4daf-a0d1-e0bb7567416c\") " pod="kuadrant-system/authorino-796d65f566-65br2" Nov 03 13:17:43.323311 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:43.323266 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/795bfdb5-d443-4daf-a0d1-e0bb7567416c-tls-cert\") pod \"authorino-796d65f566-65br2\" (UID: \"795bfdb5-d443-4daf-a0d1-e0bb7567416c\") " pod="kuadrant-system/authorino-796d65f566-65br2" Nov 03 13:17:43.335593 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:43.335564 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f6vsk\" (UniqueName: \"kubernetes.io/projected/795bfdb5-d443-4daf-a0d1-e0bb7567416c-kube-api-access-f6vsk\") pod \"authorino-796d65f566-65br2\" (UID: \"795bfdb5-d443-4daf-a0d1-e0bb7567416c\") " pod="kuadrant-system/authorino-796d65f566-65br2" Nov 03 13:17:43.360724 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:43.360692 5024 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-h2ns8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" start-of-body= Nov 03 13:17:43.361046 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:43.360750 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-h2ns8" podUID="155ba865-dcfd-4203-82b9-5399ca58d988" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" Nov 03 13:17:43.521102 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:43.521019 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-796d65f566-65br2" Nov 03 13:17:43.826197 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:43.826161 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-796d65f566-65br2"] Nov 03 13:17:43.827437 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:17:43.827407 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod795bfdb5_d443_4daf_a0d1_e0bb7567416c.slice/crio-173d323fa68ca2bf238c215f2ecf87eb98ee30195193c93c6877aa08f914da64 WatchSource:0}: Error finding container 173d323fa68ca2bf238c215f2ecf87eb98ee30195193c93c6877aa08f914da64: Status 404 returned error can't find the container with id 173d323fa68ca2bf238c215f2ecf87eb98ee30195193c93c6877aa08f914da64 Nov 03 13:17:44.360477 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:44.360438 5024 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-h2ns8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" start-of-body= Nov 03 13:17:44.360669 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:44.360497 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-h2ns8" podUID="155ba865-dcfd-4203-82b9-5399ca58d988" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" Nov 03 13:17:44.513353 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:44.513279 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-796d65f566-65br2" event={"ID":"795bfdb5-d443-4daf-a0d1-e0bb7567416c","Type":"ContainerStarted","Data":"173d323fa68ca2bf238c215f2ecf87eb98ee30195193c93c6877aa08f914da64"} Nov 03 13:17:45.360717 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:45.360682 5024 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-h2ns8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" start-of-body= Nov 03 13:17:45.360895 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:45.360777 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-h2ns8" podUID="155ba865-dcfd-4203-82b9-5399ca58d988" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" Nov 03 13:17:46.360968 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:46.360926 5024 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-h2ns8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" start-of-body= Nov 03 13:17:46.361511 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:46.360990 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-h2ns8" podUID="155ba865-dcfd-4203-82b9-5399ca58d988" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" Nov 03 13:17:46.532659 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:46.532620 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-796d65f566-65br2" event={"ID":"795bfdb5-d443-4daf-a0d1-e0bb7567416c","Type":"ContainerStarted","Data":"76686f4bb1a675f222afd26dd2868ce3caa4de61b71899b5371e5b400758fe52"} Nov 03 13:17:46.559255 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:46.559187 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-796d65f566-65br2" podStartSLOduration=1.557877957 podStartE2EDuration="3.559167956s" totalImagesPullingTime="2.001289999s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:17:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:17:46.557694468 +0000 UTC m=+546.565885746" watchObservedRunningTime="2026-11-03 13:17:46.559167956 +0000 UTC m=+546.567359235" Nov 03 13:17:46.595121 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:46.595085 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5588f885fc-qvxj7"] Nov 03 13:17:46.595926 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:46.595628 5024 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-5588f885fc-qvxj7" podUID="0f2b79da-fca5-41bf-832c-34dce95f1ca5" containerName="authorino" containerID="cri-o://868302f875c2aa5b9233e49183bfe0bc38998e0046d1191dbbfec63f7645bd15" gracePeriod=30 Nov 03 13:17:46.923142 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:46.922874 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5588f885fc-qvxj7" Nov 03 13:17:46.959855 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:46.959824 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0f2b79da-fca5-41bf-832c-34dce95f1ca5-tls-cert\" (UniqueName: \"kubernetes.io/secret/0f2b79da-fca5-41bf-832c-34dce95f1ca5-tls-cert\") pod \"0f2b79da-fca5-41bf-832c-34dce95f1ca5\" (UID: \"0f2b79da-fca5-41bf-832c-34dce95f1ca5\") " Nov 03 13:17:46.960117 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:46.960089 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f2b79da-fca5-41bf-832c-34dce95f1ca5-kube-api-access-hs7kc\" (UniqueName: \"kubernetes.io/projected/0f2b79da-fca5-41bf-832c-34dce95f1ca5-kube-api-access-hs7kc\") pod \"0f2b79da-fca5-41bf-832c-34dce95f1ca5\" (UID: \"0f2b79da-fca5-41bf-832c-34dce95f1ca5\") " Nov 03 13:17:46.965943 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:46.965909 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f2b79da-fca5-41bf-832c-34dce95f1ca5-kube-api-access-hs7kc" pod "0f2b79da-fca5-41bf-832c-34dce95f1ca5" (UID: "0f2b79da-fca5-41bf-832c-34dce95f1ca5"). InnerVolumeSpecName "kube-api-access-hs7kc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:17:47.017331 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:47.015889 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f2b79da-fca5-41bf-832c-34dce95f1ca5-tls-cert" pod "0f2b79da-fca5-41bf-832c-34dce95f1ca5" (UID: "0f2b79da-fca5-41bf-832c-34dce95f1ca5"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 13:17:47.061654 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:47.061607 5024 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/0f2b79da-fca5-41bf-832c-34dce95f1ca5-tls-cert\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:17:47.061654 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:47.061655 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hs7kc\" (UniqueName: \"kubernetes.io/projected/0f2b79da-fca5-41bf-832c-34dce95f1ca5-kube-api-access-hs7kc\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:17:47.547130 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:47.547094 5024 generic.go:356] "Generic (PLEG): container finished" podID="0f2b79da-fca5-41bf-832c-34dce95f1ca5" containerID="868302f875c2aa5b9233e49183bfe0bc38998e0046d1191dbbfec63f7645bd15" exitCode=0 Nov 03 13:17:47.547824 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:47.547182 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5588f885fc-qvxj7" Nov 03 13:17:47.547824 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:47.547200 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5588f885fc-qvxj7" event={"ID":"0f2b79da-fca5-41bf-832c-34dce95f1ca5","Type":"ContainerDied","Data":"868302f875c2aa5b9233e49183bfe0bc38998e0046d1191dbbfec63f7645bd15"} Nov 03 13:17:47.547824 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:47.547260 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5588f885fc-qvxj7" event={"ID":"0f2b79da-fca5-41bf-832c-34dce95f1ca5","Type":"ContainerDied","Data":"83ac0992502a97a875804966a31028a1726b3a251e189ec9cb0b5da7cefb0f06"} Nov 03 13:17:47.547824 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:47.547319 5024 scope.go:122] "RemoveContainer" containerID="868302f875c2aa5b9233e49183bfe0bc38998e0046d1191dbbfec63f7645bd15" Nov 03 13:17:47.560384 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:47.560239 5024 scope.go:122] "RemoveContainer" containerID="868302f875c2aa5b9233e49183bfe0bc38998e0046d1191dbbfec63f7645bd15" Nov 03 13:17:47.560594 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:17:47.560567 5024 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"868302f875c2aa5b9233e49183bfe0bc38998e0046d1191dbbfec63f7645bd15\": container with ID starting with 868302f875c2aa5b9233e49183bfe0bc38998e0046d1191dbbfec63f7645bd15 not found: ID does not exist" containerID="868302f875c2aa5b9233e49183bfe0bc38998e0046d1191dbbfec63f7645bd15" Nov 03 13:17:47.560710 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:47.560602 5024 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"868302f875c2aa5b9233e49183bfe0bc38998e0046d1191dbbfec63f7645bd15"} err="failed to get container status \"868302f875c2aa5b9233e49183bfe0bc38998e0046d1191dbbfec63f7645bd15\": rpc error: code = NotFound desc = could not find container \"868302f875c2aa5b9233e49183bfe0bc38998e0046d1191dbbfec63f7645bd15\": container with ID starting with 868302f875c2aa5b9233e49183bfe0bc38998e0046d1191dbbfec63f7645bd15 not found: ID does not exist" Nov 03 13:17:47.587425 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:47.587388 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5588f885fc-qvxj7"] Nov 03 13:17:47.597132 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:47.597094 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5588f885fc-qvxj7"] Nov 03 13:17:48.109262 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:48.109212 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f2b79da-fca5-41bf-832c-34dce95f1ca5" path="/var/lib/kubelet/pods/0f2b79da-fca5-41bf-832c-34dce95f1ca5/volumes" Nov 03 13:17:48.361712 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:48.361669 5024 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-h2ns8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 13:17:48.361917 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:48.361743 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-h2ns8" podUID="155ba865-dcfd-4203-82b9-5399ca58d988" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 13:17:49.403438 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:49.403405 5024 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-h2ns8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 13:17:49.403948 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:49.403501 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-h2ns8" podUID="155ba865-dcfd-4203-82b9-5399ca58d988" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 13:17:50.446420 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:50.446384 5024 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-h2ns8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 13:17:50.446864 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:50.446444 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-h2ns8" podUID="155ba865-dcfd-4203-82b9-5399ca58d988" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 13:17:50.744771 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:50.744679 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-h2ns8" Nov 03 13:17:50.745135 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:50.745104 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-h2ns8" Nov 03 13:17:50.745801 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:50.745770 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-h2ns8" Nov 03 13:17:54.073336 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:54.073300 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-84587756dc-vrbqc"] Nov 03 13:17:54.073899 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:54.073877 5024 cpu_manager.go:414] "containerMap: removing container" podUID="0f2b79da-fca5-41bf-832c-34dce95f1ca5" containerName="authorino" Nov 03 13:17:54.073999 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:54.073904 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f2b79da-fca5-41bf-832c-34dce95f1ca5" containerName="authorino" Nov 03 13:17:54.074061 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:54.074030 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f2b79da-fca5-41bf-832c-34dce95f1ca5" containerName="authorino" Nov 03 13:17:54.078831 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:54.078809 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-vrbqc" Nov 03 13:17:54.089334 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:54.088078 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-84587756dc-vrbqc"] Nov 03 13:17:54.145711 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:54.145677 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d12a0b88-93e3-4542-90b0-56a37dd91387-workload-socket\") pod \"maas-default-gateway-openshift-default-84587756dc-vrbqc\" (UID: \"d12a0b88-93e3-4542-90b0-56a37dd91387\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-vrbqc" Nov 03 13:17:54.145902 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:54.145732 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d12a0b88-93e3-4542-90b0-56a37dd91387-credential-socket\") pod \"maas-default-gateway-openshift-default-84587756dc-vrbqc\" (UID: \"d12a0b88-93e3-4542-90b0-56a37dd91387\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-vrbqc" Nov 03 13:17:54.145902 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:54.145819 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d12a0b88-93e3-4542-90b0-56a37dd91387-istio-envoy\") pod \"maas-default-gateway-openshift-default-84587756dc-vrbqc\" (UID: \"d12a0b88-93e3-4542-90b0-56a37dd91387\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-vrbqc" Nov 03 13:17:54.145902 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:54.145847 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d12a0b88-93e3-4542-90b0-56a37dd91387-istio-token\") pod \"maas-default-gateway-openshift-default-84587756dc-vrbqc\" (UID: \"d12a0b88-93e3-4542-90b0-56a37dd91387\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-vrbqc" Nov 03 13:17:54.145902 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:54.145864 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d12a0b88-93e3-4542-90b0-56a37dd91387-workload-certs\") pod \"maas-default-gateway-openshift-default-84587756dc-vrbqc\" (UID: \"d12a0b88-93e3-4542-90b0-56a37dd91387\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-vrbqc" Nov 03 13:17:54.146064 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:54.145994 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d12a0b88-93e3-4542-90b0-56a37dd91387-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-84587756dc-vrbqc\" (UID: \"d12a0b88-93e3-4542-90b0-56a37dd91387\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-vrbqc" Nov 03 13:17:54.146064 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:54.146050 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fjm2h\" (UniqueName: \"kubernetes.io/projected/d12a0b88-93e3-4542-90b0-56a37dd91387-kube-api-access-fjm2h\") pod \"maas-default-gateway-openshift-default-84587756dc-vrbqc\" (UID: \"d12a0b88-93e3-4542-90b0-56a37dd91387\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-vrbqc" Nov 03 13:17:54.146133 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:54.146077 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d12a0b88-93e3-4542-90b0-56a37dd91387-istio-data\") pod \"maas-default-gateway-openshift-default-84587756dc-vrbqc\" (UID: \"d12a0b88-93e3-4542-90b0-56a37dd91387\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-vrbqc" Nov 03 13:17:54.146133 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:54.146118 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d12a0b88-93e3-4542-90b0-56a37dd91387-istio-podinfo\") pod \"maas-default-gateway-openshift-default-84587756dc-vrbqc\" (UID: \"d12a0b88-93e3-4542-90b0-56a37dd91387\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-vrbqc" Nov 03 13:17:54.247331 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:54.247278 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d12a0b88-93e3-4542-90b0-56a37dd91387-credential-socket\") pod \"maas-default-gateway-openshift-default-84587756dc-vrbqc\" (UID: \"d12a0b88-93e3-4542-90b0-56a37dd91387\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-vrbqc" Nov 03 13:17:54.247492 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:54.247341 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d12a0b88-93e3-4542-90b0-56a37dd91387-istio-envoy\") pod \"maas-default-gateway-openshift-default-84587756dc-vrbqc\" (UID: \"d12a0b88-93e3-4542-90b0-56a37dd91387\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-vrbqc" Nov 03 13:17:54.247492 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:54.247359 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d12a0b88-93e3-4542-90b0-56a37dd91387-istio-token\") pod \"maas-default-gateway-openshift-default-84587756dc-vrbqc\" (UID: \"d12a0b88-93e3-4542-90b0-56a37dd91387\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-vrbqc" Nov 03 13:17:54.247492 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:54.247376 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d12a0b88-93e3-4542-90b0-56a37dd91387-workload-certs\") pod \"maas-default-gateway-openshift-default-84587756dc-vrbqc\" (UID: \"d12a0b88-93e3-4542-90b0-56a37dd91387\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-vrbqc" Nov 03 13:17:54.247492 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:54.247397 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d12a0b88-93e3-4542-90b0-56a37dd91387-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-84587756dc-vrbqc\" (UID: \"d12a0b88-93e3-4542-90b0-56a37dd91387\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-vrbqc" Nov 03 13:17:54.247492 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:54.247418 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjm2h\" (UniqueName: \"kubernetes.io/projected/d12a0b88-93e3-4542-90b0-56a37dd91387-kube-api-access-fjm2h\") pod \"maas-default-gateway-openshift-default-84587756dc-vrbqc\" (UID: \"d12a0b88-93e3-4542-90b0-56a37dd91387\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-vrbqc" Nov 03 13:17:54.247492 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:54.247454 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d12a0b88-93e3-4542-90b0-56a37dd91387-istio-data\") pod \"maas-default-gateway-openshift-default-84587756dc-vrbqc\" (UID: \"d12a0b88-93e3-4542-90b0-56a37dd91387\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-vrbqc" Nov 03 13:17:54.247873 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:54.247506 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d12a0b88-93e3-4542-90b0-56a37dd91387-istio-podinfo\") pod \"maas-default-gateway-openshift-default-84587756dc-vrbqc\" (UID: \"d12a0b88-93e3-4542-90b0-56a37dd91387\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-vrbqc" Nov 03 13:17:54.247873 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:54.247563 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d12a0b88-93e3-4542-90b0-56a37dd91387-workload-socket\") pod \"maas-default-gateway-openshift-default-84587756dc-vrbqc\" (UID: \"d12a0b88-93e3-4542-90b0-56a37dd91387\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-vrbqc" Nov 03 13:17:54.247873 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:54.247749 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d12a0b88-93e3-4542-90b0-56a37dd91387-credential-socket\") pod \"maas-default-gateway-openshift-default-84587756dc-vrbqc\" (UID: \"d12a0b88-93e3-4542-90b0-56a37dd91387\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-vrbqc" Nov 03 13:17:54.248031 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:54.247898 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d12a0b88-93e3-4542-90b0-56a37dd91387-workload-certs\") pod \"maas-default-gateway-openshift-default-84587756dc-vrbqc\" (UID: \"d12a0b88-93e3-4542-90b0-56a37dd91387\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-vrbqc" Nov 03 13:17:54.248031 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:54.247969 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d12a0b88-93e3-4542-90b0-56a37dd91387-istio-data\") pod \"maas-default-gateway-openshift-default-84587756dc-vrbqc\" (UID: \"d12a0b88-93e3-4542-90b0-56a37dd91387\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-vrbqc" Nov 03 13:17:54.248105 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:54.248052 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d12a0b88-93e3-4542-90b0-56a37dd91387-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-84587756dc-vrbqc\" (UID: \"d12a0b88-93e3-4542-90b0-56a37dd91387\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-vrbqc" Nov 03 13:17:54.248183 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:54.248164 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d12a0b88-93e3-4542-90b0-56a37dd91387-workload-socket\") pod \"maas-default-gateway-openshift-default-84587756dc-vrbqc\" (UID: \"d12a0b88-93e3-4542-90b0-56a37dd91387\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-vrbqc" Nov 03 13:17:54.249650 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:54.249624 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d12a0b88-93e3-4542-90b0-56a37dd91387-istio-envoy\") pod \"maas-default-gateway-openshift-default-84587756dc-vrbqc\" (UID: \"d12a0b88-93e3-4542-90b0-56a37dd91387\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-vrbqc" Nov 03 13:17:54.250022 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:54.250002 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d12a0b88-93e3-4542-90b0-56a37dd91387-istio-podinfo\") pod \"maas-default-gateway-openshift-default-84587756dc-vrbqc\" (UID: \"d12a0b88-93e3-4542-90b0-56a37dd91387\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-vrbqc" Nov 03 13:17:54.256192 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:54.256167 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d12a0b88-93e3-4542-90b0-56a37dd91387-istio-token\") pod \"maas-default-gateway-openshift-default-84587756dc-vrbqc\" (UID: \"d12a0b88-93e3-4542-90b0-56a37dd91387\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-vrbqc" Nov 03 13:17:54.256404 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:54.256383 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fjm2h\" (UniqueName: \"kubernetes.io/projected/d12a0b88-93e3-4542-90b0-56a37dd91387-kube-api-access-fjm2h\") pod \"maas-default-gateway-openshift-default-84587756dc-vrbqc\" (UID: \"d12a0b88-93e3-4542-90b0-56a37dd91387\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-vrbqc" Nov 03 13:17:54.400236 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:54.400204 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-vrbqc" Nov 03 13:17:54.679409 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:54.679384 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-84587756dc-vrbqc"] Nov 03 13:17:54.681167 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:17:54.681145 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd12a0b88_93e3_4542_90b0_56a37dd91387.slice/crio-1999f911d7d0c04c46b4a13efdd18a1b17459e3b09adc2e55e506c3a58534153 WatchSource:0}: Error finding container 1999f911d7d0c04c46b4a13efdd18a1b17459e3b09adc2e55e506c3a58534153: Status 404 returned error can't find the container with id 1999f911d7d0c04c46b4a13efdd18a1b17459e3b09adc2e55e506c3a58534153 Nov 03 13:17:54.683521 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:54.683490 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:17:54.683610 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:54.683553 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:17:54.683610 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:54.683595 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:17:55.603417 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:55.603376 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-vrbqc" event={"ID":"d12a0b88-93e3-4542-90b0-56a37dd91387","Type":"ContainerStarted","Data":"4def6f9ef873170255fe2ae20e0004435951de31e453bd5e4f977ebcd073ef3a"} Nov 03 13:17:55.603861 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:55.603436 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-vrbqc" event={"ID":"d12a0b88-93e3-4542-90b0-56a37dd91387","Type":"ContainerStarted","Data":"1999f911d7d0c04c46b4a13efdd18a1b17459e3b09adc2e55e506c3a58534153"} Nov 03 13:17:55.643860 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:55.643816 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-vrbqc" podStartSLOduration=1.643802579 podStartE2EDuration="1.643802579s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:17:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:17:55.641313975 +0000 UTC m=+555.649505262" watchObservedRunningTime="2026-11-03 13:17:55.643802579 +0000 UTC m=+555.651993855" Nov 03 13:17:56.401438 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:56.401384 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-vrbqc" Nov 03 13:17:56.402963 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:56.402937 5024 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-vrbqc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body= Nov 03 13:17:56.403088 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:56.402988 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-vrbqc" podUID="d12a0b88-93e3-4542-90b0-56a37dd91387" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" Nov 03 13:17:57.401223 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:57.401188 5024 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-vrbqc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body= Nov 03 13:17:57.401699 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:57.401259 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-vrbqc" podUID="d12a0b88-93e3-4542-90b0-56a37dd91387" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" Nov 03 13:17:58.400980 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:58.400945 5024 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-vrbqc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body= Nov 03 13:17:58.401159 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:58.401013 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-vrbqc" podUID="d12a0b88-93e3-4542-90b0-56a37dd91387" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" Nov 03 13:17:59.401307 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:59.401049 5024 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-vrbqc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body= Nov 03 13:17:59.401307 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:17:59.401118 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-vrbqc" podUID="d12a0b88-93e3-4542-90b0-56a37dd91387" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" Nov 03 13:18:00.401136 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:00.401101 5024 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-vrbqc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body= Nov 03 13:18:00.401347 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:00.401150 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-vrbqc" podUID="d12a0b88-93e3-4542-90b0-56a37dd91387" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" Nov 03 13:18:01.401511 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:01.401476 5024 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-vrbqc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body= Nov 03 13:18:01.403982 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:01.401534 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-vrbqc" podUID="d12a0b88-93e3-4542-90b0-56a37dd91387" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" Nov 03 13:18:02.906470 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:02.906434 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-vrbqc" Nov 03 13:18:02.906938 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:02.906902 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-vrbqc" Nov 03 13:18:02.907717 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:02.907695 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-vrbqc" Nov 03 13:18:09.506810 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:18:09.506722 5024 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d7mwz" containerName="extractor" Nov 03 13:18:16.158440 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:16.158406 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-rddcl"] Nov 03 13:18:16.162781 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:16.162757 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-rddcl" Nov 03 13:18:16.171127 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:16.170729 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-c381a7-openshift-default-dockercfg-c5nmd\"" Nov 03 13:18:16.174504 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:16.174474 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-rddcl"] Nov 03 13:18:16.260914 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:16.260881 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-lldw4"] Nov 03 13:18:16.264572 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:16.264549 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-lldw4" Nov 03 13:18:16.275533 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:16.275509 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-a4ab8e-openshift-default-dockercfg-dm758\"" Nov 03 13:18:16.278548 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:16.278523 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-lldw4"] Nov 03 13:18:16.279078 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:16.279054 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77fc5206-8873-4bcc-89b1-fb637422962b-istio-envoy\") pod \"e2e-worker-w4-c381a7-openshift-default-78bcd4695-rddcl\" (UID: \"77fc5206-8873-4bcc-89b1-fb637422962b\") " pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-rddcl" Nov 03 13:18:16.279188 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:16.279090 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77fc5206-8873-4bcc-89b1-fb637422962b-istio-token\") pod \"e2e-worker-w4-c381a7-openshift-default-78bcd4695-rddcl\" (UID: \"77fc5206-8873-4bcc-89b1-fb637422962b\") " pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-rddcl" Nov 03 13:18:16.279188 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:16.279128 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77fc5206-8873-4bcc-89b1-fb637422962b-istio-podinfo\") pod \"e2e-worker-w4-c381a7-openshift-default-78bcd4695-rddcl\" (UID: \"77fc5206-8873-4bcc-89b1-fb637422962b\") " pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-rddcl" Nov 03 13:18:16.279326 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:16.279266 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77fc5206-8873-4bcc-89b1-fb637422962b-workload-socket\") pod \"e2e-worker-w4-c381a7-openshift-default-78bcd4695-rddcl\" (UID: \"77fc5206-8873-4bcc-89b1-fb637422962b\") " pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-rddcl" Nov 03 13:18:16.279390 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:16.279365 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77fc5206-8873-4bcc-89b1-fb637422962b-credential-socket\") pod \"e2e-worker-w4-c381a7-openshift-default-78bcd4695-rddcl\" (UID: \"77fc5206-8873-4bcc-89b1-fb637422962b\") " pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-rddcl" Nov 03 13:18:16.279440 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:16.279400 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77fc5206-8873-4bcc-89b1-fb637422962b-istio-data\") pod \"e2e-worker-w4-c381a7-openshift-default-78bcd4695-rddcl\" (UID: \"77fc5206-8873-4bcc-89b1-fb637422962b\") " pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-rddcl" Nov 03 13:18:16.279581 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:16.279552 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77fc5206-8873-4bcc-89b1-fb637422962b-workload-certs\") pod \"e2e-worker-w4-c381a7-openshift-default-78bcd4695-rddcl\" (UID: \"77fc5206-8873-4bcc-89b1-fb637422962b\") " pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-rddcl" Nov 03 13:18:16.279687 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:16.279634 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77fc5206-8873-4bcc-89b1-fb637422962b-istiod-ca-cert\") pod \"e2e-worker-w4-c381a7-openshift-default-78bcd4695-rddcl\" (UID: \"77fc5206-8873-4bcc-89b1-fb637422962b\") " pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-rddcl" Nov 03 13:18:16.279768 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:16.279681 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j6pt7\" (UniqueName: \"kubernetes.io/projected/77fc5206-8873-4bcc-89b1-fb637422962b-kube-api-access-j6pt7\") pod \"e2e-worker-w4-c381a7-openshift-default-78bcd4695-rddcl\" (UID: \"77fc5206-8873-4bcc-89b1-fb637422962b\") " pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-rddcl" Nov 03 13:18:16.380736 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:16.380704 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77fc5206-8873-4bcc-89b1-fb637422962b-workload-socket\") pod \"e2e-worker-w4-c381a7-openshift-default-78bcd4695-rddcl\" (UID: \"77fc5206-8873-4bcc-89b1-fb637422962b\") " pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-rddcl" Nov 03 13:18:16.380920 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:16.380746 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/995deaed-6195-4e09-8ef4-8f1d7aa53d13-istio-envoy\") pod \"e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-lldw4\" (UID: \"995deaed-6195-4e09-8ef4-8f1d7aa53d13\") " pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-lldw4" Nov 03 13:18:16.381068 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:16.381042 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77fc5206-8873-4bcc-89b1-fb637422962b-credential-socket\") pod \"e2e-worker-w4-c381a7-openshift-default-78bcd4695-rddcl\" (UID: \"77fc5206-8873-4bcc-89b1-fb637422962b\") " pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-rddcl" Nov 03 13:18:16.381198 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:16.381156 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77fc5206-8873-4bcc-89b1-fb637422962b-workload-socket\") pod \"e2e-worker-w4-c381a7-openshift-default-78bcd4695-rddcl\" (UID: \"77fc5206-8873-4bcc-89b1-fb637422962b\") " pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-rddcl" Nov 03 13:18:16.381198 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:16.381191 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77fc5206-8873-4bcc-89b1-fb637422962b-istio-data\") pod \"e2e-worker-w4-c381a7-openshift-default-78bcd4695-rddcl\" (UID: \"77fc5206-8873-4bcc-89b1-fb637422962b\") " pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-rddcl" Nov 03 13:18:16.382124 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:16.381224 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/995deaed-6195-4e09-8ef4-8f1d7aa53d13-workload-certs\") pod \"e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-lldw4\" (UID: \"995deaed-6195-4e09-8ef4-8f1d7aa53d13\") " pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-lldw4" Nov 03 13:18:16.382124 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:16.381257 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/995deaed-6195-4e09-8ef4-8f1d7aa53d13-istio-token\") pod \"e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-lldw4\" (UID: \"995deaed-6195-4e09-8ef4-8f1d7aa53d13\") " pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-lldw4" Nov 03 13:18:16.382124 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:16.381312 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/995deaed-6195-4e09-8ef4-8f1d7aa53d13-istio-podinfo\") pod \"e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-lldw4\" (UID: \"995deaed-6195-4e09-8ef4-8f1d7aa53d13\") " pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-lldw4" Nov 03 13:18:16.382124 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:16.381385 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/995deaed-6195-4e09-8ef4-8f1d7aa53d13-workload-socket\") pod \"e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-lldw4\" (UID: \"995deaed-6195-4e09-8ef4-8f1d7aa53d13\") " pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-lldw4" Nov 03 13:18:16.382124 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:16.381409 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77fc5206-8873-4bcc-89b1-fb637422962b-workload-certs\") pod \"e2e-worker-w4-c381a7-openshift-default-78bcd4695-rddcl\" (UID: \"77fc5206-8873-4bcc-89b1-fb637422962b\") " pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-rddcl" Nov 03 13:18:16.382124 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:16.381419 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77fc5206-8873-4bcc-89b1-fb637422962b-credential-socket\") pod \"e2e-worker-w4-c381a7-openshift-default-78bcd4695-rddcl\" (UID: \"77fc5206-8873-4bcc-89b1-fb637422962b\") " pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-rddcl" Nov 03 13:18:16.382124 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:16.381433 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/995deaed-6195-4e09-8ef4-8f1d7aa53d13-istio-data\") pod \"e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-lldw4\" (UID: \"995deaed-6195-4e09-8ef4-8f1d7aa53d13\") " pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-lldw4" Nov 03 13:18:16.382124 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:16.381453 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zb6g7\" (UniqueName: \"kubernetes.io/projected/995deaed-6195-4e09-8ef4-8f1d7aa53d13-kube-api-access-zb6g7\") pod \"e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-lldw4\" (UID: \"995deaed-6195-4e09-8ef4-8f1d7aa53d13\") " pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-lldw4" Nov 03 13:18:16.382124 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:16.381488 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77fc5206-8873-4bcc-89b1-fb637422962b-istiod-ca-cert\") pod \"e2e-worker-w4-c381a7-openshift-default-78bcd4695-rddcl\" (UID: \"77fc5206-8873-4bcc-89b1-fb637422962b\") " pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-rddcl" Nov 03 13:18:16.382124 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:16.381504 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/995deaed-6195-4e09-8ef4-8f1d7aa53d13-credential-socket\") pod \"e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-lldw4\" (UID: \"995deaed-6195-4e09-8ef4-8f1d7aa53d13\") " pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-lldw4" Nov 03 13:18:16.382124 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:16.381524 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6pt7\" (UniqueName: \"kubernetes.io/projected/77fc5206-8873-4bcc-89b1-fb637422962b-kube-api-access-j6pt7\") pod \"e2e-worker-w4-c381a7-openshift-default-78bcd4695-rddcl\" (UID: \"77fc5206-8873-4bcc-89b1-fb637422962b\") " pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-rddcl" Nov 03 13:18:16.382124 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:16.381545 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77fc5206-8873-4bcc-89b1-fb637422962b-istio-envoy\") pod \"e2e-worker-w4-c381a7-openshift-default-78bcd4695-rddcl\" (UID: \"77fc5206-8873-4bcc-89b1-fb637422962b\") " pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-rddcl" Nov 03 13:18:16.382124 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:16.381561 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/995deaed-6195-4e09-8ef4-8f1d7aa53d13-istiod-ca-cert\") pod \"e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-lldw4\" (UID: \"995deaed-6195-4e09-8ef4-8f1d7aa53d13\") " pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-lldw4" Nov 03 13:18:16.382124 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:16.381578 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77fc5206-8873-4bcc-89b1-fb637422962b-istio-token\") pod \"e2e-worker-w4-c381a7-openshift-default-78bcd4695-rddcl\" (UID: \"77fc5206-8873-4bcc-89b1-fb637422962b\") " pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-rddcl" Nov 03 13:18:16.382124 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:16.381590 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77fc5206-8873-4bcc-89b1-fb637422962b-istio-data\") pod \"e2e-worker-w4-c381a7-openshift-default-78bcd4695-rddcl\" (UID: \"77fc5206-8873-4bcc-89b1-fb637422962b\") " pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-rddcl" Nov 03 13:18:16.382124 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:16.381599 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77fc5206-8873-4bcc-89b1-fb637422962b-istio-podinfo\") pod \"e2e-worker-w4-c381a7-openshift-default-78bcd4695-rddcl\" (UID: \"77fc5206-8873-4bcc-89b1-fb637422962b\") " pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-rddcl" Nov 03 13:18:16.382124 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:16.381670 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77fc5206-8873-4bcc-89b1-fb637422962b-workload-certs\") pod \"e2e-worker-w4-c381a7-openshift-default-78bcd4695-rddcl\" (UID: \"77fc5206-8873-4bcc-89b1-fb637422962b\") " pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-rddcl" Nov 03 13:18:16.382813 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:16.382388 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77fc5206-8873-4bcc-89b1-fb637422962b-istiod-ca-cert\") pod \"e2e-worker-w4-c381a7-openshift-default-78bcd4695-rddcl\" (UID: \"77fc5206-8873-4bcc-89b1-fb637422962b\") " pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-rddcl" Nov 03 13:18:16.384488 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:16.384448 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77fc5206-8873-4bcc-89b1-fb637422962b-istio-envoy\") pod \"e2e-worker-w4-c381a7-openshift-default-78bcd4695-rddcl\" (UID: \"77fc5206-8873-4bcc-89b1-fb637422962b\") " pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-rddcl" Nov 03 13:18:16.384701 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:16.384679 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77fc5206-8873-4bcc-89b1-fb637422962b-istio-podinfo\") pod \"e2e-worker-w4-c381a7-openshift-default-78bcd4695-rddcl\" (UID: \"77fc5206-8873-4bcc-89b1-fb637422962b\") " pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-rddcl" Nov 03 13:18:16.434650 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:16.433068 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6pt7\" (UniqueName: \"kubernetes.io/projected/77fc5206-8873-4bcc-89b1-fb637422962b-kube-api-access-j6pt7\") pod \"e2e-worker-w4-c381a7-openshift-default-78bcd4695-rddcl\" (UID: \"77fc5206-8873-4bcc-89b1-fb637422962b\") " pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-rddcl" Nov 03 13:18:16.440949 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:16.438919 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77fc5206-8873-4bcc-89b1-fb637422962b-istio-token\") pod \"e2e-worker-w4-c381a7-openshift-default-78bcd4695-rddcl\" (UID: \"77fc5206-8873-4bcc-89b1-fb637422962b\") " pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-rddcl" Nov 03 13:18:16.481351 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:16.481265 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-rddcl" Nov 03 13:18:16.482650 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:16.482616 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/995deaed-6195-4e09-8ef4-8f1d7aa53d13-istio-envoy\") pod \"e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-lldw4\" (UID: \"995deaed-6195-4e09-8ef4-8f1d7aa53d13\") " pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-lldw4" Nov 03 13:18:16.482775 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:16.482671 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/995deaed-6195-4e09-8ef4-8f1d7aa53d13-workload-certs\") pod \"e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-lldw4\" (UID: \"995deaed-6195-4e09-8ef4-8f1d7aa53d13\") " pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-lldw4" Nov 03 13:18:16.482775 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:16.482694 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/995deaed-6195-4e09-8ef4-8f1d7aa53d13-istio-token\") pod \"e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-lldw4\" (UID: \"995deaed-6195-4e09-8ef4-8f1d7aa53d13\") " pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-lldw4" Nov 03 13:18:16.482775 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:16.482720 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/995deaed-6195-4e09-8ef4-8f1d7aa53d13-istio-podinfo\") pod \"e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-lldw4\" (UID: \"995deaed-6195-4e09-8ef4-8f1d7aa53d13\") " pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-lldw4" Nov 03 13:18:16.482775 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:16.482738 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/995deaed-6195-4e09-8ef4-8f1d7aa53d13-workload-socket\") pod \"e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-lldw4\" (UID: \"995deaed-6195-4e09-8ef4-8f1d7aa53d13\") " pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-lldw4" Nov 03 13:18:16.482775 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:16.482763 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/995deaed-6195-4e09-8ef4-8f1d7aa53d13-istio-data\") pod \"e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-lldw4\" (UID: \"995deaed-6195-4e09-8ef4-8f1d7aa53d13\") " pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-lldw4" Nov 03 13:18:16.483085 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:16.482780 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zb6g7\" (UniqueName: \"kubernetes.io/projected/995deaed-6195-4e09-8ef4-8f1d7aa53d13-kube-api-access-zb6g7\") pod \"e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-lldw4\" (UID: \"995deaed-6195-4e09-8ef4-8f1d7aa53d13\") " pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-lldw4" Nov 03 13:18:16.483085 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:16.482816 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/995deaed-6195-4e09-8ef4-8f1d7aa53d13-credential-socket\") pod \"e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-lldw4\" (UID: \"995deaed-6195-4e09-8ef4-8f1d7aa53d13\") " pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-lldw4" Nov 03 13:18:16.483085 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:16.482842 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/995deaed-6195-4e09-8ef4-8f1d7aa53d13-istiod-ca-cert\") pod \"e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-lldw4\" (UID: \"995deaed-6195-4e09-8ef4-8f1d7aa53d13\") " pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-lldw4" Nov 03 13:18:16.483212 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:16.483158 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/995deaed-6195-4e09-8ef4-8f1d7aa53d13-workload-socket\") pod \"e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-lldw4\" (UID: \"995deaed-6195-4e09-8ef4-8f1d7aa53d13\") " pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-lldw4" Nov 03 13:18:16.483772 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:16.483492 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/995deaed-6195-4e09-8ef4-8f1d7aa53d13-credential-socket\") pod \"e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-lldw4\" (UID: \"995deaed-6195-4e09-8ef4-8f1d7aa53d13\") " pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-lldw4" Nov 03 13:18:16.483772 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:16.483584 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/995deaed-6195-4e09-8ef4-8f1d7aa53d13-istiod-ca-cert\") pod \"e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-lldw4\" (UID: \"995deaed-6195-4e09-8ef4-8f1d7aa53d13\") " pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-lldw4" Nov 03 13:18:16.483772 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:16.483631 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/995deaed-6195-4e09-8ef4-8f1d7aa53d13-istio-data\") pod \"e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-lldw4\" (UID: \"995deaed-6195-4e09-8ef4-8f1d7aa53d13\") " pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-lldw4" Nov 03 13:18:16.484261 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:16.484235 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/995deaed-6195-4e09-8ef4-8f1d7aa53d13-workload-certs\") pod \"e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-lldw4\" (UID: \"995deaed-6195-4e09-8ef4-8f1d7aa53d13\") " pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-lldw4" Nov 03 13:18:16.492701 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:16.492671 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/995deaed-6195-4e09-8ef4-8f1d7aa53d13-istio-podinfo\") pod \"e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-lldw4\" (UID: \"995deaed-6195-4e09-8ef4-8f1d7aa53d13\") " pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-lldw4" Nov 03 13:18:16.492960 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:16.492909 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/995deaed-6195-4e09-8ef4-8f1d7aa53d13-istio-envoy\") pod \"e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-lldw4\" (UID: \"995deaed-6195-4e09-8ef4-8f1d7aa53d13\") " pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-lldw4" Nov 03 13:18:16.496168 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:16.496138 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zb6g7\" (UniqueName: \"kubernetes.io/projected/995deaed-6195-4e09-8ef4-8f1d7aa53d13-kube-api-access-zb6g7\") pod \"e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-lldw4\" (UID: \"995deaed-6195-4e09-8ef4-8f1d7aa53d13\") " pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-lldw4" Nov 03 13:18:16.496985 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:16.496959 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/995deaed-6195-4e09-8ef4-8f1d7aa53d13-istio-token\") pod \"e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-lldw4\" (UID: \"995deaed-6195-4e09-8ef4-8f1d7aa53d13\") " pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-lldw4" Nov 03 13:18:16.579812 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:16.579543 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-lldw4" Nov 03 13:18:16.780306 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:16.780259 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-rddcl"] Nov 03 13:18:16.788490 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:16.788457 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:18:16.788591 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:16.788521 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:18:16.788591 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:16.788547 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:18:16.868367 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:16.868344 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-lldw4"] Nov 03 13:18:16.870647 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:18:16.870616 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod995deaed_6195_4e09_8ef4_8f1d7aa53d13.slice/crio-d17cdf854df56c71154ba658270ca1ff5c9bfd85a6c66172e5ad188d5af4c526 WatchSource:0}: Error finding container d17cdf854df56c71154ba658270ca1ff5c9bfd85a6c66172e5ad188d5af4c526: Status 404 returned error can't find the container with id d17cdf854df56c71154ba658270ca1ff5c9bfd85a6c66172e5ad188d5af4c526 Nov 03 13:18:16.878041 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:16.878004 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:18:16.878161 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:16.878093 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:18:16.878161 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:16.878143 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:18:17.743088 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:17.743044 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-rddcl" event={"ID":"77fc5206-8873-4bcc-89b1-fb637422962b","Type":"ContainerStarted","Data":"8c2a0b398d8cdb497635818df8d9a1db39abe3b060e5bbcd9d30c33667bdafc3"} Nov 03 13:18:17.743595 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:17.743164 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-rddcl" event={"ID":"77fc5206-8873-4bcc-89b1-fb637422962b","Type":"ContainerStarted","Data":"c74240ad967a5e722fe522abd24071bdbf6326c3b7fb4dc6811653679d42a35e"} Nov 03 13:18:17.745980 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:17.745946 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-lldw4" event={"ID":"995deaed-6195-4e09-8ef4-8f1d7aa53d13","Type":"ContainerStarted","Data":"e4e8ec926494865695cb452bb85ab4f496dd0f534cc7ffd08ec37bfac6acbb55"} Nov 03 13:18:17.746185 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:17.746166 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-lldw4" event={"ID":"995deaed-6195-4e09-8ef4-8f1d7aa53d13","Type":"ContainerStarted","Data":"d17cdf854df56c71154ba658270ca1ff5c9bfd85a6c66172e5ad188d5af4c526"} Nov 03 13:18:17.770225 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:17.768125 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-rddcl" podStartSLOduration=1.768106902 podStartE2EDuration="1.768106902s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:18:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:18:17.766733827 +0000 UTC m=+577.774925098" watchObservedRunningTime="2026-11-03 13:18:17.768106902 +0000 UTC m=+577.776298181" Nov 03 13:18:17.789108 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:17.789056 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-lldw4" podStartSLOduration=1.789044154 podStartE2EDuration="1.789044154s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:18:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:18:17.788206028 +0000 UTC m=+577.796397318" watchObservedRunningTime="2026-11-03 13:18:17.789044154 +0000 UTC m=+577.797235409" Nov 03 13:18:18.481764 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:18.481727 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-rddcl" Nov 03 13:18:18.486944 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:18.486918 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-rddcl" Nov 03 13:18:18.580333 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:18.580279 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-lldw4" Nov 03 13:18:18.585304 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:18.585263 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-lldw4" Nov 03 13:18:18.751900 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:18.751813 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-lldw4" Nov 03 13:18:18.751900 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:18.751867 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-rddcl" Nov 03 13:18:18.752801 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:18.752772 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-rddcl" Nov 03 13:18:18.752979 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:18.752961 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-lldw4" Nov 03 13:18:21.050740 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:21.050711 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-e440bc-8664c78f67-p6j9d"] Nov 03 13:18:21.054551 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:21.054531 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-e440bc-8664c78f67-p6j9d" Nov 03 13:18:21.070446 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:21.070420 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-e440bc-8664c78f67-p6j9d"] Nov 03 13:18:21.071200 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:21.071176 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w2-e440bc-serving-cert\"" Nov 03 13:18:21.071314 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:21.071244 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 13:18:21.071314 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:21.071179 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 03 13:18:21.071314 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:21.071253 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 13:18:21.071482 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:21.071462 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-p5qd5\"" Nov 03 13:18:21.143460 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:21.143430 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/afc7261b-bfde-425a-8c19-f2f1d521f50a-maas-api-tls\") pod \"maas-api-e2e-worker-w2-e440bc-8664c78f67-p6j9d\" (UID: \"afc7261b-bfde-425a-8c19-f2f1d521f50a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-e440bc-8664c78f67-p6j9d" Nov 03 13:18:21.143460 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:21.143462 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/afc7261b-bfde-425a-8c19-f2f1d521f50a-metrics-certs\") pod \"maas-api-e2e-worker-w2-e440bc-8664c78f67-p6j9d\" (UID: \"afc7261b-bfde-425a-8c19-f2f1d521f50a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-e440bc-8664c78f67-p6j9d" Nov 03 13:18:21.143655 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:21.143485 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/afc7261b-bfde-425a-8c19-f2f1d521f50a-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-e440bc-8664c78f67-p6j9d\" (UID: \"afc7261b-bfde-425a-8c19-f2f1d521f50a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-e440bc-8664c78f67-p6j9d" Nov 03 13:18:21.143655 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:21.143643 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7kj2q\" (UniqueName: \"kubernetes.io/projected/afc7261b-bfde-425a-8c19-f2f1d521f50a-kube-api-access-7kj2q\") pod \"maas-api-e2e-worker-w2-e440bc-8664c78f67-p6j9d\" (UID: \"afc7261b-bfde-425a-8c19-f2f1d521f50a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-e440bc-8664c78f67-p6j9d" Nov 03 13:18:21.244506 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:21.244469 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7kj2q\" (UniqueName: \"kubernetes.io/projected/afc7261b-bfde-425a-8c19-f2f1d521f50a-kube-api-access-7kj2q\") pod \"maas-api-e2e-worker-w2-e440bc-8664c78f67-p6j9d\" (UID: \"afc7261b-bfde-425a-8c19-f2f1d521f50a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-e440bc-8664c78f67-p6j9d" Nov 03 13:18:21.244685 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:21.244553 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/afc7261b-bfde-425a-8c19-f2f1d521f50a-maas-api-tls\") pod \"maas-api-e2e-worker-w2-e440bc-8664c78f67-p6j9d\" (UID: \"afc7261b-bfde-425a-8c19-f2f1d521f50a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-e440bc-8664c78f67-p6j9d" Nov 03 13:18:21.244685 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:21.244569 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/afc7261b-bfde-425a-8c19-f2f1d521f50a-metrics-certs\") pod \"maas-api-e2e-worker-w2-e440bc-8664c78f67-p6j9d\" (UID: \"afc7261b-bfde-425a-8c19-f2f1d521f50a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-e440bc-8664c78f67-p6j9d" Nov 03 13:18:21.244685 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:21.244591 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/afc7261b-bfde-425a-8c19-f2f1d521f50a-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-e440bc-8664c78f67-p6j9d\" (UID: \"afc7261b-bfde-425a-8c19-f2f1d521f50a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-e440bc-8664c78f67-p6j9d" Nov 03 13:18:21.245226 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:21.245204 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/afc7261b-bfde-425a-8c19-f2f1d521f50a-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-e440bc-8664c78f67-p6j9d\" (UID: \"afc7261b-bfde-425a-8c19-f2f1d521f50a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-e440bc-8664c78f67-p6j9d" Nov 03 13:18:21.247022 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:21.246991 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/afc7261b-bfde-425a-8c19-f2f1d521f50a-metrics-certs\") pod \"maas-api-e2e-worker-w2-e440bc-8664c78f67-p6j9d\" (UID: \"afc7261b-bfde-425a-8c19-f2f1d521f50a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-e440bc-8664c78f67-p6j9d" Nov 03 13:18:21.247135 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:21.247042 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/afc7261b-bfde-425a-8c19-f2f1d521f50a-maas-api-tls\") pod \"maas-api-e2e-worker-w2-e440bc-8664c78f67-p6j9d\" (UID: \"afc7261b-bfde-425a-8c19-f2f1d521f50a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-e440bc-8664c78f67-p6j9d" Nov 03 13:18:21.253683 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:21.253655 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7kj2q\" (UniqueName: \"kubernetes.io/projected/afc7261b-bfde-425a-8c19-f2f1d521f50a-kube-api-access-7kj2q\") pod \"maas-api-e2e-worker-w2-e440bc-8664c78f67-p6j9d\" (UID: \"afc7261b-bfde-425a-8c19-f2f1d521f50a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-e440bc-8664c78f67-p6j9d" Nov 03 13:18:21.367982 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:21.367945 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-e440bc-8664c78f67-p6j9d" Nov 03 13:18:21.665702 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:21.665673 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-e440bc-8664c78f67-p6j9d"] Nov 03 13:18:21.667413 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:18:21.667379 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podafc7261b_bfde_425a_8c19_f2f1d521f50a.slice/crio-b47315165ab5b58e356ef4c6a4abe3cbd54afd2a0aacf525fa2e2428e6de99f3 WatchSource:0}: Error finding container b47315165ab5b58e356ef4c6a4abe3cbd54afd2a0aacf525fa2e2428e6de99f3: Status 404 returned error can't find the container with id b47315165ab5b58e356ef4c6a4abe3cbd54afd2a0aacf525fa2e2428e6de99f3 Nov 03 13:18:21.767840 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:21.767798 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-e440bc-8664c78f67-p6j9d" event={"ID":"afc7261b-bfde-425a-8c19-f2f1d521f50a","Type":"ContainerStarted","Data":"b47315165ab5b58e356ef4c6a4abe3cbd54afd2a0aacf525fa2e2428e6de99f3"} Nov 03 13:18:22.013602 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:22.013530 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-c381a7-558f76676f-98tbx"] Nov 03 13:18:22.018325 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:22.018305 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-c381a7-558f76676f-98tbx" Nov 03 13:18:22.024415 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:22.024389 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-c381a7-558f76676f-98tbx"] Nov 03 13:18:22.091309 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:22.082645 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-c381a7\"" Nov 03 13:18:22.091309 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:22.082802 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-c381a7-dockercfg-hmf5s\"" Nov 03 13:18:22.167135 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:22.167088 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-94hzp\" (UniqueName: \"kubernetes.io/projected/17976293-ef64-4977-a6a2-d872ac7b9b4a-kube-api-access-94hzp\") pod \"payload-processing-e2e-worker-w4-c381a7-558f76676f-98tbx\" (UID: \"17976293-ef64-4977-a6a2-d872ac7b9b4a\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-c381a7-558f76676f-98tbx" Nov 03 13:18:22.167404 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:22.167383 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17976293-ef64-4977-a6a2-d872ac7b9b4a-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-c381a7-558f76676f-98tbx\" (UID: \"17976293-ef64-4977-a6a2-d872ac7b9b4a\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-c381a7-558f76676f-98tbx" Nov 03 13:18:22.268135 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:22.268080 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17976293-ef64-4977-a6a2-d872ac7b9b4a-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-c381a7-558f76676f-98tbx\" (UID: \"17976293-ef64-4977-a6a2-d872ac7b9b4a\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-c381a7-558f76676f-98tbx" Nov 03 13:18:22.268135 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:22.268120 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-94hzp\" (UniqueName: \"kubernetes.io/projected/17976293-ef64-4977-a6a2-d872ac7b9b4a-kube-api-access-94hzp\") pod \"payload-processing-e2e-worker-w4-c381a7-558f76676f-98tbx\" (UID: \"17976293-ef64-4977-a6a2-d872ac7b9b4a\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-c381a7-558f76676f-98tbx" Nov 03 13:18:22.268748 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:22.268725 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17976293-ef64-4977-a6a2-d872ac7b9b4a-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-c381a7-558f76676f-98tbx\" (UID: \"17976293-ef64-4977-a6a2-d872ac7b9b4a\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-c381a7-558f76676f-98tbx" Nov 03 13:18:22.277408 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:22.277379 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-94hzp\" (UniqueName: \"kubernetes.io/projected/17976293-ef64-4977-a6a2-d872ac7b9b4a-kube-api-access-94hzp\") pod \"payload-processing-e2e-worker-w4-c381a7-558f76676f-98tbx\" (UID: \"17976293-ef64-4977-a6a2-d872ac7b9b4a\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-c381a7-558f76676f-98tbx" Nov 03 13:18:22.333859 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:22.333815 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-c381a7-558f76676f-98tbx" Nov 03 13:18:22.462543 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:22.462504 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-a4ab8e-68655794c-7c8x7"] Nov 03 13:18:22.467223 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:22.467196 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-a4ab8e-68655794c-7c8x7" Nov 03 13:18:22.478508 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:22.477133 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-a4ab8e-68655794c-7c8x7"] Nov 03 13:18:22.478508 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:22.478238 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-a4ab8e\"" Nov 03 13:18:22.507134 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:22.501553 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-a4ab8e-dockercfg-mt4ql\"" Nov 03 13:18:22.572673 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:22.572586 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b571fd7-340c-455e-be5a-479ed17d7517-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-a4ab8e-68655794c-7c8x7\" (UID: \"7b571fd7-340c-455e-be5a-479ed17d7517\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-a4ab8e-68655794c-7c8x7" Nov 03 13:18:22.572673 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:22.572646 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gzr6r\" (UniqueName: \"kubernetes.io/projected/7b571fd7-340c-455e-be5a-479ed17d7517-kube-api-access-gzr6r\") pod \"payload-pre-processing-e2e-worker-w0-a4ab8e-68655794c-7c8x7\" (UID: \"7b571fd7-340c-455e-be5a-479ed17d7517\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-a4ab8e-68655794c-7c8x7" Nov 03 13:18:22.635055 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:22.635033 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-c381a7-558f76676f-98tbx"] Nov 03 13:18:22.637049 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:18:22.637013 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17976293_ef64_4977_a6a2_d872ac7b9b4a.slice/crio-b00ccff9d66e7fc2c879b952bbeda82c65251e2d1365ef63c394b00d123109e5 WatchSource:0}: Error finding container b00ccff9d66e7fc2c879b952bbeda82c65251e2d1365ef63c394b00d123109e5: Status 404 returned error can't find the container with id b00ccff9d66e7fc2c879b952bbeda82c65251e2d1365ef63c394b00d123109e5 Nov 03 13:18:22.674565 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:22.674530 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzr6r\" (UniqueName: \"kubernetes.io/projected/7b571fd7-340c-455e-be5a-479ed17d7517-kube-api-access-gzr6r\") pod \"payload-pre-processing-e2e-worker-w0-a4ab8e-68655794c-7c8x7\" (UID: \"7b571fd7-340c-455e-be5a-479ed17d7517\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-a4ab8e-68655794c-7c8x7" Nov 03 13:18:22.674709 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:22.674691 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b571fd7-340c-455e-be5a-479ed17d7517-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-a4ab8e-68655794c-7c8x7\" (UID: \"7b571fd7-340c-455e-be5a-479ed17d7517\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-a4ab8e-68655794c-7c8x7" Nov 03 13:18:22.675339 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:22.675321 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b571fd7-340c-455e-be5a-479ed17d7517-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-a4ab8e-68655794c-7c8x7\" (UID: \"7b571fd7-340c-455e-be5a-479ed17d7517\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-a4ab8e-68655794c-7c8x7" Nov 03 13:18:22.683597 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:22.683577 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gzr6r\" (UniqueName: \"kubernetes.io/projected/7b571fd7-340c-455e-be5a-479ed17d7517-kube-api-access-gzr6r\") pod \"payload-pre-processing-e2e-worker-w0-a4ab8e-68655794c-7c8x7\" (UID: \"7b571fd7-340c-455e-be5a-479ed17d7517\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-a4ab8e-68655794c-7c8x7" Nov 03 13:18:22.777369 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:22.777320 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-c381a7-558f76676f-98tbx" event={"ID":"17976293-ef64-4977-a6a2-d872ac7b9b4a","Type":"ContainerStarted","Data":"2e449a1cdee3b0a465913cc39dfa89ba62a396f845fe3224528fcc0fe9c9a9b7"} Nov 03 13:18:22.777550 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:22.777386 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-c381a7-558f76676f-98tbx" Nov 03 13:18:22.777550 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:22.777413 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-c381a7-558f76676f-98tbx" event={"ID":"17976293-ef64-4977-a6a2-d872ac7b9b4a","Type":"ContainerStarted","Data":"b00ccff9d66e7fc2c879b952bbeda82c65251e2d1365ef63c394b00d123109e5"} Nov 03 13:18:22.778993 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:22.778962 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-e440bc-8664c78f67-p6j9d" event={"ID":"afc7261b-bfde-425a-8c19-f2f1d521f50a","Type":"ContainerStarted","Data":"41fbf33cbb9d5a3006a0c550c96fc8d034fecd8f08fc889c357699ea4e7a024d"} Nov 03 13:18:22.779100 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:22.779080 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-e440bc-8664c78f67-p6j9d" Nov 03 13:18:22.785618 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:22.785598 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-a4ab8e-68655794c-7c8x7" Nov 03 13:18:22.796101 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:22.796062 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w4-c381a7-558f76676f-98tbx" podStartSLOduration=1.796049495 podStartE2EDuration="1.796049495s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:18:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:18:22.794615536 +0000 UTC m=+582.802806815" watchObservedRunningTime="2026-11-03 13:18:22.796049495 +0000 UTC m=+582.804240772" Nov 03 13:18:22.821027 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:22.820953 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-e440bc-8664c78f67-p6j9d" podStartSLOduration=1.255276424 podStartE2EDuration="1.820935498s" totalImagesPullingTime="565.659074ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:18:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:18:22.820243438 +0000 UTC m=+582.828434715" watchObservedRunningTime="2026-11-03 13:18:22.820935498 +0000 UTC m=+582.829126776" Nov 03 13:18:23.065808 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:23.065777 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-a4ab8e-68655794c-7c8x7"] Nov 03 13:18:23.067760 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:18:23.067730 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b571fd7_340c_455e_be5a_479ed17d7517.slice/crio-3552fbc31a0750fc583482247a9c390fb682d2cff898bbfed5c2985134446f38 WatchSource:0}: Error finding container 3552fbc31a0750fc583482247a9c390fb682d2cff898bbfed5c2985134446f38: Status 404 returned error can't find the container with id 3552fbc31a0750fc583482247a9c390fb682d2cff898bbfed5c2985134446f38 Nov 03 13:18:23.433180 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:23.433099 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-c381a7-7f58d8cb94-nbflc"] Nov 03 13:18:23.438043 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:23.438019 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c381a7-7f58d8cb94-nbflc" Nov 03 13:18:23.438923 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:23.438898 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-c381a7-7f58d8cb94-nbflc"] Nov 03 13:18:23.454859 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:23.446448 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w4-c381a7-serving-cert\"" Nov 03 13:18:23.585001 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:23.584970 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a5bccdb6-fde7-475e-bb9e-a256e83a83d1-maas-api-tls\") pod \"maas-api-e2e-worker-w4-c381a7-7f58d8cb94-nbflc\" (UID: \"a5bccdb6-fde7-475e-bb9e-a256e83a83d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c381a7-7f58d8cb94-nbflc" Nov 03 13:18:23.585001 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:23.585006 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9g72s\" (UniqueName: \"kubernetes.io/projected/a5bccdb6-fde7-475e-bb9e-a256e83a83d1-kube-api-access-9g72s\") pod \"maas-api-e2e-worker-w4-c381a7-7f58d8cb94-nbflc\" (UID: \"a5bccdb6-fde7-475e-bb9e-a256e83a83d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c381a7-7f58d8cb94-nbflc" Nov 03 13:18:23.585201 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:23.585122 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a5bccdb6-fde7-475e-bb9e-a256e83a83d1-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-c381a7-7f58d8cb94-nbflc\" (UID: \"a5bccdb6-fde7-475e-bb9e-a256e83a83d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c381a7-7f58d8cb94-nbflc" Nov 03 13:18:23.585201 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:23.585190 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a5bccdb6-fde7-475e-bb9e-a256e83a83d1-metrics-certs\") pod \"maas-api-e2e-worker-w4-c381a7-7f58d8cb94-nbflc\" (UID: \"a5bccdb6-fde7-475e-bb9e-a256e83a83d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c381a7-7f58d8cb94-nbflc" Nov 03 13:18:23.688357 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:23.685844 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a5bccdb6-fde7-475e-bb9e-a256e83a83d1-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-c381a7-7f58d8cb94-nbflc\" (UID: \"a5bccdb6-fde7-475e-bb9e-a256e83a83d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c381a7-7f58d8cb94-nbflc" Nov 03 13:18:23.688357 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:23.685922 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a5bccdb6-fde7-475e-bb9e-a256e83a83d1-metrics-certs\") pod \"maas-api-e2e-worker-w4-c381a7-7f58d8cb94-nbflc\" (UID: \"a5bccdb6-fde7-475e-bb9e-a256e83a83d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c381a7-7f58d8cb94-nbflc" Nov 03 13:18:23.688357 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:23.686016 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a5bccdb6-fde7-475e-bb9e-a256e83a83d1-maas-api-tls\") pod \"maas-api-e2e-worker-w4-c381a7-7f58d8cb94-nbflc\" (UID: \"a5bccdb6-fde7-475e-bb9e-a256e83a83d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c381a7-7f58d8cb94-nbflc" Nov 03 13:18:23.688357 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:23.686048 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9g72s\" (UniqueName: \"kubernetes.io/projected/a5bccdb6-fde7-475e-bb9e-a256e83a83d1-kube-api-access-9g72s\") pod \"maas-api-e2e-worker-w4-c381a7-7f58d8cb94-nbflc\" (UID: \"a5bccdb6-fde7-475e-bb9e-a256e83a83d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c381a7-7f58d8cb94-nbflc" Nov 03 13:18:23.688357 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:23.687146 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a5bccdb6-fde7-475e-bb9e-a256e83a83d1-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-c381a7-7f58d8cb94-nbflc\" (UID: \"a5bccdb6-fde7-475e-bb9e-a256e83a83d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c381a7-7f58d8cb94-nbflc" Nov 03 13:18:23.702304 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:23.696744 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a5bccdb6-fde7-475e-bb9e-a256e83a83d1-metrics-certs\") pod \"maas-api-e2e-worker-w4-c381a7-7f58d8cb94-nbflc\" (UID: \"a5bccdb6-fde7-475e-bb9e-a256e83a83d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c381a7-7f58d8cb94-nbflc" Nov 03 13:18:23.712307 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:23.707700 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a5bccdb6-fde7-475e-bb9e-a256e83a83d1-maas-api-tls\") pod \"maas-api-e2e-worker-w4-c381a7-7f58d8cb94-nbflc\" (UID: \"a5bccdb6-fde7-475e-bb9e-a256e83a83d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c381a7-7f58d8cb94-nbflc" Nov 03 13:18:23.712307 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:23.709686 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9g72s\" (UniqueName: \"kubernetes.io/projected/a5bccdb6-fde7-475e-bb9e-a256e83a83d1-kube-api-access-9g72s\") pod \"maas-api-e2e-worker-w4-c381a7-7f58d8cb94-nbflc\" (UID: \"a5bccdb6-fde7-475e-bb9e-a256e83a83d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c381a7-7f58d8cb94-nbflc" Nov 03 13:18:23.760424 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:23.760383 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c381a7-7f58d8cb94-nbflc" Nov 03 13:18:23.790570 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:23.789959 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-a4ab8e-68655794c-7c8x7" event={"ID":"7b571fd7-340c-455e-be5a-479ed17d7517","Type":"ContainerStarted","Data":"bd7484986d1d81caa8d9b7003da562b709053386e05f9d2a97fd6b84b8832e76"} Nov 03 13:18:23.790570 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:23.790028 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-a4ab8e-68655794c-7c8x7" event={"ID":"7b571fd7-340c-455e-be5a-479ed17d7517","Type":"ContainerStarted","Data":"3552fbc31a0750fc583482247a9c390fb682d2cff898bbfed5c2985134446f38"} Nov 03 13:18:23.814906 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:23.814539 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-a4ab8e-68655794c-7c8x7" podStartSLOduration=1.814517267 podStartE2EDuration="1.814517267s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:18:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:18:23.809813186 +0000 UTC m=+583.818004464" watchObservedRunningTime="2026-11-03 13:18:23.814517267 +0000 UTC m=+583.822708545" Nov 03 13:18:24.044499 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:24.044473 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-c381a7-7f58d8cb94-nbflc"] Nov 03 13:18:24.046420 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:18:24.046388 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5bccdb6_fde7_475e_bb9e_a256e83a83d1.slice/crio-86f18dce1beabd90cf4c3c85311089bbe6cbb65604c4f4bdfca4e7a8ada2449d WatchSource:0}: Error finding container 86f18dce1beabd90cf4c3c85311089bbe6cbb65604c4f4bdfca4e7a8ada2449d: Status 404 returned error can't find the container with id 86f18dce1beabd90cf4c3c85311089bbe6cbb65604c4f4bdfca4e7a8ada2449d Nov 03 13:18:24.149860 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:24.149834 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-a4ab8e-b94b5469d-kwm6h"] Nov 03 13:18:24.154937 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:24.154913 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-a4ab8e-b94b5469d-kwm6h"] Nov 03 13:18:24.155080 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:24.155000 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a4ab8e-b94b5469d-kwm6h" Nov 03 13:18:24.160943 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:24.160913 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w0-a4ab8e-serving-cert\"" Nov 03 13:18:24.303617 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:24.303533 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bb5f109b-305d-4eec-a3c9-b917a0bf9981-maas-api-tls\") pod \"maas-api-e2e-worker-w0-a4ab8e-b94b5469d-kwm6h\" (UID: \"bb5f109b-305d-4eec-a3c9-b917a0bf9981\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a4ab8e-b94b5469d-kwm6h" Nov 03 13:18:24.303617 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:24.303584 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb5f109b-305d-4eec-a3c9-b917a0bf9981-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-a4ab8e-b94b5469d-kwm6h\" (UID: \"bb5f109b-305d-4eec-a3c9-b917a0bf9981\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a4ab8e-b94b5469d-kwm6h" Nov 03 13:18:24.303825 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:24.303625 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8pvpq\" (UniqueName: \"kubernetes.io/projected/bb5f109b-305d-4eec-a3c9-b917a0bf9981-kube-api-access-8pvpq\") pod \"maas-api-e2e-worker-w0-a4ab8e-b94b5469d-kwm6h\" (UID: \"bb5f109b-305d-4eec-a3c9-b917a0bf9981\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a4ab8e-b94b5469d-kwm6h" Nov 03 13:18:24.303825 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:24.303664 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bb5f109b-305d-4eec-a3c9-b917a0bf9981-metrics-certs\") pod \"maas-api-e2e-worker-w0-a4ab8e-b94b5469d-kwm6h\" (UID: \"bb5f109b-305d-4eec-a3c9-b917a0bf9981\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a4ab8e-b94b5469d-kwm6h" Nov 03 13:18:24.404645 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:24.404611 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bb5f109b-305d-4eec-a3c9-b917a0bf9981-maas-api-tls\") pod \"maas-api-e2e-worker-w0-a4ab8e-b94b5469d-kwm6h\" (UID: \"bb5f109b-305d-4eec-a3c9-b917a0bf9981\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a4ab8e-b94b5469d-kwm6h" Nov 03 13:18:24.404811 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:24.404654 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb5f109b-305d-4eec-a3c9-b917a0bf9981-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-a4ab8e-b94b5469d-kwm6h\" (UID: \"bb5f109b-305d-4eec-a3c9-b917a0bf9981\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a4ab8e-b94b5469d-kwm6h" Nov 03 13:18:24.404811 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:24.404777 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8pvpq\" (UniqueName: \"kubernetes.io/projected/bb5f109b-305d-4eec-a3c9-b917a0bf9981-kube-api-access-8pvpq\") pod \"maas-api-e2e-worker-w0-a4ab8e-b94b5469d-kwm6h\" (UID: \"bb5f109b-305d-4eec-a3c9-b917a0bf9981\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a4ab8e-b94b5469d-kwm6h" Nov 03 13:18:24.404933 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:24.404851 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bb5f109b-305d-4eec-a3c9-b917a0bf9981-metrics-certs\") pod \"maas-api-e2e-worker-w0-a4ab8e-b94b5469d-kwm6h\" (UID: \"bb5f109b-305d-4eec-a3c9-b917a0bf9981\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a4ab8e-b94b5469d-kwm6h" Nov 03 13:18:24.405403 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:24.405376 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb5f109b-305d-4eec-a3c9-b917a0bf9981-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-a4ab8e-b94b5469d-kwm6h\" (UID: \"bb5f109b-305d-4eec-a3c9-b917a0bf9981\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a4ab8e-b94b5469d-kwm6h" Nov 03 13:18:24.407909 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:24.407881 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bb5f109b-305d-4eec-a3c9-b917a0bf9981-metrics-certs\") pod \"maas-api-e2e-worker-w0-a4ab8e-b94b5469d-kwm6h\" (UID: \"bb5f109b-305d-4eec-a3c9-b917a0bf9981\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a4ab8e-b94b5469d-kwm6h" Nov 03 13:18:24.407909 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:24.407898 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bb5f109b-305d-4eec-a3c9-b917a0bf9981-maas-api-tls\") pod \"maas-api-e2e-worker-w0-a4ab8e-b94b5469d-kwm6h\" (UID: \"bb5f109b-305d-4eec-a3c9-b917a0bf9981\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a4ab8e-b94b5469d-kwm6h" Nov 03 13:18:24.418223 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:24.418177 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8pvpq\" (UniqueName: \"kubernetes.io/projected/bb5f109b-305d-4eec-a3c9-b917a0bf9981-kube-api-access-8pvpq\") pod \"maas-api-e2e-worker-w0-a4ab8e-b94b5469d-kwm6h\" (UID: \"bb5f109b-305d-4eec-a3c9-b917a0bf9981\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a4ab8e-b94b5469d-kwm6h" Nov 03 13:18:24.472034 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:24.472007 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a4ab8e-b94b5469d-kwm6h" Nov 03 13:18:24.759621 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:24.759589 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-a4ab8e-b94b5469d-kwm6h"] Nov 03 13:18:24.761205 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:18:24.761181 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb5f109b_305d_4eec_a3c9_b917a0bf9981.slice/crio-34dc125c3728b57262418771231f4264d2b6804b8cacaebfa05a38acb2f0d953 WatchSource:0}: Error finding container 34dc125c3728b57262418771231f4264d2b6804b8cacaebfa05a38acb2f0d953: Status 404 returned error can't find the container with id 34dc125c3728b57262418771231f4264d2b6804b8cacaebfa05a38acb2f0d953 Nov 03 13:18:24.796854 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:24.796810 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a4ab8e-b94b5469d-kwm6h" event={"ID":"bb5f109b-305d-4eec-a3c9-b917a0bf9981","Type":"ContainerStarted","Data":"34dc125c3728b57262418771231f4264d2b6804b8cacaebfa05a38acb2f0d953"} Nov 03 13:18:24.798318 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:24.798270 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c381a7-7f58d8cb94-nbflc" event={"ID":"a5bccdb6-fde7-475e-bb9e-a256e83a83d1","Type":"ContainerStarted","Data":"0040d1b0f3a658cca2dac30bb14fdb049d2aa37804e308dc50d38ba9f38eba62"} Nov 03 13:18:24.798428 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:24.798342 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c381a7-7f58d8cb94-nbflc" event={"ID":"a5bccdb6-fde7-475e-bb9e-a256e83a83d1","Type":"ContainerStarted","Data":"86f18dce1beabd90cf4c3c85311089bbe6cbb65604c4f4bdfca4e7a8ada2449d"} Nov 03 13:18:24.798486 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:24.798451 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c381a7-7f58d8cb94-nbflc" Nov 03 13:18:24.798547 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:24.798503 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-a4ab8e-68655794c-7c8x7" Nov 03 13:18:25.822008 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:25.821926 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a4ab8e-b94b5469d-kwm6h" event={"ID":"bb5f109b-305d-4eec-a3c9-b917a0bf9981","Type":"ContainerStarted","Data":"90f475e3677a120a451ad1e577fe36e8f65a53b684a3d3e344a1dc3e747236cf"} Nov 03 13:18:25.822789 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:25.822745 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a4ab8e-b94b5469d-kwm6h" Nov 03 13:18:25.846393 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:25.846323 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c381a7-7f58d8cb94-nbflc" podStartSLOduration=2.3602934810000002 podStartE2EDuration="2.846308067s" totalImagesPullingTime="486.014586ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:18:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:18:24.819819315 +0000 UTC m=+584.828010592" watchObservedRunningTime="2026-11-03 13:18:25.846308067 +0000 UTC m=+585.854499345" Nov 03 13:18:25.848818 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:25.848736 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a4ab8e-b94b5469d-kwm6h" podStartSLOduration=1.458545924 podStartE2EDuration="1.848723827s" totalImagesPullingTime="390.177903ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:18:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:18:25.845077014 +0000 UTC m=+585.853268291" watchObservedRunningTime="2026-11-03 13:18:25.848723827 +0000 UTC m=+585.856915104" Nov 03 13:18:28.808597 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:28.807963 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-e440bc-8664c78f67-p6j9d" Nov 03 13:18:30.828056 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:30.828024 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c381a7-7f58d8cb94-nbflc" Nov 03 13:18:32.845224 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:32.845154 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a4ab8e-b94b5469d-kwm6h" Nov 03 13:18:33.813436 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:33.813380 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-c381a7-558f76676f-98tbx" Nov 03 13:18:34.795312 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:34.794398 5024 patch_prober.go:29] interesting pod/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-lldw4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 13:18:34.795312 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:34.794479 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-lldw4" podUID="995deaed-6195-4e09-8ef4-8f1d7aa53d13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 13:18:35.825548 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:35.825515 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-a4ab8e-68655794c-7c8x7" Nov 03 13:18:37.939464 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:37.939430 5024 generic.go:356] "Generic (PLEG): container finished" podID="995deaed-6195-4e09-8ef4-8f1d7aa53d13" containerID="e4e8ec926494865695cb452bb85ab4f496dd0f534cc7ffd08ec37bfac6acbb55" exitCode=0 Nov 03 13:18:37.941118 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:37.939513 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-lldw4" event={"ID":"995deaed-6195-4e09-8ef4-8f1d7aa53d13","Type":"ContainerDied","Data":"e4e8ec926494865695cb452bb85ab4f496dd0f534cc7ffd08ec37bfac6acbb55"} Nov 03 13:18:37.941118 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:37.940481 5024 scope.go:122] "RemoveContainer" containerID="e4e8ec926494865695cb452bb85ab4f496dd0f534cc7ffd08ec37bfac6acbb55" Nov 03 13:18:37.942986 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:37.942955 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:18:37.943083 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:37.943037 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:18:37.943148 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:37.943108 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:18:39.031306 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:39.030822 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-lldw4" event={"ID":"995deaed-6195-4e09-8ef4-8f1d7aa53d13","Type":"ContainerStarted","Data":"04d69eb53d00bd5adb24aa2a51c8747e0c8b36a2d661df1d1c5ef84aa47b1503"} Nov 03 13:18:39.581714 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:39.581678 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-lldw4" Nov 03 13:18:39.582069 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:39.582043 5024 patch_prober.go:29] interesting pod/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-lldw4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body= Nov 03 13:18:39.582224 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:39.582199 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-lldw4" podUID="995deaed-6195-4e09-8ef4-8f1d7aa53d13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" Nov 03 13:18:39.618017 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:18:39.617934 5024 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d7mwz" containerName="extractor" Nov 03 13:18:40.584377 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:40.561008 5024 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6htht_cd91de08-084e-4667-b13f-2e07b3cff624/console-operator/1.log" Nov 03 13:18:40.585172 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:40.585140 5024 patch_prober.go:29] interesting pod/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-lldw4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body= Nov 03 13:18:40.585692 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:40.585511 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-lldw4" podUID="995deaed-6195-4e09-8ef4-8f1d7aa53d13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" Nov 03 13:18:40.629211 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:40.629168 5024 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6htht_cd91de08-084e-4667-b13f-2e07b3cff624/console-operator/1.log" Nov 03 13:18:41.581118 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:41.580651 5024 patch_prober.go:29] interesting pod/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-lldw4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body= Nov 03 13:18:41.581118 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:41.580713 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-lldw4" podUID="995deaed-6195-4e09-8ef4-8f1d7aa53d13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" Nov 03 13:18:42.586305 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:42.585867 5024 patch_prober.go:29] interesting pod/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-lldw4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body= Nov 03 13:18:42.586305 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:42.585992 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-lldw4" podUID="995deaed-6195-4e09-8ef4-8f1d7aa53d13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" Nov 03 13:18:43.591309 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:43.590742 5024 patch_prober.go:29] interesting pod/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-lldw4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body= Nov 03 13:18:43.591309 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:43.590810 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-lldw4" podUID="995deaed-6195-4e09-8ef4-8f1d7aa53d13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" Nov 03 13:18:44.597309 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:44.591125 5024 patch_prober.go:29] interesting pod/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-lldw4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body= Nov 03 13:18:44.597309 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:44.591192 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-lldw4" podUID="995deaed-6195-4e09-8ef4-8f1d7aa53d13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" Nov 03 13:18:45.586105 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:45.586010 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-lldw4" Nov 03 13:18:45.586548 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:45.586519 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-lldw4" Nov 03 13:18:45.587475 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:45.587454 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-lldw4" Nov 03 13:18:46.104360 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:46.104328 5024 generic.go:356] "Generic (PLEG): container finished" podID="77fc5206-8873-4bcc-89b1-fb637422962b" containerID="8c2a0b398d8cdb497635818df8d9a1db39abe3b060e5bbcd9d30c33667bdafc3" exitCode=0 Nov 03 13:18:46.107835 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:46.107804 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-rddcl" event={"ID":"77fc5206-8873-4bcc-89b1-fb637422962b","Type":"ContainerDied","Data":"8c2a0b398d8cdb497635818df8d9a1db39abe3b060e5bbcd9d30c33667bdafc3"} Nov 03 13:18:46.108139 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:46.108124 5024 scope.go:122] "RemoveContainer" containerID="8c2a0b398d8cdb497635818df8d9a1db39abe3b060e5bbcd9d30c33667bdafc3" Nov 03 13:18:46.109395 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:46.109367 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:18:46.109526 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:46.109448 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:18:46.109526 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:46.109496 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:18:47.134333 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:47.131478 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-rddcl" event={"ID":"77fc5206-8873-4bcc-89b1-fb637422962b","Type":"ContainerStarted","Data":"aebdfbf8569969e34481bf8e94bc420143339f71a7e38b7baa320243eb1d2097"} Nov 03 13:18:47.486752 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:47.486664 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-rddcl" Nov 03 13:18:47.487246 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:47.487221 5024 patch_prober.go:29] interesting pod/e2e-worker-w4-c381a7-openshift-default-78bcd4695-rddcl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body= Nov 03 13:18:47.487462 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:47.487436 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-rddcl" podUID="77fc5206-8873-4bcc-89b1-fb637422962b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" Nov 03 13:18:48.486310 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:48.485808 5024 patch_prober.go:29] interesting pod/e2e-worker-w4-c381a7-openshift-default-78bcd4695-rddcl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body= Nov 03 13:18:48.486310 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:48.485883 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-rddcl" podUID="77fc5206-8873-4bcc-89b1-fb637422962b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" Nov 03 13:18:48.751643 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:48.751548 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-rddcl" Nov 03 13:18:49.482257 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:49.482219 5024 patch_prober.go:29] interesting pod/e2e-worker-w4-c381a7-openshift-default-78bcd4695-rddcl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body= Nov 03 13:18:49.482567 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:49.482538 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-rddcl" podUID="77fc5206-8873-4bcc-89b1-fb637422962b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" Nov 03 13:18:50.495309 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:50.494690 5024 patch_prober.go:29] interesting pod/e2e-worker-w4-c381a7-openshift-default-78bcd4695-rddcl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body= Nov 03 13:18:50.495309 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:50.494768 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-rddcl" podUID="77fc5206-8873-4bcc-89b1-fb637422962b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" Nov 03 13:18:51.484718 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:51.484677 5024 patch_prober.go:29] interesting pod/e2e-worker-w4-c381a7-openshift-default-78bcd4695-rddcl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body= Nov 03 13:18:51.485050 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:51.485022 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-rddcl" podUID="77fc5206-8873-4bcc-89b1-fb637422962b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" Nov 03 13:18:52.516401 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:52.516358 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-rddcl" Nov 03 13:18:52.518240 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:52.518216 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-rddcl" Nov 03 13:18:59.664006 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:59.663965 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-17adbf-openshift-default-75f4498d49-hsbdd"] Nov 03 13:18:59.669582 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:59.669553 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-17adbf-openshift-default-75f4498d49-hsbdd" Nov 03 13:18:59.682609 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:59.682577 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-17adbf-openshift-default-dockercfg-g5rsw\"" Nov 03 13:18:59.685521 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:59.685493 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-17adbf-openshift-default-75f4498d49-hsbdd"] Nov 03 13:18:59.802975 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:59.802922 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-flws7\" (UniqueName: \"kubernetes.io/projected/42d727c9-2a2b-4a3d-afb7-c0d2ee617b7f-kube-api-access-flws7\") pod \"e2e-inf-a-17adbf-openshift-default-75f4498d49-hsbdd\" (UID: \"42d727c9-2a2b-4a3d-afb7-c0d2ee617b7f\") " pod="openshift-ingress/e2e-inf-a-17adbf-openshift-default-75f4498d49-hsbdd" Nov 03 13:18:59.803373 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:59.803341 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42d727c9-2a2b-4a3d-afb7-c0d2ee617b7f-workload-socket\") pod \"e2e-inf-a-17adbf-openshift-default-75f4498d49-hsbdd\" (UID: \"42d727c9-2a2b-4a3d-afb7-c0d2ee617b7f\") " pod="openshift-ingress/e2e-inf-a-17adbf-openshift-default-75f4498d49-hsbdd" Nov 03 13:18:59.803614 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:59.803597 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42d727c9-2a2b-4a3d-afb7-c0d2ee617b7f-istio-token\") pod \"e2e-inf-a-17adbf-openshift-default-75f4498d49-hsbdd\" (UID: \"42d727c9-2a2b-4a3d-afb7-c0d2ee617b7f\") " pod="openshift-ingress/e2e-inf-a-17adbf-openshift-default-75f4498d49-hsbdd" Nov 03 13:18:59.803825 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:59.803774 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42d727c9-2a2b-4a3d-afb7-c0d2ee617b7f-istio-envoy\") pod \"e2e-inf-a-17adbf-openshift-default-75f4498d49-hsbdd\" (UID: \"42d727c9-2a2b-4a3d-afb7-c0d2ee617b7f\") " pod="openshift-ingress/e2e-inf-a-17adbf-openshift-default-75f4498d49-hsbdd" Nov 03 13:18:59.803975 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:59.803960 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42d727c9-2a2b-4a3d-afb7-c0d2ee617b7f-istio-podinfo\") pod \"e2e-inf-a-17adbf-openshift-default-75f4498d49-hsbdd\" (UID: \"42d727c9-2a2b-4a3d-afb7-c0d2ee617b7f\") " pod="openshift-ingress/e2e-inf-a-17adbf-openshift-default-75f4498d49-hsbdd" Nov 03 13:18:59.804166 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:59.804116 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42d727c9-2a2b-4a3d-afb7-c0d2ee617b7f-workload-certs\") pod \"e2e-inf-a-17adbf-openshift-default-75f4498d49-hsbdd\" (UID: \"42d727c9-2a2b-4a3d-afb7-c0d2ee617b7f\") " pod="openshift-ingress/e2e-inf-a-17adbf-openshift-default-75f4498d49-hsbdd" Nov 03 13:18:59.804337 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:59.804276 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42d727c9-2a2b-4a3d-afb7-c0d2ee617b7f-istiod-ca-cert\") pod \"e2e-inf-a-17adbf-openshift-default-75f4498d49-hsbdd\" (UID: \"42d727c9-2a2b-4a3d-afb7-c0d2ee617b7f\") " pod="openshift-ingress/e2e-inf-a-17adbf-openshift-default-75f4498d49-hsbdd" Nov 03 13:18:59.804532 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:59.804517 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42d727c9-2a2b-4a3d-afb7-c0d2ee617b7f-credential-socket\") pod \"e2e-inf-a-17adbf-openshift-default-75f4498d49-hsbdd\" (UID: \"42d727c9-2a2b-4a3d-afb7-c0d2ee617b7f\") " pod="openshift-ingress/e2e-inf-a-17adbf-openshift-default-75f4498d49-hsbdd" Nov 03 13:18:59.804718 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:59.804704 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42d727c9-2a2b-4a3d-afb7-c0d2ee617b7f-istio-data\") pod \"e2e-inf-a-17adbf-openshift-default-75f4498d49-hsbdd\" (UID: \"42d727c9-2a2b-4a3d-afb7-c0d2ee617b7f\") " pod="openshift-ingress/e2e-inf-a-17adbf-openshift-default-75f4498d49-hsbdd" Nov 03 13:18:59.905479 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:59.905440 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42d727c9-2a2b-4a3d-afb7-c0d2ee617b7f-istio-token\") pod \"e2e-inf-a-17adbf-openshift-default-75f4498d49-hsbdd\" (UID: \"42d727c9-2a2b-4a3d-afb7-c0d2ee617b7f\") " pod="openshift-ingress/e2e-inf-a-17adbf-openshift-default-75f4498d49-hsbdd" Nov 03 13:18:59.905823 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:59.905798 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42d727c9-2a2b-4a3d-afb7-c0d2ee617b7f-istio-envoy\") pod \"e2e-inf-a-17adbf-openshift-default-75f4498d49-hsbdd\" (UID: \"42d727c9-2a2b-4a3d-afb7-c0d2ee617b7f\") " pod="openshift-ingress/e2e-inf-a-17adbf-openshift-default-75f4498d49-hsbdd" Nov 03 13:18:59.905935 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:59.905873 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42d727c9-2a2b-4a3d-afb7-c0d2ee617b7f-istio-podinfo\") pod \"e2e-inf-a-17adbf-openshift-default-75f4498d49-hsbdd\" (UID: \"42d727c9-2a2b-4a3d-afb7-c0d2ee617b7f\") " pod="openshift-ingress/e2e-inf-a-17adbf-openshift-default-75f4498d49-hsbdd" Nov 03 13:18:59.905935 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:59.905907 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42d727c9-2a2b-4a3d-afb7-c0d2ee617b7f-workload-certs\") pod \"e2e-inf-a-17adbf-openshift-default-75f4498d49-hsbdd\" (UID: \"42d727c9-2a2b-4a3d-afb7-c0d2ee617b7f\") " pod="openshift-ingress/e2e-inf-a-17adbf-openshift-default-75f4498d49-hsbdd" Nov 03 13:18:59.906048 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:59.905951 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42d727c9-2a2b-4a3d-afb7-c0d2ee617b7f-istiod-ca-cert\") pod \"e2e-inf-a-17adbf-openshift-default-75f4498d49-hsbdd\" (UID: \"42d727c9-2a2b-4a3d-afb7-c0d2ee617b7f\") " pod="openshift-ingress/e2e-inf-a-17adbf-openshift-default-75f4498d49-hsbdd" Nov 03 13:18:59.906048 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:59.906004 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42d727c9-2a2b-4a3d-afb7-c0d2ee617b7f-credential-socket\") pod \"e2e-inf-a-17adbf-openshift-default-75f4498d49-hsbdd\" (UID: \"42d727c9-2a2b-4a3d-afb7-c0d2ee617b7f\") " pod="openshift-ingress/e2e-inf-a-17adbf-openshift-default-75f4498d49-hsbdd" Nov 03 13:18:59.906143 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:59.906047 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42d727c9-2a2b-4a3d-afb7-c0d2ee617b7f-istio-data\") pod \"e2e-inf-a-17adbf-openshift-default-75f4498d49-hsbdd\" (UID: \"42d727c9-2a2b-4a3d-afb7-c0d2ee617b7f\") " pod="openshift-ingress/e2e-inf-a-17adbf-openshift-default-75f4498d49-hsbdd" Nov 03 13:18:59.906198 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:59.906181 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-flws7\" (UniqueName: \"kubernetes.io/projected/42d727c9-2a2b-4a3d-afb7-c0d2ee617b7f-kube-api-access-flws7\") pod \"e2e-inf-a-17adbf-openshift-default-75f4498d49-hsbdd\" (UID: \"42d727c9-2a2b-4a3d-afb7-c0d2ee617b7f\") " pod="openshift-ingress/e2e-inf-a-17adbf-openshift-default-75f4498d49-hsbdd" Nov 03 13:18:59.906257 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:59.906248 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42d727c9-2a2b-4a3d-afb7-c0d2ee617b7f-workload-socket\") pod \"e2e-inf-a-17adbf-openshift-default-75f4498d49-hsbdd\" (UID: \"42d727c9-2a2b-4a3d-afb7-c0d2ee617b7f\") " pod="openshift-ingress/e2e-inf-a-17adbf-openshift-default-75f4498d49-hsbdd" Nov 03 13:18:59.907156 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:59.906791 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42d727c9-2a2b-4a3d-afb7-c0d2ee617b7f-istio-data\") pod \"e2e-inf-a-17adbf-openshift-default-75f4498d49-hsbdd\" (UID: \"42d727c9-2a2b-4a3d-afb7-c0d2ee617b7f\") " pod="openshift-ingress/e2e-inf-a-17adbf-openshift-default-75f4498d49-hsbdd" Nov 03 13:18:59.907156 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:59.907053 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42d727c9-2a2b-4a3d-afb7-c0d2ee617b7f-credential-socket\") pod \"e2e-inf-a-17adbf-openshift-default-75f4498d49-hsbdd\" (UID: \"42d727c9-2a2b-4a3d-afb7-c0d2ee617b7f\") " pod="openshift-ingress/e2e-inf-a-17adbf-openshift-default-75f4498d49-hsbdd" Nov 03 13:18:59.907156 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:59.907113 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42d727c9-2a2b-4a3d-afb7-c0d2ee617b7f-workload-certs\") pod \"e2e-inf-a-17adbf-openshift-default-75f4498d49-hsbdd\" (UID: \"42d727c9-2a2b-4a3d-afb7-c0d2ee617b7f\") " pod="openshift-ingress/e2e-inf-a-17adbf-openshift-default-75f4498d49-hsbdd" Nov 03 13:18:59.907418 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:59.907392 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42d727c9-2a2b-4a3d-afb7-c0d2ee617b7f-istiod-ca-cert\") pod \"e2e-inf-a-17adbf-openshift-default-75f4498d49-hsbdd\" (UID: \"42d727c9-2a2b-4a3d-afb7-c0d2ee617b7f\") " pod="openshift-ingress/e2e-inf-a-17adbf-openshift-default-75f4498d49-hsbdd" Nov 03 13:18:59.907556 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:59.907449 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42d727c9-2a2b-4a3d-afb7-c0d2ee617b7f-workload-socket\") pod \"e2e-inf-a-17adbf-openshift-default-75f4498d49-hsbdd\" (UID: \"42d727c9-2a2b-4a3d-afb7-c0d2ee617b7f\") " pod="openshift-ingress/e2e-inf-a-17adbf-openshift-default-75f4498d49-hsbdd" Nov 03 13:18:59.910193 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:59.910171 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42d727c9-2a2b-4a3d-afb7-c0d2ee617b7f-istio-envoy\") pod \"e2e-inf-a-17adbf-openshift-default-75f4498d49-hsbdd\" (UID: \"42d727c9-2a2b-4a3d-afb7-c0d2ee617b7f\") " pod="openshift-ingress/e2e-inf-a-17adbf-openshift-default-75f4498d49-hsbdd" Nov 03 13:18:59.910543 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:59.910518 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42d727c9-2a2b-4a3d-afb7-c0d2ee617b7f-istio-podinfo\") pod \"e2e-inf-a-17adbf-openshift-default-75f4498d49-hsbdd\" (UID: \"42d727c9-2a2b-4a3d-afb7-c0d2ee617b7f\") " pod="openshift-ingress/e2e-inf-a-17adbf-openshift-default-75f4498d49-hsbdd" Nov 03 13:18:59.917138 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:59.917081 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42d727c9-2a2b-4a3d-afb7-c0d2ee617b7f-istio-token\") pod \"e2e-inf-a-17adbf-openshift-default-75f4498d49-hsbdd\" (UID: \"42d727c9-2a2b-4a3d-afb7-c0d2ee617b7f\") " pod="openshift-ingress/e2e-inf-a-17adbf-openshift-default-75f4498d49-hsbdd" Nov 03 13:18:59.918590 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:59.918567 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-flws7\" (UniqueName: \"kubernetes.io/projected/42d727c9-2a2b-4a3d-afb7-c0d2ee617b7f-kube-api-access-flws7\") pod \"e2e-inf-a-17adbf-openshift-default-75f4498d49-hsbdd\" (UID: \"42d727c9-2a2b-4a3d-afb7-c0d2ee617b7f\") " pod="openshift-ingress/e2e-inf-a-17adbf-openshift-default-75f4498d49-hsbdd" Nov 03 13:18:59.991952 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:18:59.991917 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-17adbf-openshift-default-75f4498d49-hsbdd" Nov 03 13:19:00.279086 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:19:00.279047 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-17adbf-openshift-default-75f4498d49-hsbdd"] Nov 03 13:19:00.280339 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:19:00.280304 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42d727c9_2a2b_4a3d_afb7_c0d2ee617b7f.slice/crio-37e1c14208b20fe33c927d5d957d6d241387432b19232e1ec0ece853aae83afc WatchSource:0}: Error finding container 37e1c14208b20fe33c927d5d957d6d241387432b19232e1ec0ece853aae83afc: Status 404 returned error can't find the container with id 37e1c14208b20fe33c927d5d957d6d241387432b19232e1ec0ece853aae83afc Nov 03 13:19:00.283484 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:19:00.283440 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:19:00.283597 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:19:00.283542 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:19:00.283597 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:19:00.283583 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:19:01.229931 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:19:01.229889 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-17adbf-openshift-default-75f4498d49-hsbdd" event={"ID":"42d727c9-2a2b-4a3d-afb7-c0d2ee617b7f","Type":"ContainerStarted","Data":"60b1fa6c8cab7eb811c819276689b8f963f74a5ff0074beb66d743617ee6f9ba"} Nov 03 13:19:01.230575 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:19:01.230547 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-17adbf-openshift-default-75f4498d49-hsbdd" event={"ID":"42d727c9-2a2b-4a3d-afb7-c0d2ee617b7f","Type":"ContainerStarted","Data":"37e1c14208b20fe33c927d5d957d6d241387432b19232e1ec0ece853aae83afc"} Nov 03 13:19:01.263308 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:19:01.262883 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-17adbf-openshift-default-75f4498d49-hsbdd" podStartSLOduration=2.262863225 podStartE2EDuration="2.262863225s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:18:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:19:01.259028742 +0000 UTC m=+621.267220053" watchObservedRunningTime="2026-11-03 13:19:01.262863225 +0000 UTC m=+621.271054504" Nov 03 13:19:01.992672 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:19:01.992400 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-17adbf-openshift-default-75f4498d49-hsbdd" Nov 03 13:19:01.998334 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:19:01.998301 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-17adbf-openshift-default-75f4498d49-hsbdd" Nov 03 13:19:02.236641 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:19:02.236608 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-17adbf-openshift-default-75f4498d49-hsbdd" Nov 03 13:19:02.237740 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:19:02.237715 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-17adbf-openshift-default-75f4498d49-hsbdd" Nov 03 13:19:04.938450 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:19:04.938417 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-17adbf-696b5d5d7f-9fqx5"] Nov 03 13:19:04.942615 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:19:04.942595 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-17adbf-696b5d5d7f-9fqx5" Nov 03 13:19:04.960014 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:19:04.959975 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-17adbf\"" Nov 03 13:19:04.962217 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:19:04.962189 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-17adbf-696b5d5d7f-9fqx5"] Nov 03 13:19:04.997184 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:19:04.997144 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ptwwj\" (UniqueName: \"kubernetes.io/projected/a046d933-94e8-4244-96c5-0ac0665bf579-kube-api-access-ptwwj\") pod \"payload-pre-processing-e2e-inf-a-17adbf-696b5d5d7f-9fqx5\" (UID: \"a046d933-94e8-4244-96c5-0ac0665bf579\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-17adbf-696b5d5d7f-9fqx5" Nov 03 13:19:04.997477 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:19:04.997448 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a046d933-94e8-4244-96c5-0ac0665bf579-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-17adbf-696b5d5d7f-9fqx5\" (UID: \"a046d933-94e8-4244-96c5-0ac0665bf579\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-17adbf-696b5d5d7f-9fqx5" Nov 03 13:19:05.098864 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:19:05.098821 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ptwwj\" (UniqueName: \"kubernetes.io/projected/a046d933-94e8-4244-96c5-0ac0665bf579-kube-api-access-ptwwj\") pod \"payload-pre-processing-e2e-inf-a-17adbf-696b5d5d7f-9fqx5\" (UID: \"a046d933-94e8-4244-96c5-0ac0665bf579\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-17adbf-696b5d5d7f-9fqx5" Nov 03 13:19:05.099072 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:19:05.098897 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a046d933-94e8-4244-96c5-0ac0665bf579-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-17adbf-696b5d5d7f-9fqx5\" (UID: \"a046d933-94e8-4244-96c5-0ac0665bf579\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-17adbf-696b5d5d7f-9fqx5" Nov 03 13:19:05.099693 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:19:05.099666 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a046d933-94e8-4244-96c5-0ac0665bf579-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-17adbf-696b5d5d7f-9fqx5\" (UID: \"a046d933-94e8-4244-96c5-0ac0665bf579\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-17adbf-696b5d5d7f-9fqx5" Nov 03 13:19:05.108545 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:19:05.108516 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ptwwj\" (UniqueName: \"kubernetes.io/projected/a046d933-94e8-4244-96c5-0ac0665bf579-kube-api-access-ptwwj\") pod \"payload-pre-processing-e2e-inf-a-17adbf-696b5d5d7f-9fqx5\" (UID: \"a046d933-94e8-4244-96c5-0ac0665bf579\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-17adbf-696b5d5d7f-9fqx5" Nov 03 13:19:05.257534 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:19:05.257438 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-17adbf-696b5d5d7f-9fqx5" Nov 03 13:19:05.534114 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:19:05.534019 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-17adbf-696b5d5d7f-9fqx5"] Nov 03 13:19:05.537957 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:19:05.537930 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda046d933_94e8_4244_96c5_0ac0665bf579.slice/crio-c8b83058508331593dd3ebcd39117b236209a197c3e0b8fc00390265de6c5cb3 WatchSource:0}: Error finding container c8b83058508331593dd3ebcd39117b236209a197c3e0b8fc00390265de6c5cb3: Status 404 returned error can't find the container with id c8b83058508331593dd3ebcd39117b236209a197c3e0b8fc00390265de6c5cb3 Nov 03 13:19:05.717681 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:19:05.717638 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-17adbf-5965f57bff-h2rr9"] Nov 03 13:19:05.722261 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:19:05.722236 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-17adbf-5965f57bff-h2rr9" Nov 03 13:19:05.730558 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:19:05.730074 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-17adbf-5965f57bff-h2rr9"] Nov 03 13:19:05.803727 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:19:05.803635 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-17adbf-serving-cert\"" Nov 03 13:19:05.810218 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:19:05.810180 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a00c1abc-3c3b-4829-94ce-f0c1bcc6d862-openshift-service-ca\") pod \"maas-api-e2e-inf-a-17adbf-5965f57bff-h2rr9\" (UID: \"a00c1abc-3c3b-4829-94ce-f0c1bcc6d862\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-17adbf-5965f57bff-h2rr9" Nov 03 13:19:05.810397 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:19:05.810251 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a00c1abc-3c3b-4829-94ce-f0c1bcc6d862-maas-api-tls\") pod \"maas-api-e2e-inf-a-17adbf-5965f57bff-h2rr9\" (UID: \"a00c1abc-3c3b-4829-94ce-f0c1bcc6d862\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-17adbf-5965f57bff-h2rr9" Nov 03 13:19:05.810473 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:19:05.810404 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a00c1abc-3c3b-4829-94ce-f0c1bcc6d862-metrics-certs\") pod \"maas-api-e2e-inf-a-17adbf-5965f57bff-h2rr9\" (UID: \"a00c1abc-3c3b-4829-94ce-f0c1bcc6d862\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-17adbf-5965f57bff-h2rr9" Nov 03 13:19:05.810473 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:19:05.810441 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d5ltd\" (UniqueName: \"kubernetes.io/projected/a00c1abc-3c3b-4829-94ce-f0c1bcc6d862-kube-api-access-d5ltd\") pod \"maas-api-e2e-inf-a-17adbf-5965f57bff-h2rr9\" (UID: \"a00c1abc-3c3b-4829-94ce-f0c1bcc6d862\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-17adbf-5965f57bff-h2rr9" Nov 03 13:19:05.925795 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:19:05.925753 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a00c1abc-3c3b-4829-94ce-f0c1bcc6d862-metrics-certs\") pod \"maas-api-e2e-inf-a-17adbf-5965f57bff-h2rr9\" (UID: \"a00c1abc-3c3b-4829-94ce-f0c1bcc6d862\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-17adbf-5965f57bff-h2rr9" Nov 03 13:19:05.925993 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:19:05.925814 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5ltd\" (UniqueName: \"kubernetes.io/projected/a00c1abc-3c3b-4829-94ce-f0c1bcc6d862-kube-api-access-d5ltd\") pod \"maas-api-e2e-inf-a-17adbf-5965f57bff-h2rr9\" (UID: \"a00c1abc-3c3b-4829-94ce-f0c1bcc6d862\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-17adbf-5965f57bff-h2rr9" Nov 03 13:19:05.925993 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:19:05.925972 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a00c1abc-3c3b-4829-94ce-f0c1bcc6d862-openshift-service-ca\") pod \"maas-api-e2e-inf-a-17adbf-5965f57bff-h2rr9\" (UID: \"a00c1abc-3c3b-4829-94ce-f0c1bcc6d862\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-17adbf-5965f57bff-h2rr9" Nov 03 13:19:05.926119 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:19:05.926014 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a00c1abc-3c3b-4829-94ce-f0c1bcc6d862-maas-api-tls\") pod \"maas-api-e2e-inf-a-17adbf-5965f57bff-h2rr9\" (UID: \"a00c1abc-3c3b-4829-94ce-f0c1bcc6d862\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-17adbf-5965f57bff-h2rr9" Nov 03 13:19:05.931342 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:19:05.927452 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a00c1abc-3c3b-4829-94ce-f0c1bcc6d862-openshift-service-ca\") pod \"maas-api-e2e-inf-a-17adbf-5965f57bff-h2rr9\" (UID: \"a00c1abc-3c3b-4829-94ce-f0c1bcc6d862\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-17adbf-5965f57bff-h2rr9" Nov 03 13:19:05.942767 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:19:05.942723 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a00c1abc-3c3b-4829-94ce-f0c1bcc6d862-maas-api-tls\") pod \"maas-api-e2e-inf-a-17adbf-5965f57bff-h2rr9\" (UID: \"a00c1abc-3c3b-4829-94ce-f0c1bcc6d862\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-17adbf-5965f57bff-h2rr9" Nov 03 13:19:05.944976 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:19:05.944943 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a00c1abc-3c3b-4829-94ce-f0c1bcc6d862-metrics-certs\") pod \"maas-api-e2e-inf-a-17adbf-5965f57bff-h2rr9\" (UID: \"a00c1abc-3c3b-4829-94ce-f0c1bcc6d862\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-17adbf-5965f57bff-h2rr9" Nov 03 13:19:05.945797 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:19:05.945773 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d5ltd\" (UniqueName: \"kubernetes.io/projected/a00c1abc-3c3b-4829-94ce-f0c1bcc6d862-kube-api-access-d5ltd\") pod \"maas-api-e2e-inf-a-17adbf-5965f57bff-h2rr9\" (UID: \"a00c1abc-3c3b-4829-94ce-f0c1bcc6d862\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-17adbf-5965f57bff-h2rr9" Nov 03 13:19:06.043354 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:19:06.043315 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-17adbf-5965f57bff-h2rr9" Nov 03 13:19:06.259892 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:19:06.259852 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-17adbf-696b5d5d7f-9fqx5" event={"ID":"a046d933-94e8-4244-96c5-0ac0665bf579","Type":"ContainerStarted","Data":"f552a78cc62f8af156914e5e918b93737ffc4ddc4280d3982fd1e9e123f83560"} Nov 03 13:19:06.260098 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:19:06.259912 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-17adbf-696b5d5d7f-9fqx5" event={"ID":"a046d933-94e8-4244-96c5-0ac0665bf579","Type":"ContainerStarted","Data":"c8b83058508331593dd3ebcd39117b236209a197c3e0b8fc00390265de6c5cb3"} Nov 03 13:19:06.260098 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:19:06.259962 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-17adbf-696b5d5d7f-9fqx5" Nov 03 13:19:06.279072 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:19:06.279015 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-17adbf-696b5d5d7f-9fqx5" podStartSLOduration=2.27899981 podStartE2EDuration="2.27899981s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:19:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:19:06.277223135 +0000 UTC m=+626.285414409" watchObservedRunningTime="2026-11-03 13:19:06.27899981 +0000 UTC m=+626.287191087" Nov 03 13:19:06.324939 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:19:06.324913 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-17adbf-5965f57bff-h2rr9"] Nov 03 13:19:06.327174 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:19:06.327144 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda00c1abc_3c3b_4829_94ce_f0c1bcc6d862.slice/crio-14940865b97448433e901b8177af7b1316ee6a631847799b2f78def427cfaa53 WatchSource:0}: Error finding container 14940865b97448433e901b8177af7b1316ee6a631847799b2f78def427cfaa53: Status 404 returned error can't find the container with id 14940865b97448433e901b8177af7b1316ee6a631847799b2f78def427cfaa53 Nov 03 13:19:07.268231 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:19:07.268188 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-17adbf-5965f57bff-h2rr9" event={"ID":"a00c1abc-3c3b-4829-94ce-f0c1bcc6d862","Type":"ContainerStarted","Data":"73645f59350e35fbe8726e1d43475ffab44d1b96d91a9a3eb8a981c678205530"} Nov 03 13:19:07.268748 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:19:07.268249 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-17adbf-5965f57bff-h2rr9" event={"ID":"a00c1abc-3c3b-4829-94ce-f0c1bcc6d862","Type":"ContainerStarted","Data":"14940865b97448433e901b8177af7b1316ee6a631847799b2f78def427cfaa53"} Nov 03 13:19:07.291939 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:19:07.291878 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-17adbf-5965f57bff-h2rr9" podStartSLOduration=1.924885777 podStartE2EDuration="2.291863669s" totalImagesPullingTime="366.977892ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:19:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:19:07.289461223 +0000 UTC m=+627.297652501" watchObservedRunningTime="2026-11-03 13:19:07.291863669 +0000 UTC m=+627.300054943" Nov 03 13:19:08.275320 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:19:08.275276 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-17adbf-5965f57bff-h2rr9" Nov 03 13:19:09.508476 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:19:09.508435 5024 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d7mwz" containerName="extractor" Nov 03 13:19:14.301662 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:19:14.301632 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-17adbf-5965f57bff-h2rr9" Nov 03 13:19:16.167828 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:19:16.167787 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-d2rlz"] Nov 03 13:19:16.175747 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:19:16.175715 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-d2rlz" Nov 03 13:19:16.188305 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:19:16.181224 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-d2rlz"] Nov 03 13:19:16.191681 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:19:16.186875 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-e440bc-openshift-default-dockercfg-84hml\"" Nov 03 13:19:16.323948 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:19:16.323911 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af43a1c8-fad2-4b80-a0e9-65ab23c3612e-istio-envoy\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-d2rlz\" (UID: \"af43a1c8-fad2-4b80-a0e9-65ab23c3612e\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-d2rlz" Nov 03 13:19:16.324103 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:19:16.323991 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af43a1c8-fad2-4b80-a0e9-65ab23c3612e-istio-data\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-d2rlz\" (UID: \"af43a1c8-fad2-4b80-a0e9-65ab23c3612e\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-d2rlz" Nov 03 13:19:16.324103 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:19:16.324020 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af43a1c8-fad2-4b80-a0e9-65ab23c3612e-workload-socket\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-d2rlz\" (UID: \"af43a1c8-fad2-4b80-a0e9-65ab23c3612e\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-d2rlz" Nov 03 13:19:16.324190 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:19:16.324116 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af43a1c8-fad2-4b80-a0e9-65ab23c3612e-istio-token\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-d2rlz\" (UID: \"af43a1c8-fad2-4b80-a0e9-65ab23c3612e\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-d2rlz" Nov 03 13:19:16.324190 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:19:16.324183 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m55fg\" (UniqueName: \"kubernetes.io/projected/af43a1c8-fad2-4b80-a0e9-65ab23c3612e-kube-api-access-m55fg\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-d2rlz\" (UID: \"af43a1c8-fad2-4b80-a0e9-65ab23c3612e\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-d2rlz" Nov 03 13:19:16.324307 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:19:16.324280 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af43a1c8-fad2-4b80-a0e9-65ab23c3612e-istiod-ca-cert\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-d2rlz\" (UID: \"af43a1c8-fad2-4b80-a0e9-65ab23c3612e\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-d2rlz" Nov 03 13:19:16.324359 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:19:16.324321 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af43a1c8-fad2-4b80-a0e9-65ab23c3612e-workload-certs\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-d2rlz\" (UID: \"af43a1c8-fad2-4b80-a0e9-65ab23c3612e\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-d2rlz" Nov 03 13:19:16.324359 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:19:16.324354 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af43a1c8-fad2-4b80-a0e9-65ab23c3612e-istio-podinfo\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-d2rlz\" (UID: \"af43a1c8-fad2-4b80-a0e9-65ab23c3612e\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-d2rlz" Nov 03 13:19:16.324483 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:19:16.324466 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af43a1c8-fad2-4b80-a0e9-65ab23c3612e-credential-socket\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-d2rlz\" (UID: \"af43a1c8-fad2-4b80-a0e9-65ab23c3612e\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-d2rlz" Nov 03 13:19:16.425256 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:19:16.425144 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af43a1c8-fad2-4b80-a0e9-65ab23c3612e-workload-socket\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-d2rlz\" (UID: \"af43a1c8-fad2-4b80-a0e9-65ab23c3612e\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-d2rlz" Nov 03 13:19:16.425256 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:19:16.425195 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af43a1c8-fad2-4b80-a0e9-65ab23c3612e-istio-token\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-d2rlz\" (UID: \"af43a1c8-fad2-4b80-a0e9-65ab23c3612e\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-d2rlz" Nov 03 13:19:16.425256 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:19:16.425232 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m55fg\" (UniqueName: \"kubernetes.io/projected/af43a1c8-fad2-4b80-a0e9-65ab23c3612e-kube-api-access-m55fg\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-d2rlz\" (UID: \"af43a1c8-fad2-4b80-a0e9-65ab23c3612e\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-d2rlz" Nov 03 13:19:16.425588 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:19:16.425332 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af43a1c8-fad2-4b80-a0e9-65ab23c3612e-istiod-ca-cert\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-d2rlz\" (UID: \"af43a1c8-fad2-4b80-a0e9-65ab23c3612e\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-d2rlz" Nov 03 13:19:16.425588 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:19:16.425367 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af43a1c8-fad2-4b80-a0e9-65ab23c3612e-workload-certs\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-d2rlz\" (UID: \"af43a1c8-fad2-4b80-a0e9-65ab23c3612e\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-d2rlz" Nov 03 13:19:16.425588 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:19:16.425414 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af43a1c8-fad2-4b80-a0e9-65ab23c3612e-istio-podinfo\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-d2rlz\" (UID: \"af43a1c8-fad2-4b80-a0e9-65ab23c3612e\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-d2rlz" Nov 03 13:19:16.425588 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:19:16.425464 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af43a1c8-fad2-4b80-a0e9-65ab23c3612e-credential-socket\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-d2rlz\" (UID: \"af43a1c8-fad2-4b80-a0e9-65ab23c3612e\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-d2rlz" Nov 03 13:19:16.425588 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:19:16.425505 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af43a1c8-fad2-4b80-a0e9-65ab23c3612e-istio-envoy\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-d2rlz\" (UID: \"af43a1c8-fad2-4b80-a0e9-65ab23c3612e\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-d2rlz" Nov 03 13:19:16.425588 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:19:16.425535 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af43a1c8-fad2-4b80-a0e9-65ab23c3612e-istio-data\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-d2rlz\" (UID: \"af43a1c8-fad2-4b80-a0e9-65ab23c3612e\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-d2rlz" Nov 03 13:19:16.425891 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:19:16.425637 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af43a1c8-fad2-4b80-a0e9-65ab23c3612e-workload-socket\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-d2rlz\" (UID: \"af43a1c8-fad2-4b80-a0e9-65ab23c3612e\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-d2rlz" Nov 03 13:19:16.425951 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:19:16.425886 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af43a1c8-fad2-4b80-a0e9-65ab23c3612e-istio-data\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-d2rlz\" (UID: \"af43a1c8-fad2-4b80-a0e9-65ab23c3612e\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-d2rlz" Nov 03 13:19:16.425951 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:19:16.425923 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af43a1c8-fad2-4b80-a0e9-65ab23c3612e-credential-socket\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-d2rlz\" (UID: \"af43a1c8-fad2-4b80-a0e9-65ab23c3612e\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-d2rlz" Nov 03 13:19:16.426178 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:19:16.426156 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af43a1c8-fad2-4b80-a0e9-65ab23c3612e-workload-certs\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-d2rlz\" (UID: \"af43a1c8-fad2-4b80-a0e9-65ab23c3612e\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-d2rlz" Nov 03 13:19:16.426655 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:19:16.426630 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af43a1c8-fad2-4b80-a0e9-65ab23c3612e-istiod-ca-cert\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-d2rlz\" (UID: \"af43a1c8-fad2-4b80-a0e9-65ab23c3612e\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-d2rlz" Nov 03 13:19:16.427968 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:19:16.427940 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af43a1c8-fad2-4b80-a0e9-65ab23c3612e-istio-envoy\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-d2rlz\" (UID: \"af43a1c8-fad2-4b80-a0e9-65ab23c3612e\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-d2rlz" Nov 03 13:19:16.428253 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:19:16.428231 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af43a1c8-fad2-4b80-a0e9-65ab23c3612e-istio-podinfo\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-d2rlz\" (UID: \"af43a1c8-fad2-4b80-a0e9-65ab23c3612e\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-d2rlz" Nov 03 13:19:16.434115 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:19:16.434082 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af43a1c8-fad2-4b80-a0e9-65ab23c3612e-istio-token\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-d2rlz\" (UID: \"af43a1c8-fad2-4b80-a0e9-65ab23c3612e\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-d2rlz" Nov 03 13:19:16.434223 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:19:16.434198 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m55fg\" (UniqueName: \"kubernetes.io/projected/af43a1c8-fad2-4b80-a0e9-65ab23c3612e-kube-api-access-m55fg\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-d2rlz\" (UID: \"af43a1c8-fad2-4b80-a0e9-65ab23c3612e\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-d2rlz" Nov 03 13:19:16.517332 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:19:16.517272 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-d2rlz" Nov 03 13:19:16.821490 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:19:16.821350 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-d2rlz"] Nov 03 13:19:16.825707 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:19:16.825678 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf43a1c8_fad2_4b80_a0e9_65ab23c3612e.slice/crio-4a1615012fa67730fd4a0c11d3e10117077b9ffe5d8f2feb0a8feea72c20a1da WatchSource:0}: Error finding container 4a1615012fa67730fd4a0c11d3e10117077b9ffe5d8f2feb0a8feea72c20a1da: Status 404 returned error can't find the container with id 4a1615012fa67730fd4a0c11d3e10117077b9ffe5d8f2feb0a8feea72c20a1da Nov 03 13:19:16.828763 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:19:16.828379 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:19:16.828763 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:19:16.828446 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:19:16.828763 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:19:16.828478 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:19:17.300495 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:19:17.299483 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-17adbf-696b5d5d7f-9fqx5" Nov 03 13:19:17.345395 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:19:17.345352 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-d2rlz" event={"ID":"af43a1c8-fad2-4b80-a0e9-65ab23c3612e","Type":"ContainerStarted","Data":"638d0cf057c395a54491109576c2829707ad1164879abf8ed2ff73b9afcd49d1"} Nov 03 13:19:17.345577 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:19:17.345430 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-d2rlz" event={"ID":"af43a1c8-fad2-4b80-a0e9-65ab23c3612e","Type":"ContainerStarted","Data":"4a1615012fa67730fd4a0c11d3e10117077b9ffe5d8f2feb0a8feea72c20a1da"} Nov 03 13:19:17.368413 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:19:17.368356 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-d2rlz" podStartSLOduration=1.368341252 podStartE2EDuration="1.368341252s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:19:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:19:17.366768599 +0000 UTC m=+637.374959877" watchObservedRunningTime="2026-11-03 13:19:17.368341252 +0000 UTC m=+637.376532609" Nov 03 13:19:17.518599 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:19:17.518558 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-d2rlz" Nov 03 13:19:17.519699 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:19:17.519669 5024 patch_prober.go:29] interesting pod/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-d2rlz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Nov 03 13:19:17.519842 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:19:17.519720 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-d2rlz" podUID="af43a1c8-fad2-4b80-a0e9-65ab23c3612e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Nov 03 13:19:18.542321 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:19:18.527886 5024 patch_prober.go:29] interesting pod/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-d2rlz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Nov 03 13:19:18.542321 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:19:18.527972 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-d2rlz" podUID="af43a1c8-fad2-4b80-a0e9-65ab23c3612e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Nov 03 13:19:19.585823 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:19:19.564878 5024 patch_prober.go:29] interesting pod/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-d2rlz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Nov 03 13:19:19.585823 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:19:19.564943 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-d2rlz" podUID="af43a1c8-fad2-4b80-a0e9-65ab23c3612e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Nov 03 13:19:20.518277 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:19:20.518240 5024 patch_prober.go:29] interesting pod/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-d2rlz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Nov 03 13:19:20.518538 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:19:20.518515 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-d2rlz" podUID="af43a1c8-fad2-4b80-a0e9-65ab23c3612e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Nov 03 13:19:21.517962 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:19:21.517915 5024 patch_prober.go:29] interesting pod/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-d2rlz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Nov 03 13:19:21.518618 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:19:21.518579 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-d2rlz" podUID="af43a1c8-fad2-4b80-a0e9-65ab23c3612e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Nov 03 13:19:22.518417 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:19:22.518375 5024 patch_prober.go:29] interesting pod/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-d2rlz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Nov 03 13:19:22.518805 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:19:22.518443 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-d2rlz" podUID="af43a1c8-fad2-4b80-a0e9-65ab23c3612e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Nov 03 13:19:23.517989 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:19:23.517943 5024 patch_prober.go:29] interesting pod/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-d2rlz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Nov 03 13:19:23.518171 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:19:23.518019 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-d2rlz" podUID="af43a1c8-fad2-4b80-a0e9-65ab23c3612e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Nov 03 13:19:24.193707 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:19:24.193668 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-db71f4-6b76577dd7-cgpqq"] Nov 03 13:19:24.198132 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:19:24.198111 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-db71f4-6b76577dd7-cgpqq" Nov 03 13:19:24.205071 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:19:24.205041 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-db71f4\"" Nov 03 13:19:24.205245 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:19:24.205106 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-db71f4-dockercfg-2m5zc\"" Nov 03 13:19:24.208378 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:19:24.208352 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-db71f4-6b76577dd7-cgpqq"] Nov 03 13:19:24.349167 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:19:24.349062 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d8f419b-6347-4761-a260-a35769a80f40-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-db71f4-6b76577dd7-cgpqq\" (UID: \"4d8f419b-6347-4761-a260-a35769a80f40\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-db71f4-6b76577dd7-cgpqq" Nov 03 13:19:24.349381 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:19:24.349247 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4wr4c\" (UniqueName: \"kubernetes.io/projected/4d8f419b-6347-4761-a260-a35769a80f40-kube-api-access-4wr4c\") pod \"payload-pre-processing-e2e-inf-b-db71f4-6b76577dd7-cgpqq\" (UID: \"4d8f419b-6347-4761-a260-a35769a80f40\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-db71f4-6b76577dd7-cgpqq" Nov 03 13:19:24.451542 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:19:24.451445 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d8f419b-6347-4761-a260-a35769a80f40-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-db71f4-6b76577dd7-cgpqq\" (UID: \"4d8f419b-6347-4761-a260-a35769a80f40\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-db71f4-6b76577dd7-cgpqq" Nov 03 13:19:24.451698 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:19:24.451548 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4wr4c\" (UniqueName: \"kubernetes.io/projected/4d8f419b-6347-4761-a260-a35769a80f40-kube-api-access-4wr4c\") pod \"payload-pre-processing-e2e-inf-b-db71f4-6b76577dd7-cgpqq\" (UID: \"4d8f419b-6347-4761-a260-a35769a80f40\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-db71f4-6b76577dd7-cgpqq" Nov 03 13:19:24.452321 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:19:24.452266 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d8f419b-6347-4761-a260-a35769a80f40-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-db71f4-6b76577dd7-cgpqq\" (UID: \"4d8f419b-6347-4761-a260-a35769a80f40\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-db71f4-6b76577dd7-cgpqq" Nov 03 13:19:24.460756 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:19:24.460726 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4wr4c\" (UniqueName: \"kubernetes.io/projected/4d8f419b-6347-4761-a260-a35769a80f40-kube-api-access-4wr4c\") pod \"payload-pre-processing-e2e-inf-b-db71f4-6b76577dd7-cgpqq\" (UID: \"4d8f419b-6347-4761-a260-a35769a80f40\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-db71f4-6b76577dd7-cgpqq" Nov 03 13:19:24.480781 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:19:24.480745 5024 generic.go:356] "Generic (PLEG): container finished" podID="42d727c9-2a2b-4a3d-afb7-c0d2ee617b7f" containerID="60b1fa6c8cab7eb811c819276689b8f963f74a5ff0074beb66d743617ee6f9ba" exitCode=0 Nov 03 13:19:24.480951 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:19:24.480817 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-17adbf-openshift-default-75f4498d49-hsbdd" event={"ID":"42d727c9-2a2b-4a3d-afb7-c0d2ee617b7f","Type":"ContainerDied","Data":"60b1fa6c8cab7eb811c819276689b8f963f74a5ff0074beb66d743617ee6f9ba"} Nov 03 13:19:24.481269 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:19:24.481242 5024 scope.go:122] "RemoveContainer" containerID="60b1fa6c8cab7eb811c819276689b8f963f74a5ff0074beb66d743617ee6f9ba" Nov 03 13:19:24.482688 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:19:24.482657 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:19:24.482777 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:19:24.482731 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:19:24.482777 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:19:24.482764 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:19:24.516734 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:19:24.516700 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-db71f4-6b76577dd7-cgpqq" Nov 03 13:19:24.524636 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:19:24.523653 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-d2rlz" Nov 03 13:19:24.525004 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:19:24.524827 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-d2rlz" Nov 03 13:19:24.525979 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:19:24.525960 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-d2rlz" Nov 03 13:19:24.822676 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:19:24.822642 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-db71f4-6b76577dd7-cgpqq"] Nov 03 13:19:25.505038 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:19:25.504987 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-17adbf-openshift-default-75f4498d49-hsbdd" event={"ID":"42d727c9-2a2b-4a3d-afb7-c0d2ee617b7f","Type":"ContainerStarted","Data":"6ab9d2c6cbd7023205c3c125b210bde795226750662dd659a173d9d75c5fdc6f"} Nov 03 13:19:25.508014 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:19:25.507975 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-db71f4-6b76577dd7-cgpqq" event={"ID":"4d8f419b-6347-4761-a260-a35769a80f40","Type":"ContainerStarted","Data":"797bc8e0c83843263f9ae2e8fb172dc13dbfd01bd394d49896c72afdf7fb94ec"} Nov 03 13:19:25.508179 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:19:25.508049 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-db71f4-6b76577dd7-cgpqq" event={"ID":"4d8f419b-6347-4761-a260-a35769a80f40","Type":"ContainerStarted","Data":"be9729557c89cb2710a5ff18b4c780c7646a5868e3f3162850a4ad1c22c286b2"} Nov 03 13:19:25.508228 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:19:25.508211 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-db71f4-6b76577dd7-cgpqq" Nov 03 13:19:25.543721 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:19:25.543666 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-db71f4-6b76577dd7-cgpqq" podStartSLOduration=1.5436509470000002 podStartE2EDuration="1.543650947s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:19:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:19:25.542323349 +0000 UTC m=+645.550514639" watchObservedRunningTime="2026-11-03 13:19:25.543650947 +0000 UTC m=+645.551842224" Nov 03 13:19:26.002322 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:19:26.002023 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-17adbf-openshift-default-75f4498d49-hsbdd" Nov 03 13:19:26.002547 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:19:26.002446 5024 patch_prober.go:29] interesting pod/e2e-inf-a-17adbf-openshift-default-75f4498d49-hsbdd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Nov 03 13:19:26.002547 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:19:26.002496 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-17adbf-openshift-default-75f4498d49-hsbdd" podUID="42d727c9-2a2b-4a3d-afb7-c0d2ee617b7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Nov 03 13:19:26.999308 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:19:26.998697 5024 patch_prober.go:29] interesting pod/e2e-inf-a-17adbf-openshift-default-75f4498d49-hsbdd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Nov 03 13:19:26.999308 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:19:26.998778 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-17adbf-openshift-default-75f4498d49-hsbdd" podUID="42d727c9-2a2b-4a3d-afb7-c0d2ee617b7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Nov 03 13:19:27.995696 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:19:27.995651 5024 patch_prober.go:29] interesting pod/e2e-inf-a-17adbf-openshift-default-75f4498d49-hsbdd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Nov 03 13:19:27.995913 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:19:27.995737 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-17adbf-openshift-default-75f4498d49-hsbdd" podUID="42d727c9-2a2b-4a3d-afb7-c0d2ee617b7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Nov 03 13:19:28.995778 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:19:28.995736 5024 patch_prober.go:29] interesting pod/e2e-inf-a-17adbf-openshift-default-75f4498d49-hsbdd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Nov 03 13:19:28.996214 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:19:28.995808 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-17adbf-openshift-default-75f4498d49-hsbdd" podUID="42d727c9-2a2b-4a3d-afb7-c0d2ee617b7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Nov 03 13:19:29.992763 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:19:29.992721 5024 patch_prober.go:29] interesting pod/e2e-inf-a-17adbf-openshift-default-75f4498d49-hsbdd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Nov 03 13:19:29.993077 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:19:29.993045 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-17adbf-openshift-default-75f4498d49-hsbdd" podUID="42d727c9-2a2b-4a3d-afb7-c0d2ee617b7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Nov 03 13:19:31.090577 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:19:31.090540 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-17adbf-openshift-default-75f4498d49-hsbdd" Nov 03 13:19:31.091979 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:19:31.091952 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-17adbf-openshift-default-75f4498d49-hsbdd" Nov 03 13:19:31.092958 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:19:31.092935 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-17adbf-openshift-default-75f4498d49-hsbdd" Nov 03 13:19:36.518125 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:19:36.518093 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-db71f4-6b76577dd7-cgpqq" Nov 03 13:19:39.507891 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:19:39.507805 5024 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d7mwz" containerName="extractor" Nov 03 13:19:43.882337 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:19:43.882275 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-db71f4/test-model-db71f4-kserve-6cf694c5f9-57cmd"] Nov 03 13:19:43.887728 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:19:43.887696 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-db71f4/test-model-db71f4-kserve-6cf694c5f9-57cmd" Nov 03 13:19:43.897320 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:19:43.897270 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-db71f4\"/\"default-dockercfg-hhssj\"" Nov 03 13:19:43.900644 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:19:43.900242 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-db71f4/test-model-db71f4-kserve-6cf694c5f9-57cmd"] Nov 03 13:19:43.901147 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:19:43.901031 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-db71f4\"/\"test-model-db71f4-kserve-self-signed-certs\"" Nov 03 13:19:43.901230 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:19:43.901171 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-db71f4\"/\"kube-root-ca.crt\"" Nov 03 13:19:43.904017 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:19:43.903989 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-db71f4\"/\"openshift-service-ca.crt\"" Nov 03 13:19:43.974318 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:19:43.974263 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/029ef988-b71c-4de2-9b27-d7506a869ed8-home\") pod \"test-model-db71f4-kserve-6cf694c5f9-57cmd\" (UID: \"029ef988-b71c-4de2-9b27-d7506a869ed8\") " pod="ai-tenant-e2e-inf-b-db71f4/test-model-db71f4-kserve-6cf694c5f9-57cmd" Nov 03 13:19:43.974525 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:19:43.974412 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/029ef988-b71c-4de2-9b27-d7506a869ed8-tmp-dir\") pod \"test-model-db71f4-kserve-6cf694c5f9-57cmd\" (UID: \"029ef988-b71c-4de2-9b27-d7506a869ed8\") " pod="ai-tenant-e2e-inf-b-db71f4/test-model-db71f4-kserve-6cf694c5f9-57cmd" Nov 03 13:19:43.974525 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:19:43.974437 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jrjct\" (UniqueName: \"kubernetes.io/projected/029ef988-b71c-4de2-9b27-d7506a869ed8-kube-api-access-jrjct\") pod \"test-model-db71f4-kserve-6cf694c5f9-57cmd\" (UID: \"029ef988-b71c-4de2-9b27-d7506a869ed8\") " pod="ai-tenant-e2e-inf-b-db71f4/test-model-db71f4-kserve-6cf694c5f9-57cmd" Nov 03 13:19:43.974525 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:19:43.974515 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/029ef988-b71c-4de2-9b27-d7506a869ed8-dshm\") pod \"test-model-db71f4-kserve-6cf694c5f9-57cmd\" (UID: \"029ef988-b71c-4de2-9b27-d7506a869ed8\") " pod="ai-tenant-e2e-inf-b-db71f4/test-model-db71f4-kserve-6cf694c5f9-57cmd" Nov 03 13:19:43.974715 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:19:43.974546 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/029ef988-b71c-4de2-9b27-d7506a869ed8-tls-certs\") pod \"test-model-db71f4-kserve-6cf694c5f9-57cmd\" (UID: \"029ef988-b71c-4de2-9b27-d7506a869ed8\") " pod="ai-tenant-e2e-inf-b-db71f4/test-model-db71f4-kserve-6cf694c5f9-57cmd" Nov 03 13:19:43.974715 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:19:43.974603 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/029ef988-b71c-4de2-9b27-d7506a869ed8-model-cache\") pod \"test-model-db71f4-kserve-6cf694c5f9-57cmd\" (UID: \"029ef988-b71c-4de2-9b27-d7506a869ed8\") " pod="ai-tenant-e2e-inf-b-db71f4/test-model-db71f4-kserve-6cf694c5f9-57cmd" Nov 03 13:19:44.075925 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:19:44.075880 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/029ef988-b71c-4de2-9b27-d7506a869ed8-home\") pod \"test-model-db71f4-kserve-6cf694c5f9-57cmd\" (UID: \"029ef988-b71c-4de2-9b27-d7506a869ed8\") " pod="ai-tenant-e2e-inf-b-db71f4/test-model-db71f4-kserve-6cf694c5f9-57cmd" Nov 03 13:19:44.076126 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:19:44.075996 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/029ef988-b71c-4de2-9b27-d7506a869ed8-tmp-dir\") pod \"test-model-db71f4-kserve-6cf694c5f9-57cmd\" (UID: \"029ef988-b71c-4de2-9b27-d7506a869ed8\") " pod="ai-tenant-e2e-inf-b-db71f4/test-model-db71f4-kserve-6cf694c5f9-57cmd" Nov 03 13:19:44.076126 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:19:44.076030 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrjct\" (UniqueName: \"kubernetes.io/projected/029ef988-b71c-4de2-9b27-d7506a869ed8-kube-api-access-jrjct\") pod \"test-model-db71f4-kserve-6cf694c5f9-57cmd\" (UID: \"029ef988-b71c-4de2-9b27-d7506a869ed8\") " pod="ai-tenant-e2e-inf-b-db71f4/test-model-db71f4-kserve-6cf694c5f9-57cmd" Nov 03 13:19:44.076126 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:19:44.076067 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/029ef988-b71c-4de2-9b27-d7506a869ed8-dshm\") pod \"test-model-db71f4-kserve-6cf694c5f9-57cmd\" (UID: \"029ef988-b71c-4de2-9b27-d7506a869ed8\") " pod="ai-tenant-e2e-inf-b-db71f4/test-model-db71f4-kserve-6cf694c5f9-57cmd" Nov 03 13:19:44.076126 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:19:44.076102 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/029ef988-b71c-4de2-9b27-d7506a869ed8-tls-certs\") pod \"test-model-db71f4-kserve-6cf694c5f9-57cmd\" (UID: \"029ef988-b71c-4de2-9b27-d7506a869ed8\") " pod="ai-tenant-e2e-inf-b-db71f4/test-model-db71f4-kserve-6cf694c5f9-57cmd" Nov 03 13:19:44.076364 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:19:44.076156 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/029ef988-b71c-4de2-9b27-d7506a869ed8-model-cache\") pod \"test-model-db71f4-kserve-6cf694c5f9-57cmd\" (UID: \"029ef988-b71c-4de2-9b27-d7506a869ed8\") " pod="ai-tenant-e2e-inf-b-db71f4/test-model-db71f4-kserve-6cf694c5f9-57cmd" Nov 03 13:19:44.076557 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:19:44.076515 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/029ef988-b71c-4de2-9b27-d7506a869ed8-tmp-dir\") pod \"test-model-db71f4-kserve-6cf694c5f9-57cmd\" (UID: \"029ef988-b71c-4de2-9b27-d7506a869ed8\") " pod="ai-tenant-e2e-inf-b-db71f4/test-model-db71f4-kserve-6cf694c5f9-57cmd" Nov 03 13:19:44.076637 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:19:44.076562 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/029ef988-b71c-4de2-9b27-d7506a869ed8-model-cache\") pod \"test-model-db71f4-kserve-6cf694c5f9-57cmd\" (UID: \"029ef988-b71c-4de2-9b27-d7506a869ed8\") " pod="ai-tenant-e2e-inf-b-db71f4/test-model-db71f4-kserve-6cf694c5f9-57cmd" Nov 03 13:19:44.076692 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:19:44.076670 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/029ef988-b71c-4de2-9b27-d7506a869ed8-home\") pod \"test-model-db71f4-kserve-6cf694c5f9-57cmd\" (UID: \"029ef988-b71c-4de2-9b27-d7506a869ed8\") " pod="ai-tenant-e2e-inf-b-db71f4/test-model-db71f4-kserve-6cf694c5f9-57cmd" Nov 03 13:19:44.079041 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:19:44.079012 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/029ef988-b71c-4de2-9b27-d7506a869ed8-dshm\") pod \"test-model-db71f4-kserve-6cf694c5f9-57cmd\" (UID: \"029ef988-b71c-4de2-9b27-d7506a869ed8\") " pod="ai-tenant-e2e-inf-b-db71f4/test-model-db71f4-kserve-6cf694c5f9-57cmd" Nov 03 13:19:44.079487 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:19:44.079395 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/029ef988-b71c-4de2-9b27-d7506a869ed8-tls-certs\") pod \"test-model-db71f4-kserve-6cf694c5f9-57cmd\" (UID: \"029ef988-b71c-4de2-9b27-d7506a869ed8\") " pod="ai-tenant-e2e-inf-b-db71f4/test-model-db71f4-kserve-6cf694c5f9-57cmd" Nov 03 13:19:44.085200 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:19:44.085170 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrjct\" (UniqueName: \"kubernetes.io/projected/029ef988-b71c-4de2-9b27-d7506a869ed8-kube-api-access-jrjct\") pod \"test-model-db71f4-kserve-6cf694c5f9-57cmd\" (UID: \"029ef988-b71c-4de2-9b27-d7506a869ed8\") " pod="ai-tenant-e2e-inf-b-db71f4/test-model-db71f4-kserve-6cf694c5f9-57cmd" Nov 03 13:19:44.213546 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:19:44.210517 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-db71f4/test-model-db71f4-kserve-6cf694c5f9-57cmd" Nov 03 13:19:44.503076 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:19:44.503046 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-db71f4/test-model-db71f4-kserve-6cf694c5f9-57cmd"] Nov 03 13:19:44.505115 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:19:44.505083 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod029ef988_b71c_4de2_9b27_d7506a869ed8.slice/crio-2af654e55d22a1a25e3cc42d6a33de0bdb59e9e583c07de791cdc299a096a530 WatchSource:0}: Error finding container 2af654e55d22a1a25e3cc42d6a33de0bdb59e9e583c07de791cdc299a096a530: Status 404 returned error can't find the container with id 2af654e55d22a1a25e3cc42d6a33de0bdb59e9e583c07de791cdc299a096a530 Nov 03 13:19:44.651818 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:19:44.651776 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-db71f4/test-model-db71f4-kserve-6cf694c5f9-57cmd" event={"ID":"029ef988-b71c-4de2-9b27-d7506a869ed8","Type":"ContainerStarted","Data":"a1780f1a3eaddf562f65c244668df9e32e9062c2e94496e40f382672af4a332e"} Nov 03 13:19:44.652006 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:19:44.651833 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-db71f4/test-model-db71f4-kserve-6cf694c5f9-57cmd" event={"ID":"029ef988-b71c-4de2-9b27-d7506a869ed8","Type":"ContainerStarted","Data":"2af654e55d22a1a25e3cc42d6a33de0bdb59e9e583c07de791cdc299a096a530"} Nov 03 13:19:44.675658 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:19:44.675605 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-db71f4/test-model-db71f4-kserve-6cf694c5f9-57cmd" podStartSLOduration=1.67558942 podStartE2EDuration="1.67558942s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:19:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:19:44.672901279 +0000 UTC m=+664.681092575" watchObservedRunningTime="2026-11-03 13:19:44.67558942 +0000 UTC m=+664.683780700" Nov 03 13:19:45.211171 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:19:45.211138 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-db71f4/test-model-db71f4-kserve-6cf694c5f9-57cmd" Nov 03 13:19:54.211044 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:19:54.211002 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-db71f4/test-model-db71f4-kserve-6cf694c5f9-57cmd" Nov 03 13:19:54.224813 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:19:54.224787 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-db71f4/test-model-db71f4-kserve-6cf694c5f9-57cmd" Nov 03 13:19:54.762446 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:19:54.762408 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-db71f4/test-model-db71f4-kserve-6cf694c5f9-57cmd" Nov 03 13:20:01.259028 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:01.258988 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-d7gxh"] Nov 03 13:20:01.267391 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:01.267361 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-d7gxh" Nov 03 13:20:01.276340 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:01.276307 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-d7gxh"] Nov 03 13:20:01.308119 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:01.308086 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ec0db42-ee6d-496c-8792-a17fe716e820-credential-socket\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-d7gxh\" (UID: \"6ec0db42-ee6d-496c-8792-a17fe716e820\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-d7gxh" Nov 03 13:20:01.308321 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:01.308124 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ec0db42-ee6d-496c-8792-a17fe716e820-istio-data\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-d7gxh\" (UID: \"6ec0db42-ee6d-496c-8792-a17fe716e820\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-d7gxh" Nov 03 13:20:01.308321 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:01.308212 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ec0db42-ee6d-496c-8792-a17fe716e820-istiod-ca-cert\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-d7gxh\" (UID: \"6ec0db42-ee6d-496c-8792-a17fe716e820\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-d7gxh" Nov 03 13:20:01.308321 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:01.308294 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ec0db42-ee6d-496c-8792-a17fe716e820-istio-envoy\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-d7gxh\" (UID: \"6ec0db42-ee6d-496c-8792-a17fe716e820\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-d7gxh" Nov 03 13:20:01.308439 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:01.308339 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fdr27\" (UniqueName: \"kubernetes.io/projected/6ec0db42-ee6d-496c-8792-a17fe716e820-kube-api-access-fdr27\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-d7gxh\" (UID: \"6ec0db42-ee6d-496c-8792-a17fe716e820\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-d7gxh" Nov 03 13:20:01.308439 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:01.308380 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ec0db42-ee6d-496c-8792-a17fe716e820-workload-certs\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-d7gxh\" (UID: \"6ec0db42-ee6d-496c-8792-a17fe716e820\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-d7gxh" Nov 03 13:20:01.308439 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:01.308426 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ec0db42-ee6d-496c-8792-a17fe716e820-istio-podinfo\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-d7gxh\" (UID: \"6ec0db42-ee6d-496c-8792-a17fe716e820\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-d7gxh" Nov 03 13:20:01.308533 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:01.308460 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ec0db42-ee6d-496c-8792-a17fe716e820-workload-socket\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-d7gxh\" (UID: \"6ec0db42-ee6d-496c-8792-a17fe716e820\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-d7gxh" Nov 03 13:20:01.308533 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:01.308529 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ec0db42-ee6d-496c-8792-a17fe716e820-istio-token\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-d7gxh\" (UID: \"6ec0db42-ee6d-496c-8792-a17fe716e820\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-d7gxh" Nov 03 13:20:01.409628 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:01.409589 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ec0db42-ee6d-496c-8792-a17fe716e820-istio-data\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-d7gxh\" (UID: \"6ec0db42-ee6d-496c-8792-a17fe716e820\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-d7gxh" Nov 03 13:20:01.409813 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:01.409653 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ec0db42-ee6d-496c-8792-a17fe716e820-istiod-ca-cert\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-d7gxh\" (UID: \"6ec0db42-ee6d-496c-8792-a17fe716e820\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-d7gxh" Nov 03 13:20:01.409813 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:01.409717 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ec0db42-ee6d-496c-8792-a17fe716e820-istio-envoy\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-d7gxh\" (UID: \"6ec0db42-ee6d-496c-8792-a17fe716e820\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-d7gxh" Nov 03 13:20:01.410126 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:01.409859 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdr27\" (UniqueName: \"kubernetes.io/projected/6ec0db42-ee6d-496c-8792-a17fe716e820-kube-api-access-fdr27\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-d7gxh\" (UID: \"6ec0db42-ee6d-496c-8792-a17fe716e820\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-d7gxh" Nov 03 13:20:01.410126 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:01.409911 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ec0db42-ee6d-496c-8792-a17fe716e820-workload-certs\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-d7gxh\" (UID: \"6ec0db42-ee6d-496c-8792-a17fe716e820\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-d7gxh" Nov 03 13:20:01.410126 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:01.409945 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ec0db42-ee6d-496c-8792-a17fe716e820-istio-podinfo\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-d7gxh\" (UID: \"6ec0db42-ee6d-496c-8792-a17fe716e820\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-d7gxh" Nov 03 13:20:01.410126 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:01.410063 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ec0db42-ee6d-496c-8792-a17fe716e820-istio-data\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-d7gxh\" (UID: \"6ec0db42-ee6d-496c-8792-a17fe716e820\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-d7gxh" Nov 03 13:20:01.410126 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:01.410133 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ec0db42-ee6d-496c-8792-a17fe716e820-workload-socket\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-d7gxh\" (UID: \"6ec0db42-ee6d-496c-8792-a17fe716e820\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-d7gxh" Nov 03 13:20:01.410606 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:01.410175 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ec0db42-ee6d-496c-8792-a17fe716e820-istio-token\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-d7gxh\" (UID: \"6ec0db42-ee6d-496c-8792-a17fe716e820\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-d7gxh" Nov 03 13:20:01.410606 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:01.410330 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ec0db42-ee6d-496c-8792-a17fe716e820-credential-socket\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-d7gxh\" (UID: \"6ec0db42-ee6d-496c-8792-a17fe716e820\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-d7gxh" Nov 03 13:20:01.410715 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:01.410666 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ec0db42-ee6d-496c-8792-a17fe716e820-workload-socket\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-d7gxh\" (UID: \"6ec0db42-ee6d-496c-8792-a17fe716e820\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-d7gxh" Nov 03 13:20:01.410715 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:01.410689 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ec0db42-ee6d-496c-8792-a17fe716e820-istiod-ca-cert\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-d7gxh\" (UID: \"6ec0db42-ee6d-496c-8792-a17fe716e820\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-d7gxh" Nov 03 13:20:01.411719 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:01.411688 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ec0db42-ee6d-496c-8792-a17fe716e820-workload-certs\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-d7gxh\" (UID: \"6ec0db42-ee6d-496c-8792-a17fe716e820\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-d7gxh" Nov 03 13:20:01.412416 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:01.412389 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ec0db42-ee6d-496c-8792-a17fe716e820-istio-envoy\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-d7gxh\" (UID: \"6ec0db42-ee6d-496c-8792-a17fe716e820\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-d7gxh" Nov 03 13:20:01.413015 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:01.412993 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ec0db42-ee6d-496c-8792-a17fe716e820-istio-podinfo\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-d7gxh\" (UID: \"6ec0db42-ee6d-496c-8792-a17fe716e820\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-d7gxh" Nov 03 13:20:01.413499 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:01.413477 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ec0db42-ee6d-496c-8792-a17fe716e820-credential-socket\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-d7gxh\" (UID: \"6ec0db42-ee6d-496c-8792-a17fe716e820\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-d7gxh" Nov 03 13:20:01.425153 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:01.425115 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdr27\" (UniqueName: \"kubernetes.io/projected/6ec0db42-ee6d-496c-8792-a17fe716e820-kube-api-access-fdr27\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-d7gxh\" (UID: \"6ec0db42-ee6d-496c-8792-a17fe716e820\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-d7gxh" Nov 03 13:20:01.426207 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:01.426171 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ec0db42-ee6d-496c-8792-a17fe716e820-istio-token\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-d7gxh\" (UID: \"6ec0db42-ee6d-496c-8792-a17fe716e820\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-d7gxh" Nov 03 13:20:01.588991 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:01.588902 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-d7gxh" Nov 03 13:20:01.873079 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:01.873050 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-d7gxh"] Nov 03 13:20:01.875157 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:20:01.875130 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ec0db42_ee6d_496c_8792_a17fe716e820.slice/crio-aff61b0630e8ffc87d23002febe3c6f023c341ce8d198d6e11dd6ac97cc63138 WatchSource:0}: Error finding container aff61b0630e8ffc87d23002febe3c6f023c341ce8d198d6e11dd6ac97cc63138: Status 404 returned error can't find the container with id aff61b0630e8ffc87d23002febe3c6f023c341ce8d198d6e11dd6ac97cc63138 Nov 03 13:20:01.877519 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:01.877487 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:20:01.877617 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:01.877559 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:20:01.877617 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:01.877590 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:20:02.844322 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:02.842989 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-d7gxh" event={"ID":"6ec0db42-ee6d-496c-8792-a17fe716e820","Type":"ContainerStarted","Data":"6c3dc16a75a1544367fe596cd5d106649ccf4e7a165aa80d943b388a325928d7"} Nov 03 13:20:02.844322 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:02.843063 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-d7gxh" event={"ID":"6ec0db42-ee6d-496c-8792-a17fe716e820","Type":"ContainerStarted","Data":"aff61b0630e8ffc87d23002febe3c6f023c341ce8d198d6e11dd6ac97cc63138"} Nov 03 13:20:02.868442 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:02.868384 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-d7gxh" podStartSLOduration=1.868368062 podStartE2EDuration="1.868368062s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:20:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:20:02.867012799 +0000 UTC m=+682.875204077" watchObservedRunningTime="2026-11-03 13:20:02.868368062 +0000 UTC m=+682.876559339" Nov 03 13:20:03.593307 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:03.592301 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-d7gxh" Nov 03 13:20:03.593307 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:03.592497 5024 patch_prober.go:29] interesting pod/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-d7gxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body= Nov 03 13:20:03.593307 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:03.592557 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-d7gxh" podUID="6ec0db42-ee6d-496c-8792-a17fe716e820" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" Nov 03 13:20:04.589677 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:04.589636 5024 patch_prober.go:29] interesting pod/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-d7gxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body= Nov 03 13:20:04.590303 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:04.590253 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-d7gxh" podUID="6ec0db42-ee6d-496c-8792-a17fe716e820" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" Nov 03 13:20:05.594732 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:05.594694 5024 patch_prober.go:29] interesting pod/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-d7gxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body= Nov 03 13:20:05.595381 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:05.595344 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-d7gxh" podUID="6ec0db42-ee6d-496c-8792-a17fe716e820" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" Nov 03 13:20:06.592710 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:06.592669 5024 patch_prober.go:29] interesting pod/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-d7gxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body= Nov 03 13:20:06.592999 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:06.592967 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-d7gxh" podUID="6ec0db42-ee6d-496c-8792-a17fe716e820" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" Nov 03 13:20:07.591320 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:07.590759 5024 patch_prober.go:29] interesting pod/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-d7gxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body= Nov 03 13:20:07.591320 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:07.590822 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-d7gxh" podUID="6ec0db42-ee6d-496c-8792-a17fe716e820" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" Nov 03 13:20:08.823765 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:08.823728 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-d7gxh" Nov 03 13:20:08.824263 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:08.824113 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-d7gxh" Nov 03 13:20:08.825047 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:08.825022 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-d7gxh" Nov 03 13:20:09.517415 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:20:09.517351 5024 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d7mwz" containerName="extractor" Nov 03 13:20:31.395370 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:31.395330 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-fglkk"] Nov 03 13:20:31.401983 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:31.401958 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-fglkk" Nov 03 13:20:31.415058 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:31.415026 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-fglkk"] Nov 03 13:20:31.493049 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:31.493000 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-2jswl"] Nov 03 13:20:31.498404 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:31.498374 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-2jswl" Nov 03 13:20:31.512659 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:31.512630 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-2jswl"] Nov 03 13:20:31.566552 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:31.566510 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6d115b8-3b33-43c6-890b-a87bb608444b-istio-token\") pod \"e2e-worker-w4-c381a7-openshift-default-78bcd4695-fglkk\" (UID: \"e6d115b8-3b33-43c6-890b-a87bb608444b\") " pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-fglkk" Nov 03 13:20:31.566745 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:31.566665 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6d115b8-3b33-43c6-890b-a87bb608444b-credential-socket\") pod \"e2e-worker-w4-c381a7-openshift-default-78bcd4695-fglkk\" (UID: \"e6d115b8-3b33-43c6-890b-a87bb608444b\") " pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-fglkk" Nov 03 13:20:31.566815 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:31.566762 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6d115b8-3b33-43c6-890b-a87bb608444b-istio-data\") pod \"e2e-worker-w4-c381a7-openshift-default-78bcd4695-fglkk\" (UID: \"e6d115b8-3b33-43c6-890b-a87bb608444b\") " pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-fglkk" Nov 03 13:20:31.566815 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:31.566794 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6d115b8-3b33-43c6-890b-a87bb608444b-workload-socket\") pod \"e2e-worker-w4-c381a7-openshift-default-78bcd4695-fglkk\" (UID: \"e6d115b8-3b33-43c6-890b-a87bb608444b\") " pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-fglkk" Nov 03 13:20:31.566921 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:31.566813 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bn7nf\" (UniqueName: \"kubernetes.io/projected/e6d115b8-3b33-43c6-890b-a87bb608444b-kube-api-access-bn7nf\") pod \"e2e-worker-w4-c381a7-openshift-default-78bcd4695-fglkk\" (UID: \"e6d115b8-3b33-43c6-890b-a87bb608444b\") " pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-fglkk" Nov 03 13:20:31.566921 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:31.566832 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6d115b8-3b33-43c6-890b-a87bb608444b-istio-envoy\") pod \"e2e-worker-w4-c381a7-openshift-default-78bcd4695-fglkk\" (UID: \"e6d115b8-3b33-43c6-890b-a87bb608444b\") " pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-fglkk" Nov 03 13:20:31.566921 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:31.566856 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6d115b8-3b33-43c6-890b-a87bb608444b-istiod-ca-cert\") pod \"e2e-worker-w4-c381a7-openshift-default-78bcd4695-fglkk\" (UID: \"e6d115b8-3b33-43c6-890b-a87bb608444b\") " pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-fglkk" Nov 03 13:20:31.566921 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:31.566881 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6d115b8-3b33-43c6-890b-a87bb608444b-workload-certs\") pod \"e2e-worker-w4-c381a7-openshift-default-78bcd4695-fglkk\" (UID: \"e6d115b8-3b33-43c6-890b-a87bb608444b\") " pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-fglkk" Nov 03 13:20:31.567096 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:31.567060 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6d115b8-3b33-43c6-890b-a87bb608444b-istio-podinfo\") pod \"e2e-worker-w4-c381a7-openshift-default-78bcd4695-fglkk\" (UID: \"e6d115b8-3b33-43c6-890b-a87bb608444b\") " pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-fglkk" Nov 03 13:20:31.668539 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:31.668433 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6d115b8-3b33-43c6-890b-a87bb608444b-istio-data\") pod \"e2e-worker-w4-c381a7-openshift-default-78bcd4695-fglkk\" (UID: \"e6d115b8-3b33-43c6-890b-a87bb608444b\") " pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-fglkk" Nov 03 13:20:31.668539 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:31.668504 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6d115b8-3b33-43c6-890b-a87bb608444b-workload-socket\") pod \"e2e-worker-w4-c381a7-openshift-default-78bcd4695-fglkk\" (UID: \"e6d115b8-3b33-43c6-890b-a87bb608444b\") " pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-fglkk" Nov 03 13:20:31.668539 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:31.668533 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bn7nf\" (UniqueName: \"kubernetes.io/projected/e6d115b8-3b33-43c6-890b-a87bb608444b-kube-api-access-bn7nf\") pod \"e2e-worker-w4-c381a7-openshift-default-78bcd4695-fglkk\" (UID: \"e6d115b8-3b33-43c6-890b-a87bb608444b\") " pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-fglkk" Nov 03 13:20:31.668838 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:31.668556 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6d115b8-3b33-43c6-890b-a87bb608444b-istio-envoy\") pod \"e2e-worker-w4-c381a7-openshift-default-78bcd4695-fglkk\" (UID: \"e6d115b8-3b33-43c6-890b-a87bb608444b\") " pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-fglkk" Nov 03 13:20:31.668838 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:31.668754 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed014864-f935-4abc-8cac-8f989d8b679b-workload-certs\") pod \"e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-2jswl\" (UID: \"ed014864-f935-4abc-8cac-8f989d8b679b\") " pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-2jswl" Nov 03 13:20:31.668838 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:31.668812 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6d115b8-3b33-43c6-890b-a87bb608444b-istiod-ca-cert\") pod \"e2e-worker-w4-c381a7-openshift-default-78bcd4695-fglkk\" (UID: \"e6d115b8-3b33-43c6-890b-a87bb608444b\") " pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-fglkk" Nov 03 13:20:31.668997 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:31.668842 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6d115b8-3b33-43c6-890b-a87bb608444b-istio-data\") pod \"e2e-worker-w4-c381a7-openshift-default-78bcd4695-fglkk\" (UID: \"e6d115b8-3b33-43c6-890b-a87bb608444b\") " pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-fglkk" Nov 03 13:20:31.668997 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:31.668855 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed014864-f935-4abc-8cac-8f989d8b679b-workload-socket\") pod \"e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-2jswl\" (UID: \"ed014864-f935-4abc-8cac-8f989d8b679b\") " pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-2jswl" Nov 03 13:20:31.669062 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:31.668991 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6d115b8-3b33-43c6-890b-a87bb608444b-workload-certs\") pod \"e2e-worker-w4-c381a7-openshift-default-78bcd4695-fglkk\" (UID: \"e6d115b8-3b33-43c6-890b-a87bb608444b\") " pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-fglkk" Nov 03 13:20:31.669062 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:31.669051 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6d115b8-3b33-43c6-890b-a87bb608444b-workload-socket\") pod \"e2e-worker-w4-c381a7-openshift-default-78bcd4695-fglkk\" (UID: \"e6d115b8-3b33-43c6-890b-a87bb608444b\") " pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-fglkk" Nov 03 13:20:31.669137 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:31.669075 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjkht\" (UniqueName: \"kubernetes.io/projected/ed014864-f935-4abc-8cac-8f989d8b679b-kube-api-access-xjkht\") pod \"e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-2jswl\" (UID: \"ed014864-f935-4abc-8cac-8f989d8b679b\") " pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-2jswl" Nov 03 13:20:31.669137 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:31.669104 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed014864-f935-4abc-8cac-8f989d8b679b-credential-socket\") pod \"e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-2jswl\" (UID: \"ed014864-f935-4abc-8cac-8f989d8b679b\") " pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-2jswl" Nov 03 13:20:31.669137 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:31.669121 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed014864-f935-4abc-8cac-8f989d8b679b-istiod-ca-cert\") pod \"e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-2jswl\" (UID: \"ed014864-f935-4abc-8cac-8f989d8b679b\") " pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-2jswl" Nov 03 13:20:31.669248 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:31.669190 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6d115b8-3b33-43c6-890b-a87bb608444b-istio-podinfo\") pod \"e2e-worker-w4-c381a7-openshift-default-78bcd4695-fglkk\" (UID: \"e6d115b8-3b33-43c6-890b-a87bb608444b\") " pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-fglkk" Nov 03 13:20:31.669248 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:31.669213 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed014864-f935-4abc-8cac-8f989d8b679b-istio-data\") pod \"e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-2jswl\" (UID: \"ed014864-f935-4abc-8cac-8f989d8b679b\") " pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-2jswl" Nov 03 13:20:31.669371 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:31.669243 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6d115b8-3b33-43c6-890b-a87bb608444b-workload-certs\") pod \"e2e-worker-w4-c381a7-openshift-default-78bcd4695-fglkk\" (UID: \"e6d115b8-3b33-43c6-890b-a87bb608444b\") " pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-fglkk" Nov 03 13:20:31.669407 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:31.669375 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed014864-f935-4abc-8cac-8f989d8b679b-istio-envoy\") pod \"e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-2jswl\" (UID: \"ed014864-f935-4abc-8cac-8f989d8b679b\") " pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-2jswl" Nov 03 13:20:31.669440 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:31.669415 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6d115b8-3b33-43c6-890b-a87bb608444b-istio-token\") pod \"e2e-worker-w4-c381a7-openshift-default-78bcd4695-fglkk\" (UID: \"e6d115b8-3b33-43c6-890b-a87bb608444b\") " pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-fglkk" Nov 03 13:20:31.669571 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:31.669455 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed014864-f935-4abc-8cac-8f989d8b679b-istio-token\") pod \"e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-2jswl\" (UID: \"ed014864-f935-4abc-8cac-8f989d8b679b\") " pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-2jswl" Nov 03 13:20:31.669642 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:31.669603 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed014864-f935-4abc-8cac-8f989d8b679b-istio-podinfo\") pod \"e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-2jswl\" (UID: \"ed014864-f935-4abc-8cac-8f989d8b679b\") " pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-2jswl" Nov 03 13:20:31.669695 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:31.669548 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6d115b8-3b33-43c6-890b-a87bb608444b-istiod-ca-cert\") pod \"e2e-worker-w4-c381a7-openshift-default-78bcd4695-fglkk\" (UID: \"e6d115b8-3b33-43c6-890b-a87bb608444b\") " pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-fglkk" Nov 03 13:20:31.669695 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:31.669665 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6d115b8-3b33-43c6-890b-a87bb608444b-credential-socket\") pod \"e2e-worker-w4-c381a7-openshift-default-78bcd4695-fglkk\" (UID: \"e6d115b8-3b33-43c6-890b-a87bb608444b\") " pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-fglkk" Nov 03 13:20:31.669895 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:31.669878 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6d115b8-3b33-43c6-890b-a87bb608444b-credential-socket\") pod \"e2e-worker-w4-c381a7-openshift-default-78bcd4695-fglkk\" (UID: \"e6d115b8-3b33-43c6-890b-a87bb608444b\") " pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-fglkk" Nov 03 13:20:31.670931 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:31.670906 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6d115b8-3b33-43c6-890b-a87bb608444b-istio-envoy\") pod \"e2e-worker-w4-c381a7-openshift-default-78bcd4695-fglkk\" (UID: \"e6d115b8-3b33-43c6-890b-a87bb608444b\") " pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-fglkk" Nov 03 13:20:31.671856 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:31.671832 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6d115b8-3b33-43c6-890b-a87bb608444b-istio-podinfo\") pod \"e2e-worker-w4-c381a7-openshift-default-78bcd4695-fglkk\" (UID: \"e6d115b8-3b33-43c6-890b-a87bb608444b\") " pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-fglkk" Nov 03 13:20:31.679229 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:31.679179 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bn7nf\" (UniqueName: \"kubernetes.io/projected/e6d115b8-3b33-43c6-890b-a87bb608444b-kube-api-access-bn7nf\") pod \"e2e-worker-w4-c381a7-openshift-default-78bcd4695-fglkk\" (UID: \"e6d115b8-3b33-43c6-890b-a87bb608444b\") " pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-fglkk" Nov 03 13:20:31.681763 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:31.681736 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6d115b8-3b33-43c6-890b-a87bb608444b-istio-token\") pod \"e2e-worker-w4-c381a7-openshift-default-78bcd4695-fglkk\" (UID: \"e6d115b8-3b33-43c6-890b-a87bb608444b\") " pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-fglkk" Nov 03 13:20:31.739854 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:31.739809 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-fglkk" Nov 03 13:20:31.774628 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:31.774566 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjkht\" (UniqueName: \"kubernetes.io/projected/ed014864-f935-4abc-8cac-8f989d8b679b-kube-api-access-xjkht\") pod \"e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-2jswl\" (UID: \"ed014864-f935-4abc-8cac-8f989d8b679b\") " pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-2jswl" Nov 03 13:20:31.774803 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:31.774635 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed014864-f935-4abc-8cac-8f989d8b679b-credential-socket\") pod \"e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-2jswl\" (UID: \"ed014864-f935-4abc-8cac-8f989d8b679b\") " pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-2jswl" Nov 03 13:20:31.774803 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:31.774663 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed014864-f935-4abc-8cac-8f989d8b679b-istiod-ca-cert\") pod \"e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-2jswl\" (UID: \"ed014864-f935-4abc-8cac-8f989d8b679b\") " pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-2jswl" Nov 03 13:20:31.774803 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:31.774697 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed014864-f935-4abc-8cac-8f989d8b679b-istio-data\") pod \"e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-2jswl\" (UID: \"ed014864-f935-4abc-8cac-8f989d8b679b\") " pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-2jswl" Nov 03 13:20:31.774803 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:31.774765 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed014864-f935-4abc-8cac-8f989d8b679b-istio-envoy\") pod \"e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-2jswl\" (UID: \"ed014864-f935-4abc-8cac-8f989d8b679b\") " pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-2jswl" Nov 03 13:20:31.774803 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:31.774799 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed014864-f935-4abc-8cac-8f989d8b679b-istio-token\") pod \"e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-2jswl\" (UID: \"ed014864-f935-4abc-8cac-8f989d8b679b\") " pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-2jswl" Nov 03 13:20:31.775057 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:31.774835 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed014864-f935-4abc-8cac-8f989d8b679b-istio-podinfo\") pod \"e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-2jswl\" (UID: \"ed014864-f935-4abc-8cac-8f989d8b679b\") " pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-2jswl" Nov 03 13:20:31.775057 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:31.774986 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed014864-f935-4abc-8cac-8f989d8b679b-workload-certs\") pod \"e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-2jswl\" (UID: \"ed014864-f935-4abc-8cac-8f989d8b679b\") " pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-2jswl" Nov 03 13:20:31.775057 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:31.775028 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed014864-f935-4abc-8cac-8f989d8b679b-workload-socket\") pod \"e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-2jswl\" (UID: \"ed014864-f935-4abc-8cac-8f989d8b679b\") " pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-2jswl" Nov 03 13:20:31.775818 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:31.775617 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed014864-f935-4abc-8cac-8f989d8b679b-istio-data\") pod \"e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-2jswl\" (UID: \"ed014864-f935-4abc-8cac-8f989d8b679b\") " pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-2jswl" Nov 03 13:20:31.776168 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:31.776109 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed014864-f935-4abc-8cac-8f989d8b679b-workload-certs\") pod \"e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-2jswl\" (UID: \"ed014864-f935-4abc-8cac-8f989d8b679b\") " pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-2jswl" Nov 03 13:20:31.776528 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:31.776473 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed014864-f935-4abc-8cac-8f989d8b679b-workload-socket\") pod \"e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-2jswl\" (UID: \"ed014864-f935-4abc-8cac-8f989d8b679b\") " pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-2jswl" Nov 03 13:20:31.776861 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:31.776840 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed014864-f935-4abc-8cac-8f989d8b679b-credential-socket\") pod \"e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-2jswl\" (UID: \"ed014864-f935-4abc-8cac-8f989d8b679b\") " pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-2jswl" Nov 03 13:20:31.777049 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:31.776873 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed014864-f935-4abc-8cac-8f989d8b679b-istiod-ca-cert\") pod \"e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-2jswl\" (UID: \"ed014864-f935-4abc-8cac-8f989d8b679b\") " pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-2jswl" Nov 03 13:20:31.780304 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:31.778383 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed014864-f935-4abc-8cac-8f989d8b679b-istio-envoy\") pod \"e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-2jswl\" (UID: \"ed014864-f935-4abc-8cac-8f989d8b679b\") " pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-2jswl" Nov 03 13:20:31.780304 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:31.779031 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed014864-f935-4abc-8cac-8f989d8b679b-istio-podinfo\") pod \"e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-2jswl\" (UID: \"ed014864-f935-4abc-8cac-8f989d8b679b\") " pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-2jswl" Nov 03 13:20:31.794306 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:31.792417 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjkht\" (UniqueName: \"kubernetes.io/projected/ed014864-f935-4abc-8cac-8f989d8b679b-kube-api-access-xjkht\") pod \"e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-2jswl\" (UID: \"ed014864-f935-4abc-8cac-8f989d8b679b\") " pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-2jswl" Nov 03 13:20:31.794306 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:31.794025 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed014864-f935-4abc-8cac-8f989d8b679b-istio-token\") pod \"e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-2jswl\" (UID: \"ed014864-f935-4abc-8cac-8f989d8b679b\") " pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-2jswl" Nov 03 13:20:31.823344 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:31.822528 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-2jswl" Nov 03 13:20:32.041680 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:32.041646 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-fglkk"] Nov 03 13:20:32.042403 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:20:32.042374 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6d115b8_3b33_43c6_890b_a87bb608444b.slice/crio-d9577aea3e9d0705d737162c904d6c84a9b8f4aef6e57356126a781ef383938c WatchSource:0}: Error finding container d9577aea3e9d0705d737162c904d6c84a9b8f4aef6e57356126a781ef383938c: Status 404 returned error can't find the container with id d9577aea3e9d0705d737162c904d6c84a9b8f4aef6e57356126a781ef383938c Nov 03 13:20:32.044858 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:32.044824 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:20:32.044919 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:32.044897 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:20:32.044965 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:32.044933 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:20:32.131067 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:32.131038 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-2jswl"] Nov 03 13:20:32.131530 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:20:32.131505 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded014864_f935_4abc_8cac_8f989d8b679b.slice/crio-c027506fed1472c7aee15f47e8b2dfa093a7ac0bd24289588ae13e0bdaa669a3 WatchSource:0}: Error finding container c027506fed1472c7aee15f47e8b2dfa093a7ac0bd24289588ae13e0bdaa669a3: Status 404 returned error can't find the container with id c027506fed1472c7aee15f47e8b2dfa093a7ac0bd24289588ae13e0bdaa669a3 Nov 03 13:20:32.132335 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:32.132311 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-fglkk" event={"ID":"e6d115b8-3b33-43c6-890b-a87bb608444b","Type":"ContainerStarted","Data":"d9577aea3e9d0705d737162c904d6c84a9b8f4aef6e57356126a781ef383938c"} Nov 03 13:20:32.133455 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:32.133427 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:20:32.133520 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:32.133491 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:20:32.133560 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:32.133519 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:20:33.151446 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:33.151398 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-2jswl" event={"ID":"ed014864-f935-4abc-8cac-8f989d8b679b","Type":"ContainerStarted","Data":"ffd5a28f72afdbb8e31c47fc97af0390cac40756db95832eff4882f784417a76"} Nov 03 13:20:33.151891 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:33.151470 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-2jswl" event={"ID":"ed014864-f935-4abc-8cac-8f989d8b679b","Type":"ContainerStarted","Data":"c027506fed1472c7aee15f47e8b2dfa093a7ac0bd24289588ae13e0bdaa669a3"} Nov 03 13:20:33.153043 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:33.153020 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-fglkk" event={"ID":"e6d115b8-3b33-43c6-890b-a87bb608444b","Type":"ContainerStarted","Data":"e3d4317b51fa508f0711383ce8b044eb1bf9c9b7003d31e0cd4f1526b551615a"} Nov 03 13:20:33.201308 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:33.192069 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-2jswl" podStartSLOduration=2.192049409 podStartE2EDuration="2.192049409s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:20:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:20:33.19169759 +0000 UTC m=+713.199888868" watchObservedRunningTime="2026-11-03 13:20:33.192049409 +0000 UTC m=+713.200240688" Nov 03 13:20:33.230306 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:33.229565 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-fglkk" podStartSLOduration=2.229544437 podStartE2EDuration="2.229544437s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:20:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:20:33.228804041 +0000 UTC m=+713.236995323" watchObservedRunningTime="2026-11-03 13:20:33.229544437 +0000 UTC m=+713.237735715" Nov 03 13:20:33.743065 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:33.743028 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-fglkk" Nov 03 13:20:33.743406 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:33.743381 5024 patch_prober.go:29] interesting pod/e2e-worker-w4-c381a7-openshift-default-78bcd4695-fglkk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Nov 03 13:20:33.743631 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:33.743604 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-fglkk" podUID="e6d115b8-3b33-43c6-890b-a87bb608444b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Nov 03 13:20:33.828351 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:33.824987 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-2jswl" Nov 03 13:20:33.828351 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:33.825109 5024 patch_prober.go:29] interesting pod/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-2jswl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Nov 03 13:20:33.828351 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:33.825162 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-2jswl" podUID="ed014864-f935-4abc-8cac-8f989d8b679b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Nov 03 13:20:34.740802 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:34.740767 5024 patch_prober.go:29] interesting pod/e2e-worker-w4-c381a7-openshift-default-78bcd4695-fglkk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Nov 03 13:20:34.741275 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:34.741246 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-fglkk" podUID="e6d115b8-3b33-43c6-890b-a87bb608444b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Nov 03 13:20:34.828309 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:34.827862 5024 patch_prober.go:29] interesting pod/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-2jswl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Nov 03 13:20:34.828309 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:34.827949 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-2jswl" podUID="ed014864-f935-4abc-8cac-8f989d8b679b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Nov 03 13:20:35.747783 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:35.747747 5024 patch_prober.go:29] interesting pod/e2e-worker-w4-c381a7-openshift-default-78bcd4695-fglkk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Nov 03 13:20:35.748439 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:35.748405 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-fglkk" podUID="e6d115b8-3b33-43c6-890b-a87bb608444b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Nov 03 13:20:35.824037 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:35.824000 5024 patch_prober.go:29] interesting pod/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-2jswl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Nov 03 13:20:35.824376 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:35.824332 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-2jswl" podUID="ed014864-f935-4abc-8cac-8f989d8b679b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Nov 03 13:20:36.752309 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:36.744753 5024 patch_prober.go:29] interesting pod/e2e-worker-w4-c381a7-openshift-default-78bcd4695-fglkk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Nov 03 13:20:36.752309 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:36.744817 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-fglkk" podUID="e6d115b8-3b33-43c6-890b-a87bb608444b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Nov 03 13:20:36.827310 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:36.827054 5024 patch_prober.go:29] interesting pod/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-2jswl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Nov 03 13:20:36.827310 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:36.827125 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-2jswl" podUID="ed014864-f935-4abc-8cac-8f989d8b679b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Nov 03 13:20:37.169113 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:37.169064 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-ded926-openshift-default-6bdf65b545-pjqfb"] Nov 03 13:20:37.188619 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:37.188583 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-ded926-openshift-default-6bdf65b545-pjqfb" Nov 03 13:20:37.264541 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:37.264484 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-ded926-openshift-default-dockercfg-97zw8\"" Nov 03 13:20:37.268029 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:37.267992 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-ded926-openshift-default-6bdf65b545-pjqfb"] Nov 03 13:20:37.322760 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:37.322718 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/736b02c5-9833-4f1b-9f9a-e4c22ab23ac9-istiod-ca-cert\") pod \"e2e-extmod-a-ded926-openshift-default-6bdf65b545-pjqfb\" (UID: \"736b02c5-9833-4f1b-9f9a-e4c22ab23ac9\") " pod="openshift-ingress/e2e-extmod-a-ded926-openshift-default-6bdf65b545-pjqfb" Nov 03 13:20:37.323102 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:37.323072 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/736b02c5-9833-4f1b-9f9a-e4c22ab23ac9-istio-podinfo\") pod \"e2e-extmod-a-ded926-openshift-default-6bdf65b545-pjqfb\" (UID: \"736b02c5-9833-4f1b-9f9a-e4c22ab23ac9\") " pod="openshift-ingress/e2e-extmod-a-ded926-openshift-default-6bdf65b545-pjqfb" Nov 03 13:20:37.468239 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:37.467117 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/736b02c5-9833-4f1b-9f9a-e4c22ab23ac9-istio-token\") pod \"e2e-extmod-a-ded926-openshift-default-6bdf65b545-pjqfb\" (UID: \"736b02c5-9833-4f1b-9f9a-e4c22ab23ac9\") " pod="openshift-ingress/e2e-extmod-a-ded926-openshift-default-6bdf65b545-pjqfb" Nov 03 13:20:37.468239 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:37.467197 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/736b02c5-9833-4f1b-9f9a-e4c22ab23ac9-istio-data\") pod \"e2e-extmod-a-ded926-openshift-default-6bdf65b545-pjqfb\" (UID: \"736b02c5-9833-4f1b-9f9a-e4c22ab23ac9\") " pod="openshift-ingress/e2e-extmod-a-ded926-openshift-default-6bdf65b545-pjqfb" Nov 03 13:20:37.468239 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:37.467273 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/736b02c5-9833-4f1b-9f9a-e4c22ab23ac9-workload-certs\") pod \"e2e-extmod-a-ded926-openshift-default-6bdf65b545-pjqfb\" (UID: \"736b02c5-9833-4f1b-9f9a-e4c22ab23ac9\") " pod="openshift-ingress/e2e-extmod-a-ded926-openshift-default-6bdf65b545-pjqfb" Nov 03 13:20:37.468239 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:37.467346 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-djf4m\" (UniqueName: \"kubernetes.io/projected/736b02c5-9833-4f1b-9f9a-e4c22ab23ac9-kube-api-access-djf4m\") pod \"e2e-extmod-a-ded926-openshift-default-6bdf65b545-pjqfb\" (UID: \"736b02c5-9833-4f1b-9f9a-e4c22ab23ac9\") " pod="openshift-ingress/e2e-extmod-a-ded926-openshift-default-6bdf65b545-pjqfb" Nov 03 13:20:37.468239 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:37.467397 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/736b02c5-9833-4f1b-9f9a-e4c22ab23ac9-credential-socket\") pod \"e2e-extmod-a-ded926-openshift-default-6bdf65b545-pjqfb\" (UID: \"736b02c5-9833-4f1b-9f9a-e4c22ab23ac9\") " pod="openshift-ingress/e2e-extmod-a-ded926-openshift-default-6bdf65b545-pjqfb" Nov 03 13:20:37.468239 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:37.467451 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/736b02c5-9833-4f1b-9f9a-e4c22ab23ac9-istio-envoy\") pod \"e2e-extmod-a-ded926-openshift-default-6bdf65b545-pjqfb\" (UID: \"736b02c5-9833-4f1b-9f9a-e4c22ab23ac9\") " pod="openshift-ingress/e2e-extmod-a-ded926-openshift-default-6bdf65b545-pjqfb" Nov 03 13:20:37.468239 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:37.467496 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/736b02c5-9833-4f1b-9f9a-e4c22ab23ac9-workload-socket\") pod \"e2e-extmod-a-ded926-openshift-default-6bdf65b545-pjqfb\" (UID: \"736b02c5-9833-4f1b-9f9a-e4c22ab23ac9\") " pod="openshift-ingress/e2e-extmod-a-ded926-openshift-default-6bdf65b545-pjqfb" Nov 03 13:20:37.589308 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:37.588259 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/736b02c5-9833-4f1b-9f9a-e4c22ab23ac9-istio-podinfo\") pod \"e2e-extmod-a-ded926-openshift-default-6bdf65b545-pjqfb\" (UID: \"736b02c5-9833-4f1b-9f9a-e4c22ab23ac9\") " pod="openshift-ingress/e2e-extmod-a-ded926-openshift-default-6bdf65b545-pjqfb" Nov 03 13:20:37.589308 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:37.588333 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/736b02c5-9833-4f1b-9f9a-e4c22ab23ac9-istio-token\") pod \"e2e-extmod-a-ded926-openshift-default-6bdf65b545-pjqfb\" (UID: \"736b02c5-9833-4f1b-9f9a-e4c22ab23ac9\") " pod="openshift-ingress/e2e-extmod-a-ded926-openshift-default-6bdf65b545-pjqfb" Nov 03 13:20:37.589308 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:37.588388 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/736b02c5-9833-4f1b-9f9a-e4c22ab23ac9-istio-data\") pod \"e2e-extmod-a-ded926-openshift-default-6bdf65b545-pjqfb\" (UID: \"736b02c5-9833-4f1b-9f9a-e4c22ab23ac9\") " pod="openshift-ingress/e2e-extmod-a-ded926-openshift-default-6bdf65b545-pjqfb" Nov 03 13:20:37.589308 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:37.588451 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/736b02c5-9833-4f1b-9f9a-e4c22ab23ac9-workload-certs\") pod \"e2e-extmod-a-ded926-openshift-default-6bdf65b545-pjqfb\" (UID: \"736b02c5-9833-4f1b-9f9a-e4c22ab23ac9\") " pod="openshift-ingress/e2e-extmod-a-ded926-openshift-default-6bdf65b545-pjqfb" Nov 03 13:20:37.589308 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:37.588498 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djf4m\" (UniqueName: \"kubernetes.io/projected/736b02c5-9833-4f1b-9f9a-e4c22ab23ac9-kube-api-access-djf4m\") pod \"e2e-extmod-a-ded926-openshift-default-6bdf65b545-pjqfb\" (UID: \"736b02c5-9833-4f1b-9f9a-e4c22ab23ac9\") " pod="openshift-ingress/e2e-extmod-a-ded926-openshift-default-6bdf65b545-pjqfb" Nov 03 13:20:37.589308 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:37.588542 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/736b02c5-9833-4f1b-9f9a-e4c22ab23ac9-credential-socket\") pod \"e2e-extmod-a-ded926-openshift-default-6bdf65b545-pjqfb\" (UID: \"736b02c5-9833-4f1b-9f9a-e4c22ab23ac9\") " pod="openshift-ingress/e2e-extmod-a-ded926-openshift-default-6bdf65b545-pjqfb" Nov 03 13:20:37.589308 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:37.588592 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/736b02c5-9833-4f1b-9f9a-e4c22ab23ac9-istio-envoy\") pod \"e2e-extmod-a-ded926-openshift-default-6bdf65b545-pjqfb\" (UID: \"736b02c5-9833-4f1b-9f9a-e4c22ab23ac9\") " pod="openshift-ingress/e2e-extmod-a-ded926-openshift-default-6bdf65b545-pjqfb" Nov 03 13:20:37.589308 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:37.588638 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/736b02c5-9833-4f1b-9f9a-e4c22ab23ac9-workload-socket\") pod \"e2e-extmod-a-ded926-openshift-default-6bdf65b545-pjqfb\" (UID: \"736b02c5-9833-4f1b-9f9a-e4c22ab23ac9\") " pod="openshift-ingress/e2e-extmod-a-ded926-openshift-default-6bdf65b545-pjqfb" Nov 03 13:20:37.589308 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:37.588760 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/736b02c5-9833-4f1b-9f9a-e4c22ab23ac9-istiod-ca-cert\") pod \"e2e-extmod-a-ded926-openshift-default-6bdf65b545-pjqfb\" (UID: \"736b02c5-9833-4f1b-9f9a-e4c22ab23ac9\") " pod="openshift-ingress/e2e-extmod-a-ded926-openshift-default-6bdf65b545-pjqfb" Nov 03 13:20:37.589925 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:37.589590 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/736b02c5-9833-4f1b-9f9a-e4c22ab23ac9-istiod-ca-cert\") pod \"e2e-extmod-a-ded926-openshift-default-6bdf65b545-pjqfb\" (UID: \"736b02c5-9833-4f1b-9f9a-e4c22ab23ac9\") " pod="openshift-ingress/e2e-extmod-a-ded926-openshift-default-6bdf65b545-pjqfb" Nov 03 13:20:37.594314 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:37.594109 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/736b02c5-9833-4f1b-9f9a-e4c22ab23ac9-istio-data\") pod \"e2e-extmod-a-ded926-openshift-default-6bdf65b545-pjqfb\" (UID: \"736b02c5-9833-4f1b-9f9a-e4c22ab23ac9\") " pod="openshift-ingress/e2e-extmod-a-ded926-openshift-default-6bdf65b545-pjqfb" Nov 03 13:20:37.594492 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:37.594377 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/736b02c5-9833-4f1b-9f9a-e4c22ab23ac9-workload-certs\") pod \"e2e-extmod-a-ded926-openshift-default-6bdf65b545-pjqfb\" (UID: \"736b02c5-9833-4f1b-9f9a-e4c22ab23ac9\") " pod="openshift-ingress/e2e-extmod-a-ded926-openshift-default-6bdf65b545-pjqfb" Nov 03 13:20:37.615319 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:37.600813 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/736b02c5-9833-4f1b-9f9a-e4c22ab23ac9-workload-socket\") pod \"e2e-extmod-a-ded926-openshift-default-6bdf65b545-pjqfb\" (UID: \"736b02c5-9833-4f1b-9f9a-e4c22ab23ac9\") " pod="openshift-ingress/e2e-extmod-a-ded926-openshift-default-6bdf65b545-pjqfb" Nov 03 13:20:37.615319 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:37.601083 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/736b02c5-9833-4f1b-9f9a-e4c22ab23ac9-credential-socket\") pod \"e2e-extmod-a-ded926-openshift-default-6bdf65b545-pjqfb\" (UID: \"736b02c5-9833-4f1b-9f9a-e4c22ab23ac9\") " pod="openshift-ingress/e2e-extmod-a-ded926-openshift-default-6bdf65b545-pjqfb" Nov 03 13:20:37.637306 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:37.631938 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/736b02c5-9833-4f1b-9f9a-e4c22ab23ac9-istio-podinfo\") pod \"e2e-extmod-a-ded926-openshift-default-6bdf65b545-pjqfb\" (UID: \"736b02c5-9833-4f1b-9f9a-e4c22ab23ac9\") " pod="openshift-ingress/e2e-extmod-a-ded926-openshift-default-6bdf65b545-pjqfb" Nov 03 13:20:37.651922 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:37.651889 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/736b02c5-9833-4f1b-9f9a-e4c22ab23ac9-istio-envoy\") pod \"e2e-extmod-a-ded926-openshift-default-6bdf65b545-pjqfb\" (UID: \"736b02c5-9833-4f1b-9f9a-e4c22ab23ac9\") " pod="openshift-ingress/e2e-extmod-a-ded926-openshift-default-6bdf65b545-pjqfb" Nov 03 13:20:37.663555 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:37.663523 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/736b02c5-9833-4f1b-9f9a-e4c22ab23ac9-istio-token\") pod \"e2e-extmod-a-ded926-openshift-default-6bdf65b545-pjqfb\" (UID: \"736b02c5-9833-4f1b-9f9a-e4c22ab23ac9\") " pod="openshift-ingress/e2e-extmod-a-ded926-openshift-default-6bdf65b545-pjqfb" Nov 03 13:20:37.664061 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:37.664033 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-djf4m\" (UniqueName: \"kubernetes.io/projected/736b02c5-9833-4f1b-9f9a-e4c22ab23ac9-kube-api-access-djf4m\") pod \"e2e-extmod-a-ded926-openshift-default-6bdf65b545-pjqfb\" (UID: \"736b02c5-9833-4f1b-9f9a-e4c22ab23ac9\") " pod="openshift-ingress/e2e-extmod-a-ded926-openshift-default-6bdf65b545-pjqfb" Nov 03 13:20:37.741843 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:37.741743 5024 patch_prober.go:29] interesting pod/e2e-worker-w4-c381a7-openshift-default-78bcd4695-fglkk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Nov 03 13:20:37.742443 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:37.742405 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-fglkk" podUID="e6d115b8-3b33-43c6-890b-a87bb608444b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Nov 03 13:20:37.836310 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:37.835639 5024 patch_prober.go:29] interesting pod/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-2jswl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Nov 03 13:20:37.836310 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:37.835702 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-2jswl" podUID="ed014864-f935-4abc-8cac-8f989d8b679b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Nov 03 13:20:37.902503 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:37.902464 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-ded926-openshift-default-6bdf65b545-pjqfb" Nov 03 13:20:38.752970 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:38.752932 5024 patch_prober.go:29] interesting pod/e2e-worker-w4-c381a7-openshift-default-78bcd4695-fglkk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Nov 03 13:20:38.753261 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:38.753220 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-fglkk" podUID="e6d115b8-3b33-43c6-890b-a87bb608444b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Nov 03 13:20:38.800857 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:38.800782 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-ded926-openshift-default-6bdf65b545-pjqfb"] Nov 03 13:20:38.814206 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:38.813028 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:20:38.814206 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:38.813111 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:20:38.814206 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:38.813148 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:20:38.823848 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:38.823821 5024 patch_prober.go:29] interesting pod/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-2jswl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Nov 03 13:20:38.824032 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:38.824008 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-2jswl" podUID="ed014864-f935-4abc-8cac-8f989d8b679b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Nov 03 13:20:39.519312 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:20:39.518460 5024 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d7mwz" containerName="extractor" Nov 03 13:20:39.536320 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:39.534526 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-ded926-openshift-default-6bdf65b545-pjqfb" event={"ID":"736b02c5-9833-4f1b-9f9a-e4c22ab23ac9","Type":"ContainerStarted","Data":"80f9d6d535afd0f850b398d01f7f1d9c71d9147c90bff43a2c95173a88c39ca9"} Nov 03 13:20:39.536320 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:39.534606 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-ded926-openshift-default-6bdf65b545-pjqfb" event={"ID":"736b02c5-9833-4f1b-9f9a-e4c22ab23ac9","Type":"ContainerStarted","Data":"c029b30dcf6dc8d768eb4fd79e6b3fbd528db5137c7c1f64a0c6b279a790ae8c"} Nov 03 13:20:39.560938 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:39.560869 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-ded926-openshift-default-6bdf65b545-pjqfb" podStartSLOduration=2.560849635 podStartE2EDuration="2.560849635s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:20:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:20:39.558353276 +0000 UTC m=+719.566544557" watchObservedRunningTime="2026-11-03 13:20:39.560849635 +0000 UTC m=+719.569040912" Nov 03 13:20:39.740929 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:39.740891 5024 patch_prober.go:29] interesting pod/e2e-worker-w4-c381a7-openshift-default-78bcd4695-fglkk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Nov 03 13:20:39.741129 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:39.741012 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-fglkk" podUID="e6d115b8-3b33-43c6-890b-a87bb608444b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Nov 03 13:20:39.855358 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:39.850756 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-2jswl" Nov 03 13:20:39.855358 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:39.852645 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-2jswl" Nov 03 13:20:39.855358 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:39.853582 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-2jswl" Nov 03 13:20:39.936313 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:39.930347 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-ded926-openshift-default-6bdf65b545-pjqfb" Nov 03 13:20:40.005611 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:40.005577 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-ded926-openshift-default-6bdf65b545-pjqfb" Nov 03 13:20:40.541926 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:40.541890 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-ded926-openshift-default-6bdf65b545-pjqfb" Nov 03 13:20:40.542725 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:40.542703 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-ded926-openshift-default-6bdf65b545-pjqfb" Nov 03 13:20:40.745273 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:40.745243 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-fglkk" Nov 03 13:20:40.745564 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:40.745539 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-fglkk" Nov 03 13:20:40.746219 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:40.746204 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-fglkk" Nov 03 13:20:46.471310 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:46.461993 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-7tc2j"] Nov 03 13:20:46.547307 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:46.546957 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-7tc2j"] Nov 03 13:20:46.547307 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:46.547096 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-7tc2j" Nov 03 13:20:46.677310 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:46.665659 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b61db32a-7a3b-4aff-b331-f7888a1a4e0e-istio-podinfo\") pod \"e2e-worker-w4-c381a7-openshift-default-78bcd4695-7tc2j\" (UID: \"b61db32a-7a3b-4aff-b331-f7888a1a4e0e\") " pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-7tc2j" Nov 03 13:20:46.677310 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:46.665802 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b61db32a-7a3b-4aff-b331-f7888a1a4e0e-istio-data\") pod \"e2e-worker-w4-c381a7-openshift-default-78bcd4695-7tc2j\" (UID: \"b61db32a-7a3b-4aff-b331-f7888a1a4e0e\") " pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-7tc2j" Nov 03 13:20:46.677310 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:46.665848 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b61db32a-7a3b-4aff-b331-f7888a1a4e0e-workload-socket\") pod \"e2e-worker-w4-c381a7-openshift-default-78bcd4695-7tc2j\" (UID: \"b61db32a-7a3b-4aff-b331-f7888a1a4e0e\") " pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-7tc2j" Nov 03 13:20:46.677310 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:46.665876 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qnjq6\" (UniqueName: \"kubernetes.io/projected/b61db32a-7a3b-4aff-b331-f7888a1a4e0e-kube-api-access-qnjq6\") pod \"e2e-worker-w4-c381a7-openshift-default-78bcd4695-7tc2j\" (UID: \"b61db32a-7a3b-4aff-b331-f7888a1a4e0e\") " pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-7tc2j" Nov 03 13:20:46.677310 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:46.665915 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b61db32a-7a3b-4aff-b331-f7888a1a4e0e-credential-socket\") pod \"e2e-worker-w4-c381a7-openshift-default-78bcd4695-7tc2j\" (UID: \"b61db32a-7a3b-4aff-b331-f7888a1a4e0e\") " pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-7tc2j" Nov 03 13:20:46.677310 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:46.665953 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b61db32a-7a3b-4aff-b331-f7888a1a4e0e-istio-token\") pod \"e2e-worker-w4-c381a7-openshift-default-78bcd4695-7tc2j\" (UID: \"b61db32a-7a3b-4aff-b331-f7888a1a4e0e\") " pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-7tc2j" Nov 03 13:20:46.677310 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:46.665996 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b61db32a-7a3b-4aff-b331-f7888a1a4e0e-istiod-ca-cert\") pod \"e2e-worker-w4-c381a7-openshift-default-78bcd4695-7tc2j\" (UID: \"b61db32a-7a3b-4aff-b331-f7888a1a4e0e\") " pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-7tc2j" Nov 03 13:20:46.677310 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:46.666072 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b61db32a-7a3b-4aff-b331-f7888a1a4e0e-workload-certs\") pod \"e2e-worker-w4-c381a7-openshift-default-78bcd4695-7tc2j\" (UID: \"b61db32a-7a3b-4aff-b331-f7888a1a4e0e\") " pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-7tc2j" Nov 03 13:20:46.677310 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:46.666114 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b61db32a-7a3b-4aff-b331-f7888a1a4e0e-istio-envoy\") pod \"e2e-worker-w4-c381a7-openshift-default-78bcd4695-7tc2j\" (UID: \"b61db32a-7a3b-4aff-b331-f7888a1a4e0e\") " pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-7tc2j" Nov 03 13:20:46.732348 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:46.719931 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-kjbs5"] Nov 03 13:20:46.774830 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:46.774780 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b61db32a-7a3b-4aff-b331-f7888a1a4e0e-istio-data\") pod \"e2e-worker-w4-c381a7-openshift-default-78bcd4695-7tc2j\" (UID: \"b61db32a-7a3b-4aff-b331-f7888a1a4e0e\") " pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-7tc2j" Nov 03 13:20:46.775110 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:46.775081 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b61db32a-7a3b-4aff-b331-f7888a1a4e0e-workload-socket\") pod \"e2e-worker-w4-c381a7-openshift-default-78bcd4695-7tc2j\" (UID: \"b61db32a-7a3b-4aff-b331-f7888a1a4e0e\") " pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-7tc2j" Nov 03 13:20:46.775256 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:46.775237 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qnjq6\" (UniqueName: \"kubernetes.io/projected/b61db32a-7a3b-4aff-b331-f7888a1a4e0e-kube-api-access-qnjq6\") pod \"e2e-worker-w4-c381a7-openshift-default-78bcd4695-7tc2j\" (UID: \"b61db32a-7a3b-4aff-b331-f7888a1a4e0e\") " pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-7tc2j" Nov 03 13:20:46.775425 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:46.775406 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b61db32a-7a3b-4aff-b331-f7888a1a4e0e-credential-socket\") pod \"e2e-worker-w4-c381a7-openshift-default-78bcd4695-7tc2j\" (UID: \"b61db32a-7a3b-4aff-b331-f7888a1a4e0e\") " pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-7tc2j" Nov 03 13:20:46.775579 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:46.775561 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b61db32a-7a3b-4aff-b331-f7888a1a4e0e-istio-token\") pod \"e2e-worker-w4-c381a7-openshift-default-78bcd4695-7tc2j\" (UID: \"b61db32a-7a3b-4aff-b331-f7888a1a4e0e\") " pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-7tc2j" Nov 03 13:20:46.775788 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:46.775768 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b61db32a-7a3b-4aff-b331-f7888a1a4e0e-istiod-ca-cert\") pod \"e2e-worker-w4-c381a7-openshift-default-78bcd4695-7tc2j\" (UID: \"b61db32a-7a3b-4aff-b331-f7888a1a4e0e\") " pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-7tc2j" Nov 03 13:20:46.775981 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:46.775964 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b61db32a-7a3b-4aff-b331-f7888a1a4e0e-workload-certs\") pod \"e2e-worker-w4-c381a7-openshift-default-78bcd4695-7tc2j\" (UID: \"b61db32a-7a3b-4aff-b331-f7888a1a4e0e\") " pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-7tc2j" Nov 03 13:20:46.776114 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:46.776099 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b61db32a-7a3b-4aff-b331-f7888a1a4e0e-istio-envoy\") pod \"e2e-worker-w4-c381a7-openshift-default-78bcd4695-7tc2j\" (UID: \"b61db32a-7a3b-4aff-b331-f7888a1a4e0e\") " pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-7tc2j" Nov 03 13:20:46.776258 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:46.776242 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b61db32a-7a3b-4aff-b331-f7888a1a4e0e-istio-podinfo\") pod \"e2e-worker-w4-c381a7-openshift-default-78bcd4695-7tc2j\" (UID: \"b61db32a-7a3b-4aff-b331-f7888a1a4e0e\") " pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-7tc2j" Nov 03 13:20:46.777536 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:46.777508 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b61db32a-7a3b-4aff-b331-f7888a1a4e0e-credential-socket\") pod \"e2e-worker-w4-c381a7-openshift-default-78bcd4695-7tc2j\" (UID: \"b61db32a-7a3b-4aff-b331-f7888a1a4e0e\") " pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-7tc2j" Nov 03 13:20:46.780038 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:46.778311 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b61db32a-7a3b-4aff-b331-f7888a1a4e0e-istiod-ca-cert\") pod \"e2e-worker-w4-c381a7-openshift-default-78bcd4695-7tc2j\" (UID: \"b61db32a-7a3b-4aff-b331-f7888a1a4e0e\") " pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-7tc2j" Nov 03 13:20:46.780204 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:46.778885 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b61db32a-7a3b-4aff-b331-f7888a1a4e0e-workload-socket\") pod \"e2e-worker-w4-c381a7-openshift-default-78bcd4695-7tc2j\" (UID: \"b61db32a-7a3b-4aff-b331-f7888a1a4e0e\") " pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-7tc2j" Nov 03 13:20:46.780344 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:46.779143 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b61db32a-7a3b-4aff-b331-f7888a1a4e0e-istio-data\") pod \"e2e-worker-w4-c381a7-openshift-default-78bcd4695-7tc2j\" (UID: \"b61db32a-7a3b-4aff-b331-f7888a1a4e0e\") " pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-7tc2j" Nov 03 13:20:46.780461 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:46.779398 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b61db32a-7a3b-4aff-b331-f7888a1a4e0e-workload-certs\") pod \"e2e-worker-w4-c381a7-openshift-default-78bcd4695-7tc2j\" (UID: \"b61db32a-7a3b-4aff-b331-f7888a1a4e0e\") " pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-7tc2j" Nov 03 13:20:46.941311 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:46.938590 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-kjbs5"] Nov 03 13:20:46.941311 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:46.938750 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-kjbs5" Nov 03 13:20:46.953921 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:46.953887 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qnjq6\" (UniqueName: \"kubernetes.io/projected/b61db32a-7a3b-4aff-b331-f7888a1a4e0e-kube-api-access-qnjq6\") pod \"e2e-worker-w4-c381a7-openshift-default-78bcd4695-7tc2j\" (UID: \"b61db32a-7a3b-4aff-b331-f7888a1a4e0e\") " pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-7tc2j" Nov 03 13:20:46.980467 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:46.960963 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b61db32a-7a3b-4aff-b331-f7888a1a4e0e-istio-envoy\") pod \"e2e-worker-w4-c381a7-openshift-default-78bcd4695-7tc2j\" (UID: \"b61db32a-7a3b-4aff-b331-f7888a1a4e0e\") " pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-7tc2j" Nov 03 13:20:46.980760 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:46.961327 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b61db32a-7a3b-4aff-b331-f7888a1a4e0e-istio-token\") pod \"e2e-worker-w4-c381a7-openshift-default-78bcd4695-7tc2j\" (UID: \"b61db32a-7a3b-4aff-b331-f7888a1a4e0e\") " pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-7tc2j" Nov 03 13:20:47.008913 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:47.008765 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b61db32a-7a3b-4aff-b331-f7888a1a4e0e-istio-podinfo\") pod \"e2e-worker-w4-c381a7-openshift-default-78bcd4695-7tc2j\" (UID: \"b61db32a-7a3b-4aff-b331-f7888a1a4e0e\") " pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-7tc2j" Nov 03 13:20:47.066309 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:47.059299 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-7tc2j" Nov 03 13:20:47.175309 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:47.166250 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4eeaf0ad-8a27-44c2-8641-88d4d64e7969-istiod-ca-cert\") pod \"e2e-worker-w4-c381a7-openshift-default-78bcd4695-kjbs5\" (UID: \"4eeaf0ad-8a27-44c2-8641-88d4d64e7969\") " pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-kjbs5" Nov 03 13:20:47.219429 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:47.219361 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hbh6f\" (UniqueName: \"kubernetes.io/projected/4eeaf0ad-8a27-44c2-8641-88d4d64e7969-kube-api-access-hbh6f\") pod \"e2e-worker-w4-c381a7-openshift-default-78bcd4695-kjbs5\" (UID: \"4eeaf0ad-8a27-44c2-8641-88d4d64e7969\") " pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-kjbs5" Nov 03 13:20:47.221407 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:47.221359 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4eeaf0ad-8a27-44c2-8641-88d4d64e7969-istio-token\") pod \"e2e-worker-w4-c381a7-openshift-default-78bcd4695-kjbs5\" (UID: \"4eeaf0ad-8a27-44c2-8641-88d4d64e7969\") " pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-kjbs5" Nov 03 13:20:47.223928 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:47.221610 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4eeaf0ad-8a27-44c2-8641-88d4d64e7969-istio-envoy\") pod \"e2e-worker-w4-c381a7-openshift-default-78bcd4695-kjbs5\" (UID: \"4eeaf0ad-8a27-44c2-8641-88d4d64e7969\") " pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-kjbs5" Nov 03 13:20:47.224135 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:47.224115 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4eeaf0ad-8a27-44c2-8641-88d4d64e7969-workload-certs\") pod \"e2e-worker-w4-c381a7-openshift-default-78bcd4695-kjbs5\" (UID: \"4eeaf0ad-8a27-44c2-8641-88d4d64e7969\") " pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-kjbs5" Nov 03 13:20:47.228916 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:47.228891 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4eeaf0ad-8a27-44c2-8641-88d4d64e7969-istio-podinfo\") pod \"e2e-worker-w4-c381a7-openshift-default-78bcd4695-kjbs5\" (UID: \"4eeaf0ad-8a27-44c2-8641-88d4d64e7969\") " pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-kjbs5" Nov 03 13:20:47.229485 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:47.229460 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4eeaf0ad-8a27-44c2-8641-88d4d64e7969-istio-data\") pod \"e2e-worker-w4-c381a7-openshift-default-78bcd4695-kjbs5\" (UID: \"4eeaf0ad-8a27-44c2-8641-88d4d64e7969\") " pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-kjbs5" Nov 03 13:20:47.229691 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:47.229675 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4eeaf0ad-8a27-44c2-8641-88d4d64e7969-credential-socket\") pod \"e2e-worker-w4-c381a7-openshift-default-78bcd4695-kjbs5\" (UID: \"4eeaf0ad-8a27-44c2-8641-88d4d64e7969\") " pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-kjbs5" Nov 03 13:20:47.229908 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:47.229890 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4eeaf0ad-8a27-44c2-8641-88d4d64e7969-workload-socket\") pod \"e2e-worker-w4-c381a7-openshift-default-78bcd4695-kjbs5\" (UID: \"4eeaf0ad-8a27-44c2-8641-88d4d64e7969\") " pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-kjbs5" Nov 03 13:20:47.361462 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:47.356003 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4eeaf0ad-8a27-44c2-8641-88d4d64e7969-istio-podinfo\") pod \"e2e-worker-w4-c381a7-openshift-default-78bcd4695-kjbs5\" (UID: \"4eeaf0ad-8a27-44c2-8641-88d4d64e7969\") " pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-kjbs5" Nov 03 13:20:47.361462 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:47.356063 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4eeaf0ad-8a27-44c2-8641-88d4d64e7969-istio-data\") pod \"e2e-worker-w4-c381a7-openshift-default-78bcd4695-kjbs5\" (UID: \"4eeaf0ad-8a27-44c2-8641-88d4d64e7969\") " pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-kjbs5" Nov 03 13:20:47.361462 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:47.356102 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4eeaf0ad-8a27-44c2-8641-88d4d64e7969-credential-socket\") pod \"e2e-worker-w4-c381a7-openshift-default-78bcd4695-kjbs5\" (UID: \"4eeaf0ad-8a27-44c2-8641-88d4d64e7969\") " pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-kjbs5" Nov 03 13:20:47.361462 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:47.356194 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4eeaf0ad-8a27-44c2-8641-88d4d64e7969-workload-socket\") pod \"e2e-worker-w4-c381a7-openshift-default-78bcd4695-kjbs5\" (UID: \"4eeaf0ad-8a27-44c2-8641-88d4d64e7969\") " pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-kjbs5" Nov 03 13:20:47.361462 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:47.356254 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4eeaf0ad-8a27-44c2-8641-88d4d64e7969-istiod-ca-cert\") pod \"e2e-worker-w4-c381a7-openshift-default-78bcd4695-kjbs5\" (UID: \"4eeaf0ad-8a27-44c2-8641-88d4d64e7969\") " pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-kjbs5" Nov 03 13:20:47.361462 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:47.356304 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbh6f\" (UniqueName: \"kubernetes.io/projected/4eeaf0ad-8a27-44c2-8641-88d4d64e7969-kube-api-access-hbh6f\") pod \"e2e-worker-w4-c381a7-openshift-default-78bcd4695-kjbs5\" (UID: \"4eeaf0ad-8a27-44c2-8641-88d4d64e7969\") " pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-kjbs5" Nov 03 13:20:47.361462 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:47.356436 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4eeaf0ad-8a27-44c2-8641-88d4d64e7969-istio-token\") pod \"e2e-worker-w4-c381a7-openshift-default-78bcd4695-kjbs5\" (UID: \"4eeaf0ad-8a27-44c2-8641-88d4d64e7969\") " pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-kjbs5" Nov 03 13:20:47.361462 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:47.356477 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4eeaf0ad-8a27-44c2-8641-88d4d64e7969-istio-envoy\") pod \"e2e-worker-w4-c381a7-openshift-default-78bcd4695-kjbs5\" (UID: \"4eeaf0ad-8a27-44c2-8641-88d4d64e7969\") " pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-kjbs5" Nov 03 13:20:47.361462 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:47.356505 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4eeaf0ad-8a27-44c2-8641-88d4d64e7969-workload-certs\") pod \"e2e-worker-w4-c381a7-openshift-default-78bcd4695-kjbs5\" (UID: \"4eeaf0ad-8a27-44c2-8641-88d4d64e7969\") " pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-kjbs5" Nov 03 13:20:47.361462 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:47.357018 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4eeaf0ad-8a27-44c2-8641-88d4d64e7969-workload-certs\") pod \"e2e-worker-w4-c381a7-openshift-default-78bcd4695-kjbs5\" (UID: \"4eeaf0ad-8a27-44c2-8641-88d4d64e7969\") " pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-kjbs5" Nov 03 13:20:47.361462 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:47.357775 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4eeaf0ad-8a27-44c2-8641-88d4d64e7969-credential-socket\") pod \"e2e-worker-w4-c381a7-openshift-default-78bcd4695-kjbs5\" (UID: \"4eeaf0ad-8a27-44c2-8641-88d4d64e7969\") " pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-kjbs5" Nov 03 13:20:47.361462 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:47.358006 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4eeaf0ad-8a27-44c2-8641-88d4d64e7969-istio-data\") pod \"e2e-worker-w4-c381a7-openshift-default-78bcd4695-kjbs5\" (UID: \"4eeaf0ad-8a27-44c2-8641-88d4d64e7969\") " pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-kjbs5" Nov 03 13:20:47.361462 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:47.358220 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4eeaf0ad-8a27-44c2-8641-88d4d64e7969-workload-socket\") pod \"e2e-worker-w4-c381a7-openshift-default-78bcd4695-kjbs5\" (UID: \"4eeaf0ad-8a27-44c2-8641-88d4d64e7969\") " pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-kjbs5" Nov 03 13:20:47.380976 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:47.380891 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4eeaf0ad-8a27-44c2-8641-88d4d64e7969-istio-podinfo\") pod \"e2e-worker-w4-c381a7-openshift-default-78bcd4695-kjbs5\" (UID: \"4eeaf0ad-8a27-44c2-8641-88d4d64e7969\") " pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-kjbs5" Nov 03 13:20:47.381359 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:47.381328 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4eeaf0ad-8a27-44c2-8641-88d4d64e7969-istiod-ca-cert\") pod \"e2e-worker-w4-c381a7-openshift-default-78bcd4695-kjbs5\" (UID: \"4eeaf0ad-8a27-44c2-8641-88d4d64e7969\") " pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-kjbs5" Nov 03 13:20:47.402316 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:47.401703 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4eeaf0ad-8a27-44c2-8641-88d4d64e7969-istio-envoy\") pod \"e2e-worker-w4-c381a7-openshift-default-78bcd4695-kjbs5\" (UID: \"4eeaf0ad-8a27-44c2-8641-88d4d64e7969\") " pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-kjbs5" Nov 03 13:20:47.409420 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:47.408792 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4eeaf0ad-8a27-44c2-8641-88d4d64e7969-istio-token\") pod \"e2e-worker-w4-c381a7-openshift-default-78bcd4695-kjbs5\" (UID: \"4eeaf0ad-8a27-44c2-8641-88d4d64e7969\") " pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-kjbs5" Nov 03 13:20:47.409420 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:47.409375 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hbh6f\" (UniqueName: \"kubernetes.io/projected/4eeaf0ad-8a27-44c2-8641-88d4d64e7969-kube-api-access-hbh6f\") pod \"e2e-worker-w4-c381a7-openshift-default-78bcd4695-kjbs5\" (UID: \"4eeaf0ad-8a27-44c2-8641-88d4d64e7969\") " pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-kjbs5" Nov 03 13:20:47.471155 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:47.470523 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-kjbs5" Nov 03 13:20:47.931475 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:47.931444 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-7tc2j"] Nov 03 13:20:47.932572 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:20:47.932530 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb61db32a_7a3b_4aff_b331_f7888a1a4e0e.slice/crio-784edf9c87f371fbe139fb67db023e3de59bf7bb65ff5a8ce8e04aa7fe62122b WatchSource:0}: Error finding container 784edf9c87f371fbe139fb67db023e3de59bf7bb65ff5a8ce8e04aa7fe62122b: Status 404 returned error can't find the container with id 784edf9c87f371fbe139fb67db023e3de59bf7bb65ff5a8ce8e04aa7fe62122b Nov 03 13:20:47.935261 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:47.935223 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:20:47.936681 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:47.936650 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:20:47.936863 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:47.936842 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:20:48.077224 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:20:48.077184 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4eeaf0ad_8a27_44c2_8641_88d4d64e7969.slice/crio-3a6652419812eb4faea52e9e5742073ecbd9197cac6473f2bd01b26496379afe WatchSource:0}: Error finding container 3a6652419812eb4faea52e9e5742073ecbd9197cac6473f2bd01b26496379afe: Status 404 returned error can't find the container with id 3a6652419812eb4faea52e9e5742073ecbd9197cac6473f2bd01b26496379afe Nov 03 13:20:48.077404 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:48.077301 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-kjbs5"] Nov 03 13:20:48.080078 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:48.080038 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:20:48.080191 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:48.080111 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:20:48.080191 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:48.080143 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:20:48.745601 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:48.745559 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-7tc2j" event={"ID":"b61db32a-7a3b-4aff-b331-f7888a1a4e0e","Type":"ContainerStarted","Data":"c18e004cc03ecc2eb605c781789855e30f789cdb09d599e4ef0ce83f11b920a6"} Nov 03 13:20:48.745826 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:48.745618 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-7tc2j" event={"ID":"b61db32a-7a3b-4aff-b331-f7888a1a4e0e","Type":"ContainerStarted","Data":"784edf9c87f371fbe139fb67db023e3de59bf7bb65ff5a8ce8e04aa7fe62122b"} Nov 03 13:20:48.747402 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:48.747369 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-kjbs5" event={"ID":"4eeaf0ad-8a27-44c2-8641-88d4d64e7969","Type":"ContainerStarted","Data":"fbdcddf96d6c74e3a56a93df3868b856b4423d9bf7ad5fa43d10d6b5cd79be48"} Nov 03 13:20:48.747538 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:48.747413 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-kjbs5" event={"ID":"4eeaf0ad-8a27-44c2-8641-88d4d64e7969","Type":"ContainerStarted","Data":"3a6652419812eb4faea52e9e5742073ecbd9197cac6473f2bd01b26496379afe"} Nov 03 13:20:48.809309 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:48.808713 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-7tc2j" podStartSLOduration=2.8086896169999997 podStartE2EDuration="2.808689617s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:20:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:20:48.768831127 +0000 UTC m=+728.777022406" watchObservedRunningTime="2026-11-03 13:20:48.808689617 +0000 UTC m=+728.816880899" Nov 03 13:20:49.062841 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:49.062739 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-7tc2j" Nov 03 13:20:49.064236 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:49.064205 5024 patch_prober.go:29] interesting pod/e2e-worker-w4-c381a7-openshift-default-78bcd4695-7tc2j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body= Nov 03 13:20:49.064381 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:49.064264 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-7tc2j" podUID="b61db32a-7a3b-4aff-b331-f7888a1a4e0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" Nov 03 13:20:49.481317 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:49.478680 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-kjbs5" Nov 03 13:20:49.494327 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:49.491859 5024 patch_prober.go:29] interesting pod/e2e-worker-w4-c381a7-openshift-default-78bcd4695-kjbs5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body= Nov 03 13:20:49.494327 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:49.491924 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-kjbs5" podUID="4eeaf0ad-8a27-44c2-8641-88d4d64e7969" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" Nov 03 13:20:50.062676 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:50.062627 5024 patch_prober.go:29] interesting pod/e2e-worker-w4-c381a7-openshift-default-78bcd4695-7tc2j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body= Nov 03 13:20:50.062890 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:50.062697 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-7tc2j" podUID="b61db32a-7a3b-4aff-b331-f7888a1a4e0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" Nov 03 13:20:50.508748 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:50.508713 5024 patch_prober.go:29] interesting pod/e2e-worker-w4-c381a7-openshift-default-78bcd4695-kjbs5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body= Nov 03 13:20:50.509035 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:50.509001 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-kjbs5" podUID="4eeaf0ad-8a27-44c2-8641-88d4d64e7969" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" Nov 03 13:20:51.062392 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:51.062348 5024 patch_prober.go:29] interesting pod/e2e-worker-w4-c381a7-openshift-default-78bcd4695-7tc2j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body= Nov 03 13:20:51.062582 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:51.062412 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-7tc2j" podUID="b61db32a-7a3b-4aff-b331-f7888a1a4e0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" Nov 03 13:20:51.142330 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:51.142138 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-kjbs5" podStartSLOduration=5.142105715 podStartE2EDuration="5.142105715s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:20:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:20:48.830690702 +0000 UTC m=+728.838881980" watchObservedRunningTime="2026-11-03 13:20:51.142105715 +0000 UTC m=+731.150296993" Nov 03 13:20:51.143625 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:51.143594 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-ded926-5f5bffd8fd-kn5lm"] Nov 03 13:20:51.159783 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:51.159706 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ded926-5f5bffd8fd-kn5lm" Nov 03 13:20:51.162351 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:51.162323 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-ded926-5f5bffd8fd-kn5lm"] Nov 03 13:20:51.166812 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:51.166781 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-ded926-serving-cert\"" Nov 03 13:20:51.263382 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:51.263337 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e1668ec-1b0f-4600-8935-64733905b0c6-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-ded926-5f5bffd8fd-kn5lm\" (UID: \"5e1668ec-1b0f-4600-8935-64733905b0c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ded926-5f5bffd8fd-kn5lm" Nov 03 13:20:51.263545 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:51.263467 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5e1668ec-1b0f-4600-8935-64733905b0c6-maas-api-tls\") pod \"maas-api-e2e-extmod-a-ded926-5f5bffd8fd-kn5lm\" (UID: \"5e1668ec-1b0f-4600-8935-64733905b0c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ded926-5f5bffd8fd-kn5lm" Nov 03 13:20:51.263545 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:51.263536 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5e1668ec-1b0f-4600-8935-64733905b0c6-metrics-certs\") pod \"maas-api-e2e-extmod-a-ded926-5f5bffd8fd-kn5lm\" (UID: \"5e1668ec-1b0f-4600-8935-64733905b0c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ded926-5f5bffd8fd-kn5lm" Nov 03 13:20:51.263713 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:51.263695 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rmcqt\" (UniqueName: \"kubernetes.io/projected/5e1668ec-1b0f-4600-8935-64733905b0c6-kube-api-access-rmcqt\") pod \"maas-api-e2e-extmod-a-ded926-5f5bffd8fd-kn5lm\" (UID: \"5e1668ec-1b0f-4600-8935-64733905b0c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ded926-5f5bffd8fd-kn5lm" Nov 03 13:20:51.365300 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:51.365257 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e1668ec-1b0f-4600-8935-64733905b0c6-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-ded926-5f5bffd8fd-kn5lm\" (UID: \"5e1668ec-1b0f-4600-8935-64733905b0c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ded926-5f5bffd8fd-kn5lm" Nov 03 13:20:51.365462 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:51.365362 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5e1668ec-1b0f-4600-8935-64733905b0c6-maas-api-tls\") pod \"maas-api-e2e-extmod-a-ded926-5f5bffd8fd-kn5lm\" (UID: \"5e1668ec-1b0f-4600-8935-64733905b0c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ded926-5f5bffd8fd-kn5lm" Nov 03 13:20:51.365462 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:51.365409 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5e1668ec-1b0f-4600-8935-64733905b0c6-metrics-certs\") pod \"maas-api-e2e-extmod-a-ded926-5f5bffd8fd-kn5lm\" (UID: \"5e1668ec-1b0f-4600-8935-64733905b0c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ded926-5f5bffd8fd-kn5lm" Nov 03 13:20:51.365556 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:51.365476 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rmcqt\" (UniqueName: \"kubernetes.io/projected/5e1668ec-1b0f-4600-8935-64733905b0c6-kube-api-access-rmcqt\") pod \"maas-api-e2e-extmod-a-ded926-5f5bffd8fd-kn5lm\" (UID: \"5e1668ec-1b0f-4600-8935-64733905b0c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ded926-5f5bffd8fd-kn5lm" Nov 03 13:20:51.366362 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:51.366340 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e1668ec-1b0f-4600-8935-64733905b0c6-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-ded926-5f5bffd8fd-kn5lm\" (UID: \"5e1668ec-1b0f-4600-8935-64733905b0c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ded926-5f5bffd8fd-kn5lm" Nov 03 13:20:51.368982 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:51.368952 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5e1668ec-1b0f-4600-8935-64733905b0c6-maas-api-tls\") pod \"maas-api-e2e-extmod-a-ded926-5f5bffd8fd-kn5lm\" (UID: \"5e1668ec-1b0f-4600-8935-64733905b0c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ded926-5f5bffd8fd-kn5lm" Nov 03 13:20:51.398513 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:51.375591 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rmcqt\" (UniqueName: \"kubernetes.io/projected/5e1668ec-1b0f-4600-8935-64733905b0c6-kube-api-access-rmcqt\") pod \"maas-api-e2e-extmod-a-ded926-5f5bffd8fd-kn5lm\" (UID: \"5e1668ec-1b0f-4600-8935-64733905b0c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ded926-5f5bffd8fd-kn5lm" Nov 03 13:20:51.398513 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:51.375710 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5e1668ec-1b0f-4600-8935-64733905b0c6-metrics-certs\") pod \"maas-api-e2e-extmod-a-ded926-5f5bffd8fd-kn5lm\" (UID: \"5e1668ec-1b0f-4600-8935-64733905b0c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ded926-5f5bffd8fd-kn5lm" Nov 03 13:20:51.481785 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:51.481740 5024 patch_prober.go:29] interesting pod/e2e-worker-w4-c381a7-openshift-default-78bcd4695-kjbs5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body= Nov 03 13:20:51.482074 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:51.482039 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-kjbs5" podUID="4eeaf0ad-8a27-44c2-8641-88d4d64e7969" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" Nov 03 13:20:51.557600 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:51.557557 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ded926-5f5bffd8fd-kn5lm" Nov 03 13:20:52.039269 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:52.039233 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-ded926-5f5bffd8fd-kn5lm"] Nov 03 13:20:52.042712 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:20:52.042675 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e1668ec_1b0f_4600_8935_64733905b0c6.slice/crio-1eb9a4ee7648e7b2fae5f47bd1dc6f831b42aeba9929d3a8703371c2459c5a08 WatchSource:0}: Error finding container 1eb9a4ee7648e7b2fae5f47bd1dc6f831b42aeba9929d3a8703371c2459c5a08: Status 404 returned error can't find the container with id 1eb9a4ee7648e7b2fae5f47bd1dc6f831b42aeba9929d3a8703371c2459c5a08 Nov 03 13:20:52.044917 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:52.044329 5024 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 13:20:52.062699 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:52.062668 5024 patch_prober.go:29] interesting pod/e2e-worker-w4-c381a7-openshift-default-78bcd4695-7tc2j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body= Nov 03 13:20:52.062853 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:52.062717 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-7tc2j" podUID="b61db32a-7a3b-4aff-b331-f7888a1a4e0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" Nov 03 13:20:52.480770 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:52.480729 5024 patch_prober.go:29] interesting pod/e2e-worker-w4-c381a7-openshift-default-78bcd4695-kjbs5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body= Nov 03 13:20:52.481361 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:52.481326 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-kjbs5" podUID="4eeaf0ad-8a27-44c2-8641-88d4d64e7969" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" Nov 03 13:20:52.845258 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:52.845215 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ded926-5f5bffd8fd-kn5lm" event={"ID":"5e1668ec-1b0f-4600-8935-64733905b0c6","Type":"ContainerStarted","Data":"1eb9a4ee7648e7b2fae5f47bd1dc6f831b42aeba9929d3a8703371c2459c5a08"} Nov 03 13:20:53.062545 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:53.062449 5024 patch_prober.go:29] interesting pod/e2e-worker-w4-c381a7-openshift-default-78bcd4695-7tc2j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body= Nov 03 13:20:53.062545 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:53.062510 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-7tc2j" podUID="b61db32a-7a3b-4aff-b331-f7888a1a4e0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" Nov 03 13:20:53.477311 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:53.474829 5024 patch_prober.go:29] interesting pod/e2e-worker-w4-c381a7-openshift-default-78bcd4695-kjbs5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body= Nov 03 13:20:53.477311 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:53.474908 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-kjbs5" podUID="4eeaf0ad-8a27-44c2-8641-88d4d64e7969" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" Nov 03 13:20:53.893041 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:53.891375 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ded926-5f5bffd8fd-kn5lm" event={"ID":"5e1668ec-1b0f-4600-8935-64733905b0c6","Type":"ContainerStarted","Data":"ad0fc84950464f28185e109ed9e9ced1c4c95401ec78ed523a1c04b4ca92163c"} Nov 03 13:20:53.893041 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:53.892441 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ded926-5f5bffd8fd-kn5lm" Nov 03 13:20:53.960347 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:53.960259 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ded926-5f5bffd8fd-kn5lm" podStartSLOduration=2.235290138 podStartE2EDuration="2.960229722s" totalImagesPullingTime="724.939584ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:20:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:20:53.91674814 +0000 UTC m=+733.924939425" watchObservedRunningTime="2026-11-03 13:20:53.960229722 +0000 UTC m=+733.968421000" Nov 03 13:20:54.091775 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:54.091740 5024 patch_prober.go:29] interesting pod/e2e-worker-w4-c381a7-openshift-default-78bcd4695-7tc2j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body= Nov 03 13:20:54.092036 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:54.092009 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-7tc2j" podUID="b61db32a-7a3b-4aff-b331-f7888a1a4e0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" Nov 03 13:20:54.472418 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:54.472367 5024 patch_prober.go:29] interesting pod/e2e-worker-w4-c381a7-openshift-default-78bcd4695-kjbs5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body= Nov 03 13:20:54.472700 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:54.472660 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-kjbs5" podUID="4eeaf0ad-8a27-44c2-8641-88d4d64e7969" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" Nov 03 13:20:55.062143 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:55.062102 5024 patch_prober.go:29] interesting pod/e2e-worker-w4-c381a7-openshift-default-78bcd4695-7tc2j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body= Nov 03 13:20:55.062741 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:55.062173 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-7tc2j" podUID="b61db32a-7a3b-4aff-b331-f7888a1a4e0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" Nov 03 13:20:56.479701 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:56.479659 5024 patch_prober.go:29] interesting pod/e2e-worker-w4-c381a7-openshift-default-78bcd4695-kjbs5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 13:20:56.482337 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:56.480301 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-kjbs5" podUID="4eeaf0ad-8a27-44c2-8641-88d4d64e7969" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 13:20:56.535320 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:56.520795 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-kjbs5" Nov 03 13:20:56.535320 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:56.522318 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-kjbs5" Nov 03 13:20:56.535320 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:56.528776 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-kjbs5" Nov 03 13:20:57.068433 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:57.068386 5024 patch_prober.go:29] interesting pod/e2e-worker-w4-c381a7-openshift-default-78bcd4695-7tc2j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 13:20:57.068734 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:57.068703 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-7tc2j" podUID="b61db32a-7a3b-4aff-b331-f7888a1a4e0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 13:20:58.136838 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:58.136782 5024 patch_prober.go:29] interesting pod/e2e-worker-w4-c381a7-openshift-default-78bcd4695-7tc2j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 13:20:58.137544 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:58.137492 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-7tc2j" podUID="b61db32a-7a3b-4aff-b331-f7888a1a4e0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 13:20:58.501024 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:58.500942 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-7tc2j" Nov 03 13:20:58.501585 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:58.501555 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-7tc2j" Nov 03 13:20:58.502679 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:20:58.502664 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-7tc2j" Nov 03 13:21:00.943072 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:00.943037 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ded926-5f5bffd8fd-kn5lm" Nov 03 13:21:01.414996 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:01.414961 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-d66hs"] Nov 03 13:21:01.424007 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:01.423980 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-d66hs" Nov 03 13:21:01.431330 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:01.431303 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-d66hs"] Nov 03 13:21:01.489778 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:01.489731 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14bfce12-7ae7-4866-aa46-bce174d28ca6-istiod-ca-cert\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-d66hs\" (UID: \"14bfce12-7ae7-4866-aa46-bce174d28ca6\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-d66hs" Nov 03 13:21:01.489950 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:01.489799 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gv797\" (UniqueName: \"kubernetes.io/projected/14bfce12-7ae7-4866-aa46-bce174d28ca6-kube-api-access-gv797\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-d66hs\" (UID: \"14bfce12-7ae7-4866-aa46-bce174d28ca6\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-d66hs" Nov 03 13:21:01.489950 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:01.489881 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/14bfce12-7ae7-4866-aa46-bce174d28ca6-istio-token\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-d66hs\" (UID: \"14bfce12-7ae7-4866-aa46-bce174d28ca6\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-d66hs" Nov 03 13:21:01.489950 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:01.489918 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14bfce12-7ae7-4866-aa46-bce174d28ca6-istio-data\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-d66hs\" (UID: \"14bfce12-7ae7-4866-aa46-bce174d28ca6\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-d66hs" Nov 03 13:21:01.490093 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:01.490074 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14bfce12-7ae7-4866-aa46-bce174d28ca6-istio-podinfo\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-d66hs\" (UID: \"14bfce12-7ae7-4866-aa46-bce174d28ca6\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-d66hs" Nov 03 13:21:01.490147 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:01.490119 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14bfce12-7ae7-4866-aa46-bce174d28ca6-workload-certs\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-d66hs\" (UID: \"14bfce12-7ae7-4866-aa46-bce174d28ca6\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-d66hs" Nov 03 13:21:01.490219 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:01.490204 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14bfce12-7ae7-4866-aa46-bce174d28ca6-workload-socket\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-d66hs\" (UID: \"14bfce12-7ae7-4866-aa46-bce174d28ca6\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-d66hs" Nov 03 13:21:01.490254 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:01.490228 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14bfce12-7ae7-4866-aa46-bce174d28ca6-istio-envoy\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-d66hs\" (UID: \"14bfce12-7ae7-4866-aa46-bce174d28ca6\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-d66hs" Nov 03 13:21:01.490392 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:01.490374 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14bfce12-7ae7-4866-aa46-bce174d28ca6-credential-socket\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-d66hs\" (UID: \"14bfce12-7ae7-4866-aa46-bce174d28ca6\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-d66hs" Nov 03 13:21:01.591735 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:01.591696 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gv797\" (UniqueName: \"kubernetes.io/projected/14bfce12-7ae7-4866-aa46-bce174d28ca6-kube-api-access-gv797\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-d66hs\" (UID: \"14bfce12-7ae7-4866-aa46-bce174d28ca6\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-d66hs" Nov 03 13:21:01.591735 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:01.591735 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/14bfce12-7ae7-4866-aa46-bce174d28ca6-istio-token\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-d66hs\" (UID: \"14bfce12-7ae7-4866-aa46-bce174d28ca6\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-d66hs" Nov 03 13:21:01.592044 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:01.591761 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14bfce12-7ae7-4866-aa46-bce174d28ca6-istio-data\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-d66hs\" (UID: \"14bfce12-7ae7-4866-aa46-bce174d28ca6\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-d66hs" Nov 03 13:21:01.592044 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:01.591839 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14bfce12-7ae7-4866-aa46-bce174d28ca6-istio-podinfo\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-d66hs\" (UID: \"14bfce12-7ae7-4866-aa46-bce174d28ca6\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-d66hs" Nov 03 13:21:01.592044 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:01.591879 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14bfce12-7ae7-4866-aa46-bce174d28ca6-workload-certs\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-d66hs\" (UID: \"14bfce12-7ae7-4866-aa46-bce174d28ca6\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-d66hs" Nov 03 13:21:01.592044 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:01.591918 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14bfce12-7ae7-4866-aa46-bce174d28ca6-workload-socket\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-d66hs\" (UID: \"14bfce12-7ae7-4866-aa46-bce174d28ca6\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-d66hs" Nov 03 13:21:01.592044 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:01.591944 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14bfce12-7ae7-4866-aa46-bce174d28ca6-istio-envoy\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-d66hs\" (UID: \"14bfce12-7ae7-4866-aa46-bce174d28ca6\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-d66hs" Nov 03 13:21:01.592044 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:01.592007 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14bfce12-7ae7-4866-aa46-bce174d28ca6-credential-socket\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-d66hs\" (UID: \"14bfce12-7ae7-4866-aa46-bce174d28ca6\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-d66hs" Nov 03 13:21:01.592448 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:01.592098 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14bfce12-7ae7-4866-aa46-bce174d28ca6-istiod-ca-cert\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-d66hs\" (UID: \"14bfce12-7ae7-4866-aa46-bce174d28ca6\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-d66hs" Nov 03 13:21:01.592448 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:01.592361 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14bfce12-7ae7-4866-aa46-bce174d28ca6-workload-certs\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-d66hs\" (UID: \"14bfce12-7ae7-4866-aa46-bce174d28ca6\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-d66hs" Nov 03 13:21:01.592679 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:01.592637 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14bfce12-7ae7-4866-aa46-bce174d28ca6-istio-data\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-d66hs\" (UID: \"14bfce12-7ae7-4866-aa46-bce174d28ca6\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-d66hs" Nov 03 13:21:01.592756 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:01.592715 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14bfce12-7ae7-4866-aa46-bce174d28ca6-credential-socket\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-d66hs\" (UID: \"14bfce12-7ae7-4866-aa46-bce174d28ca6\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-d66hs" Nov 03 13:21:01.592990 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:01.592964 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14bfce12-7ae7-4866-aa46-bce174d28ca6-istiod-ca-cert\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-d66hs\" (UID: \"14bfce12-7ae7-4866-aa46-bce174d28ca6\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-d66hs" Nov 03 13:21:01.593073 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:01.592983 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14bfce12-7ae7-4866-aa46-bce174d28ca6-workload-socket\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-d66hs\" (UID: \"14bfce12-7ae7-4866-aa46-bce174d28ca6\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-d66hs" Nov 03 13:21:01.594770 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:01.594743 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14bfce12-7ae7-4866-aa46-bce174d28ca6-istio-envoy\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-d66hs\" (UID: \"14bfce12-7ae7-4866-aa46-bce174d28ca6\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-d66hs" Nov 03 13:21:01.594987 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:01.594963 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14bfce12-7ae7-4866-aa46-bce174d28ca6-istio-podinfo\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-d66hs\" (UID: \"14bfce12-7ae7-4866-aa46-bce174d28ca6\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-d66hs" Nov 03 13:21:01.601963 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:01.601940 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/14bfce12-7ae7-4866-aa46-bce174d28ca6-istio-token\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-d66hs\" (UID: \"14bfce12-7ae7-4866-aa46-bce174d28ca6\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-d66hs" Nov 03 13:21:01.602076 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:01.602009 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gv797\" (UniqueName: \"kubernetes.io/projected/14bfce12-7ae7-4866-aa46-bce174d28ca6-kube-api-access-gv797\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-d66hs\" (UID: \"14bfce12-7ae7-4866-aa46-bce174d28ca6\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-d66hs" Nov 03 13:21:01.747435 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:01.747348 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-d66hs" Nov 03 13:21:02.034180 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:02.034078 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-d66hs"] Nov 03 13:21:02.039906 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:21:02.039868 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14bfce12_7ae7_4866_aa46_bce174d28ca6.slice/crio-db87f7221ec232574aa0ddb54f0eb29f4c55ca4b81071f0f877f24fef00749be WatchSource:0}: Error finding container db87f7221ec232574aa0ddb54f0eb29f4c55ca4b81071f0f877f24fef00749be: Status 404 returned error can't find the container with id db87f7221ec232574aa0ddb54f0eb29f4c55ca4b81071f0f877f24fef00749be Nov 03 13:21:02.045327 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:02.045265 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:21:02.045441 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:02.045381 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:21:02.045441 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:02.045433 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:21:03.022382 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:03.022338 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-d66hs" event={"ID":"14bfce12-7ae7-4866-aa46-bce174d28ca6","Type":"ContainerStarted","Data":"75148bbfc8ef536a74d3997858931cd0358236c4a5f8adbc9a723758f3a07290"} Nov 03 13:21:03.022586 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:03.022438 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-d66hs" event={"ID":"14bfce12-7ae7-4866-aa46-bce174d28ca6","Type":"ContainerStarted","Data":"db87f7221ec232574aa0ddb54f0eb29f4c55ca4b81071f0f877f24fef00749be"} Nov 03 13:21:03.048392 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:03.048341 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-d66hs" podStartSLOduration=2.048326884 podStartE2EDuration="2.048326884s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:21:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:21:03.045430376 +0000 UTC m=+743.053621653" watchObservedRunningTime="2026-11-03 13:21:03.048326884 +0000 UTC m=+743.056518218" Nov 03 13:21:03.748345 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:03.748280 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-d66hs" Nov 03 13:21:03.750296 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:03.750252 5024 patch_prober.go:29] interesting pod/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-d66hs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Nov 03 13:21:03.750429 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:03.750330 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-d66hs" podUID="14bfce12-7ae7-4866-aa46-bce174d28ca6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Nov 03 13:21:04.747954 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:04.747916 5024 patch_prober.go:29] interesting pod/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-d66hs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Nov 03 13:21:04.748343 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:04.747973 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-d66hs" podUID="14bfce12-7ae7-4866-aa46-bce174d28ca6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Nov 03 13:21:05.748569 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:05.748534 5024 patch_prober.go:29] interesting pod/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-d66hs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Nov 03 13:21:05.748964 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:05.748596 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-d66hs" podUID="14bfce12-7ae7-4866-aa46-bce174d28ca6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Nov 03 13:21:06.748391 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:06.748353 5024 patch_prober.go:29] interesting pod/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-d66hs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Nov 03 13:21:06.748680 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:06.748651 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-d66hs" podUID="14bfce12-7ae7-4866-aa46-bce174d28ca6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Nov 03 13:21:07.749436 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:07.749397 5024 patch_prober.go:29] interesting pod/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-d66hs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Nov 03 13:21:07.750127 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:07.750095 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-d66hs" podUID="14bfce12-7ae7-4866-aa46-bce174d28ca6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Nov 03 13:21:08.752705 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:08.752665 5024 patch_prober.go:29] interesting pod/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-d66hs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Nov 03 13:21:08.753317 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:08.753254 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-d66hs" podUID="14bfce12-7ae7-4866-aa46-bce174d28ca6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Nov 03 13:21:09.023356 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:09.016095 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-wkz2r"] Nov 03 13:21:09.127310 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:09.125780 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-wkz2r"] Nov 03 13:21:09.127310 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:09.125980 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-wkz2r" Nov 03 13:21:09.213315 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:09.190712 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-7c192b-openshift-default-dockercfg-rx6fs\"" Nov 03 13:21:09.348971 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:09.348924 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c49c4c7b-3182-4699-898a-5407d9a3fea1-istio-token\") pod \"e2e-extmod-b-7c192b-openshift-default-5c496b978f-wkz2r\" (UID: \"c49c4c7b-3182-4699-898a-5407d9a3fea1\") " pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-wkz2r" Nov 03 13:21:09.418307 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:09.400978 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c49c4c7b-3182-4699-898a-5407d9a3fea1-credential-socket\") pod \"e2e-extmod-b-7c192b-openshift-default-5c496b978f-wkz2r\" (UID: \"c49c4c7b-3182-4699-898a-5407d9a3fea1\") " pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-wkz2r" Nov 03 13:21:09.418307 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:09.401147 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c49c4c7b-3182-4699-898a-5407d9a3fea1-workload-socket\") pod \"e2e-extmod-b-7c192b-openshift-default-5c496b978f-wkz2r\" (UID: \"c49c4c7b-3182-4699-898a-5407d9a3fea1\") " pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-wkz2r" Nov 03 13:21:09.546887 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:09.401415 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7fp4d\" (UniqueName: \"kubernetes.io/projected/c49c4c7b-3182-4699-898a-5407d9a3fea1-kube-api-access-7fp4d\") pod \"e2e-extmod-b-7c192b-openshift-default-5c496b978f-wkz2r\" (UID: \"c49c4c7b-3182-4699-898a-5407d9a3fea1\") " pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-wkz2r" Nov 03 13:21:09.547265 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:09.547235 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c49c4c7b-3182-4699-898a-5407d9a3fea1-istiod-ca-cert\") pod \"e2e-extmod-b-7c192b-openshift-default-5c496b978f-wkz2r\" (UID: \"c49c4c7b-3182-4699-898a-5407d9a3fea1\") " pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-wkz2r" Nov 03 13:21:09.551415 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:09.551367 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c49c4c7b-3182-4699-898a-5407d9a3fea1-istio-data\") pod \"e2e-extmod-b-7c192b-openshift-default-5c496b978f-wkz2r\" (UID: \"c49c4c7b-3182-4699-898a-5407d9a3fea1\") " pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-wkz2r" Nov 03 13:21:09.551730 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:09.551696 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c49c4c7b-3182-4699-898a-5407d9a3fea1-istio-envoy\") pod \"e2e-extmod-b-7c192b-openshift-default-5c496b978f-wkz2r\" (UID: \"c49c4c7b-3182-4699-898a-5407d9a3fea1\") " pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-wkz2r" Nov 03 13:21:09.551923 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:09.551895 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c49c4c7b-3182-4699-898a-5407d9a3fea1-istio-podinfo\") pod \"e2e-extmod-b-7c192b-openshift-default-5c496b978f-wkz2r\" (UID: \"c49c4c7b-3182-4699-898a-5407d9a3fea1\") " pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-wkz2r" Nov 03 13:21:09.552104 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:09.552072 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c49c4c7b-3182-4699-898a-5407d9a3fea1-workload-certs\") pod \"e2e-extmod-b-7c192b-openshift-default-5c496b978f-wkz2r\" (UID: \"c49c4c7b-3182-4699-898a-5407d9a3fea1\") " pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-wkz2r" Nov 03 13:21:09.678367 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:09.675467 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c49c4c7b-3182-4699-898a-5407d9a3fea1-istiod-ca-cert\") pod \"e2e-extmod-b-7c192b-openshift-default-5c496b978f-wkz2r\" (UID: \"c49c4c7b-3182-4699-898a-5407d9a3fea1\") " pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-wkz2r" Nov 03 13:21:09.678367 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:09.675521 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c49c4c7b-3182-4699-898a-5407d9a3fea1-istio-data\") pod \"e2e-extmod-b-7c192b-openshift-default-5c496b978f-wkz2r\" (UID: \"c49c4c7b-3182-4699-898a-5407d9a3fea1\") " pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-wkz2r" Nov 03 13:21:09.678367 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:09.675620 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c49c4c7b-3182-4699-898a-5407d9a3fea1-istio-envoy\") pod \"e2e-extmod-b-7c192b-openshift-default-5c496b978f-wkz2r\" (UID: \"c49c4c7b-3182-4699-898a-5407d9a3fea1\") " pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-wkz2r" Nov 03 13:21:09.678367 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:09.675672 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c49c4c7b-3182-4699-898a-5407d9a3fea1-istio-podinfo\") pod \"e2e-extmod-b-7c192b-openshift-default-5c496b978f-wkz2r\" (UID: \"c49c4c7b-3182-4699-898a-5407d9a3fea1\") " pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-wkz2r" Nov 03 13:21:09.678367 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:09.675733 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c49c4c7b-3182-4699-898a-5407d9a3fea1-workload-certs\") pod \"e2e-extmod-b-7c192b-openshift-default-5c496b978f-wkz2r\" (UID: \"c49c4c7b-3182-4699-898a-5407d9a3fea1\") " pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-wkz2r" Nov 03 13:21:09.678367 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:09.675834 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c49c4c7b-3182-4699-898a-5407d9a3fea1-istio-token\") pod \"e2e-extmod-b-7c192b-openshift-default-5c496b978f-wkz2r\" (UID: \"c49c4c7b-3182-4699-898a-5407d9a3fea1\") " pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-wkz2r" Nov 03 13:21:09.678367 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:09.675881 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c49c4c7b-3182-4699-898a-5407d9a3fea1-credential-socket\") pod \"e2e-extmod-b-7c192b-openshift-default-5c496b978f-wkz2r\" (UID: \"c49c4c7b-3182-4699-898a-5407d9a3fea1\") " pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-wkz2r" Nov 03 13:21:09.678367 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:09.675919 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c49c4c7b-3182-4699-898a-5407d9a3fea1-workload-socket\") pod \"e2e-extmod-b-7c192b-openshift-default-5c496b978f-wkz2r\" (UID: \"c49c4c7b-3182-4699-898a-5407d9a3fea1\") " pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-wkz2r" Nov 03 13:21:09.678367 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:09.675987 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7fp4d\" (UniqueName: \"kubernetes.io/projected/c49c4c7b-3182-4699-898a-5407d9a3fea1-kube-api-access-7fp4d\") pod \"e2e-extmod-b-7c192b-openshift-default-5c496b978f-wkz2r\" (UID: \"c49c4c7b-3182-4699-898a-5407d9a3fea1\") " pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-wkz2r" Nov 03 13:21:09.678367 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:09.677151 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c49c4c7b-3182-4699-898a-5407d9a3fea1-istiod-ca-cert\") pod \"e2e-extmod-b-7c192b-openshift-default-5c496b978f-wkz2r\" (UID: \"c49c4c7b-3182-4699-898a-5407d9a3fea1\") " pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-wkz2r" Nov 03 13:21:09.701123 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:09.700751 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c49c4c7b-3182-4699-898a-5407d9a3fea1-workload-certs\") pod \"e2e-extmod-b-7c192b-openshift-default-5c496b978f-wkz2r\" (UID: \"c49c4c7b-3182-4699-898a-5407d9a3fea1\") " pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-wkz2r" Nov 03 13:21:09.701844 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:09.701800 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c49c4c7b-3182-4699-898a-5407d9a3fea1-credential-socket\") pod \"e2e-extmod-b-7c192b-openshift-default-5c496b978f-wkz2r\" (UID: \"c49c4c7b-3182-4699-898a-5407d9a3fea1\") " pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-wkz2r" Nov 03 13:21:09.738750 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:09.738213 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c49c4c7b-3182-4699-898a-5407d9a3fea1-istio-podinfo\") pod \"e2e-extmod-b-7c192b-openshift-default-5c496b978f-wkz2r\" (UID: \"c49c4c7b-3182-4699-898a-5407d9a3fea1\") " pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-wkz2r" Nov 03 13:21:09.738750 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:09.738541 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c49c4c7b-3182-4699-898a-5407d9a3fea1-workload-socket\") pod \"e2e-extmod-b-7c192b-openshift-default-5c496b978f-wkz2r\" (UID: \"c49c4c7b-3182-4699-898a-5407d9a3fea1\") " pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-wkz2r" Nov 03 13:21:09.738750 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:09.738709 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c49c4c7b-3182-4699-898a-5407d9a3fea1-istio-envoy\") pod \"e2e-extmod-b-7c192b-openshift-default-5c496b978f-wkz2r\" (UID: \"c49c4c7b-3182-4699-898a-5407d9a3fea1\") " pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-wkz2r" Nov 03 13:21:09.759253 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:09.758944 5024 patch_prober.go:29] interesting pod/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-d66hs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Nov 03 13:21:09.759253 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:09.759011 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-d66hs" podUID="14bfce12-7ae7-4866-aa46-bce174d28ca6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Nov 03 13:21:09.761032 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:09.760967 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c49c4c7b-3182-4699-898a-5407d9a3fea1-istio-token\") pod \"e2e-extmod-b-7c192b-openshift-default-5c496b978f-wkz2r\" (UID: \"c49c4c7b-3182-4699-898a-5407d9a3fea1\") " pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-wkz2r" Nov 03 13:21:09.799256 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:09.799221 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c49c4c7b-3182-4699-898a-5407d9a3fea1-istio-data\") pod \"e2e-extmod-b-7c192b-openshift-default-5c496b978f-wkz2r\" (UID: \"c49c4c7b-3182-4699-898a-5407d9a3fea1\") " pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-wkz2r" Nov 03 13:21:09.886811 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:09.885987 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7fp4d\" (UniqueName: \"kubernetes.io/projected/c49c4c7b-3182-4699-898a-5407d9a3fea1-kube-api-access-7fp4d\") pod \"e2e-extmod-b-7c192b-openshift-default-5c496b978f-wkz2r\" (UID: \"c49c4c7b-3182-4699-898a-5407d9a3fea1\") " pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-wkz2r" Nov 03 13:21:09.995357 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:21:09.990842 5024 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d7mwz" containerName="extractor" Nov 03 13:21:10.222307 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:10.162896 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-wkz2r" Nov 03 13:21:10.693990 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:10.693947 5024 patch_prober.go:29] interesting pod/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-d2rlz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 13:21:10.694419 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:10.694384 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-d2rlz" podUID="af43a1c8-fad2-4b80-a0e9-65ab23c3612e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 13:21:11.767391 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:11.766392 5024 patch_prober.go:29] interesting pod/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-d66hs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 13:21:11.767391 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:11.766472 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-d66hs" podUID="14bfce12-7ae7-4866-aa46-bce174d28ca6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 13:21:11.812382 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:11.795392 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-wkz2r"] Nov 03 13:21:11.816407 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:21:11.814991 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc49c4c7b_3182_4699_898a_5407d9a3fea1.slice/crio-586a327000d96a7254f0e5b00388bba99719696e25e0eb79f20ee0a410ac6575 WatchSource:0}: Error finding container 586a327000d96a7254f0e5b00388bba99719696e25e0eb79f20ee0a410ac6575: Status 404 returned error can't find the container with id 586a327000d96a7254f0e5b00388bba99719696e25e0eb79f20ee0a410ac6575 Nov 03 13:21:11.844603 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:11.844559 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:21:11.844866 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:11.844840 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:21:11.845077 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:11.845035 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:21:12.497310 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:12.496744 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-d66hs" Nov 03 13:21:12.501313 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:12.497968 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-d66hs" Nov 03 13:21:12.509870 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:12.509611 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-d66hs" Nov 03 13:21:12.721211 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:12.721137 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-wkz2r" event={"ID":"c49c4c7b-3182-4699-898a-5407d9a3fea1","Type":"ContainerStarted","Data":"fed93113530f4bcfe3c23ec37252c5c1ca8e3331bd12050e56155ad271ef7d79"} Nov 03 13:21:12.721668 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:12.721640 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-wkz2r" event={"ID":"c49c4c7b-3182-4699-898a-5407d9a3fea1","Type":"ContainerStarted","Data":"586a327000d96a7254f0e5b00388bba99719696e25e0eb79f20ee0a410ac6575"} Nov 03 13:21:12.797357 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:12.784704 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-wkz2r" podStartSLOduration=4.784680058 podStartE2EDuration="4.784680058s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:21:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:21:12.780600481 +0000 UTC m=+752.788791763" watchObservedRunningTime="2026-11-03 13:21:12.784680058 +0000 UTC m=+752.792871336" Nov 03 13:21:13.164222 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:13.164181 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-wkz2r" Nov 03 13:21:13.169859 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:13.169833 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-wkz2r" Nov 03 13:21:13.725463 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:13.725426 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-wkz2r" Nov 03 13:21:13.726360 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:13.726342 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-wkz2r" Nov 03 13:21:16.867355 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:16.867322 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-wwvqr"] Nov 03 13:21:16.886618 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:16.886587 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-wwvqr" Nov 03 13:21:16.951928 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:16.951891 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-wwvqr"] Nov 03 13:21:17.056806 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:17.055507 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fffd24a4-d55d-4661-a37d-c344d3343698-istio-data\") pod \"e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-wwvqr\" (UID: \"fffd24a4-d55d-4661-a37d-c344d3343698\") " pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-wwvqr" Nov 03 13:21:17.056806 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:17.056405 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fffd24a4-d55d-4661-a37d-c344d3343698-istio-podinfo\") pod \"e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-wwvqr\" (UID: \"fffd24a4-d55d-4661-a37d-c344d3343698\") " pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-wwvqr" Nov 03 13:21:17.056806 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:17.056530 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-74rcf\" (UniqueName: \"kubernetes.io/projected/fffd24a4-d55d-4661-a37d-c344d3343698-kube-api-access-74rcf\") pod \"e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-wwvqr\" (UID: \"fffd24a4-d55d-4661-a37d-c344d3343698\") " pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-wwvqr" Nov 03 13:21:17.056806 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:17.056588 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fffd24a4-d55d-4661-a37d-c344d3343698-workload-certs\") pod \"e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-wwvqr\" (UID: \"fffd24a4-d55d-4661-a37d-c344d3343698\") " pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-wwvqr" Nov 03 13:21:17.056806 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:17.056610 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fffd24a4-d55d-4661-a37d-c344d3343698-istio-envoy\") pod \"e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-wwvqr\" (UID: \"fffd24a4-d55d-4661-a37d-c344d3343698\") " pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-wwvqr" Nov 03 13:21:17.056806 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:17.056679 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fffd24a4-d55d-4661-a37d-c344d3343698-credential-socket\") pod \"e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-wwvqr\" (UID: \"fffd24a4-d55d-4661-a37d-c344d3343698\") " pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-wwvqr" Nov 03 13:21:17.056806 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:17.056734 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fffd24a4-d55d-4661-a37d-c344d3343698-workload-socket\") pod \"e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-wwvqr\" (UID: \"fffd24a4-d55d-4661-a37d-c344d3343698\") " pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-wwvqr" Nov 03 13:21:17.057335 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:17.056843 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fffd24a4-d55d-4661-a37d-c344d3343698-istio-token\") pod \"e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-wwvqr\" (UID: \"fffd24a4-d55d-4661-a37d-c344d3343698\") " pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-wwvqr" Nov 03 13:21:17.057335 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:17.056898 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fffd24a4-d55d-4661-a37d-c344d3343698-istiod-ca-cert\") pod \"e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-wwvqr\" (UID: \"fffd24a4-d55d-4661-a37d-c344d3343698\") " pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-wwvqr" Nov 03 13:21:17.158227 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:17.158132 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fffd24a4-d55d-4661-a37d-c344d3343698-workload-socket\") pod \"e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-wwvqr\" (UID: \"fffd24a4-d55d-4661-a37d-c344d3343698\") " pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-wwvqr" Nov 03 13:21:17.158675 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:17.158642 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fffd24a4-d55d-4661-a37d-c344d3343698-istio-token\") pod \"e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-wwvqr\" (UID: \"fffd24a4-d55d-4661-a37d-c344d3343698\") " pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-wwvqr" Nov 03 13:21:17.158675 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:17.158668 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fffd24a4-d55d-4661-a37d-c344d3343698-workload-socket\") pod \"e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-wwvqr\" (UID: \"fffd24a4-d55d-4661-a37d-c344d3343698\") " pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-wwvqr" Nov 03 13:21:17.158811 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:17.158724 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fffd24a4-d55d-4661-a37d-c344d3343698-istiod-ca-cert\") pod \"e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-wwvqr\" (UID: \"fffd24a4-d55d-4661-a37d-c344d3343698\") " pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-wwvqr" Nov 03 13:21:17.158811 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:17.158761 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fffd24a4-d55d-4661-a37d-c344d3343698-istio-data\") pod \"e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-wwvqr\" (UID: \"fffd24a4-d55d-4661-a37d-c344d3343698\") " pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-wwvqr" Nov 03 13:21:17.158811 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:17.158792 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fffd24a4-d55d-4661-a37d-c344d3343698-istio-podinfo\") pod \"e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-wwvqr\" (UID: \"fffd24a4-d55d-4661-a37d-c344d3343698\") " pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-wwvqr" Nov 03 13:21:17.158960 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:17.158845 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74rcf\" (UniqueName: \"kubernetes.io/projected/fffd24a4-d55d-4661-a37d-c344d3343698-kube-api-access-74rcf\") pod \"e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-wwvqr\" (UID: \"fffd24a4-d55d-4661-a37d-c344d3343698\") " pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-wwvqr" Nov 03 13:21:17.158960 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:17.158907 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fffd24a4-d55d-4661-a37d-c344d3343698-workload-certs\") pod \"e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-wwvqr\" (UID: \"fffd24a4-d55d-4661-a37d-c344d3343698\") " pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-wwvqr" Nov 03 13:21:17.158960 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:17.158928 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fffd24a4-d55d-4661-a37d-c344d3343698-istio-envoy\") pod \"e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-wwvqr\" (UID: \"fffd24a4-d55d-4661-a37d-c344d3343698\") " pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-wwvqr" Nov 03 13:21:17.159123 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:17.158988 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fffd24a4-d55d-4661-a37d-c344d3343698-credential-socket\") pod \"e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-wwvqr\" (UID: \"fffd24a4-d55d-4661-a37d-c344d3343698\") " pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-wwvqr" Nov 03 13:21:17.159505 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:17.159479 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fffd24a4-d55d-4661-a37d-c344d3343698-istio-data\") pod \"e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-wwvqr\" (UID: \"fffd24a4-d55d-4661-a37d-c344d3343698\") " pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-wwvqr" Nov 03 13:21:17.159661 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:17.159527 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fffd24a4-d55d-4661-a37d-c344d3343698-workload-certs\") pod \"e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-wwvqr\" (UID: \"fffd24a4-d55d-4661-a37d-c344d3343698\") " pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-wwvqr" Nov 03 13:21:17.159764 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:17.159745 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fffd24a4-d55d-4661-a37d-c344d3343698-credential-socket\") pod \"e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-wwvqr\" (UID: \"fffd24a4-d55d-4661-a37d-c344d3343698\") " pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-wwvqr" Nov 03 13:21:17.160150 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:17.160123 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fffd24a4-d55d-4661-a37d-c344d3343698-istiod-ca-cert\") pod \"e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-wwvqr\" (UID: \"fffd24a4-d55d-4661-a37d-c344d3343698\") " pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-wwvqr" Nov 03 13:21:17.162216 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:17.162164 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fffd24a4-d55d-4661-a37d-c344d3343698-istio-podinfo\") pod \"e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-wwvqr\" (UID: \"fffd24a4-d55d-4661-a37d-c344d3343698\") " pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-wwvqr" Nov 03 13:21:17.163558 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:17.163525 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fffd24a4-d55d-4661-a37d-c344d3343698-istio-envoy\") pod \"e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-wwvqr\" (UID: \"fffd24a4-d55d-4661-a37d-c344d3343698\") " pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-wwvqr" Nov 03 13:21:17.174392 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:17.174347 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fffd24a4-d55d-4661-a37d-c344d3343698-istio-token\") pod \"e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-wwvqr\" (UID: \"fffd24a4-d55d-4661-a37d-c344d3343698\") " pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-wwvqr" Nov 03 13:21:17.187134 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:17.187078 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-74rcf\" (UniqueName: \"kubernetes.io/projected/fffd24a4-d55d-4661-a37d-c344d3343698-kube-api-access-74rcf\") pod \"e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-wwvqr\" (UID: \"fffd24a4-d55d-4661-a37d-c344d3343698\") " pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-wwvqr" Nov 03 13:21:17.208847 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:17.208808 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-wwvqr" Nov 03 13:21:17.545648 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:17.545605 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-wwvqr"] Nov 03 13:21:17.546403 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:21:17.546355 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfffd24a4_d55d_4661_a37d_c344d3343698.slice/crio-01027176032f1ba6e445c9a9520865f3c993d7e7b2ce2c170cec6e7845375509 WatchSource:0}: Error finding container 01027176032f1ba6e445c9a9520865f3c993d7e7b2ce2c170cec6e7845375509: Status 404 returned error can't find the container with id 01027176032f1ba6e445c9a9520865f3c993d7e7b2ce2c170cec6e7845375509 Nov 03 13:21:17.549816 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:17.549705 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:21:17.549816 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:17.549794 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:21:17.549816 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:17.549850 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:21:17.770304 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:17.767394 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-wwvqr" event={"ID":"fffd24a4-d55d-4661-a37d-c344d3343698","Type":"ContainerStarted","Data":"32d699d9bfb32447f36dcd5aa244107d932d046cdacc4d81479e50a884ded653"} Nov 03 13:21:17.770304 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:17.767518 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-wwvqr" event={"ID":"fffd24a4-d55d-4661-a37d-c344d3343698","Type":"ContainerStarted","Data":"01027176032f1ba6e445c9a9520865f3c993d7e7b2ce2c170cec6e7845375509"} Nov 03 13:21:17.796882 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:17.796427 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-wwvqr" podStartSLOduration=1.796408644 podStartE2EDuration="1.796408644s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:21:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:21:17.792200486 +0000 UTC m=+757.800391762" watchObservedRunningTime="2026-11-03 13:21:17.796408644 +0000 UTC m=+757.804599922" Nov 03 13:21:18.223716 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:18.223208 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-wwvqr" Nov 03 13:21:18.223716 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:18.223369 5024 patch_prober.go:29] interesting pod/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-wwvqr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Nov 03 13:21:18.223716 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:18.223435 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-wwvqr" podUID="fffd24a4-d55d-4661-a37d-c344d3343698" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Nov 03 13:21:19.209785 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:19.209738 5024 patch_prober.go:29] interesting pod/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-wwvqr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Nov 03 13:21:19.210075 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:19.210046 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-wwvqr" podUID="fffd24a4-d55d-4661-a37d-c344d3343698" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Nov 03 13:21:20.215256 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:20.214805 5024 patch_prober.go:29] interesting pod/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-wwvqr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Nov 03 13:21:20.215256 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:20.214868 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-wwvqr" podUID="fffd24a4-d55d-4661-a37d-c344d3343698" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Nov 03 13:21:20.822763 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:20.822734 5024 generic.go:356] "Generic (PLEG): container finished" podID="736b02c5-9833-4f1b-9f9a-e4c22ab23ac9" containerID="80f9d6d535afd0f850b398d01f7f1d9c71d9147c90bff43a2c95173a88c39ca9" exitCode=0 Nov 03 13:21:20.822949 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:20.822789 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-ded926-openshift-default-6bdf65b545-pjqfb" event={"ID":"736b02c5-9833-4f1b-9f9a-e4c22ab23ac9","Type":"ContainerDied","Data":"80f9d6d535afd0f850b398d01f7f1d9c71d9147c90bff43a2c95173a88c39ca9"} Nov 03 13:21:20.823246 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:20.823225 5024 scope.go:122] "RemoveContainer" containerID="80f9d6d535afd0f850b398d01f7f1d9c71d9147c90bff43a2c95173a88c39ca9" Nov 03 13:21:20.824790 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:20.824763 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:21:20.824864 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:20.824838 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:21:20.824908 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:20.824869 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:21:20.839136 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:20.839109 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-7c192b-65886b4cf7-n24wm"] Nov 03 13:21:20.845122 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:20.845098 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-7c192b-65886b4cf7-n24wm" Nov 03 13:21:20.850314 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:20.850275 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-7c192b-65886b4cf7-n24wm"] Nov 03 13:21:20.858274 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:20.856983 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/11a9f177-f213-4929-9a64-19065b49b3f5-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-7c192b-65886b4cf7-n24wm\" (UID: \"11a9f177-f213-4929-9a64-19065b49b3f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-7c192b-65886b4cf7-n24wm" Nov 03 13:21:20.858274 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:20.857027 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/11a9f177-f213-4929-9a64-19065b49b3f5-metrics-certs\") pod \"maas-api-e2e-extmod-b-7c192b-65886b4cf7-n24wm\" (UID: \"11a9f177-f213-4929-9a64-19065b49b3f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-7c192b-65886b4cf7-n24wm" Nov 03 13:21:20.858274 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:20.857144 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rdnwq\" (UniqueName: \"kubernetes.io/projected/11a9f177-f213-4929-9a64-19065b49b3f5-kube-api-access-rdnwq\") pod \"maas-api-e2e-extmod-b-7c192b-65886b4cf7-n24wm\" (UID: \"11a9f177-f213-4929-9a64-19065b49b3f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-7c192b-65886b4cf7-n24wm" Nov 03 13:21:20.858274 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:20.857396 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/11a9f177-f213-4929-9a64-19065b49b3f5-maas-api-tls\") pod \"maas-api-e2e-extmod-b-7c192b-65886b4cf7-n24wm\" (UID: \"11a9f177-f213-4929-9a64-19065b49b3f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-7c192b-65886b4cf7-n24wm" Nov 03 13:21:20.862702 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:20.861080 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-b-7c192b-serving-cert\"" Nov 03 13:21:21.016072 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:20.992638 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/11a9f177-f213-4929-9a64-19065b49b3f5-maas-api-tls\") pod \"maas-api-e2e-extmod-b-7c192b-65886b4cf7-n24wm\" (UID: \"11a9f177-f213-4929-9a64-19065b49b3f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-7c192b-65886b4cf7-n24wm" Nov 03 13:21:21.016072 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:21.014572 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/11a9f177-f213-4929-9a64-19065b49b3f5-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-7c192b-65886b4cf7-n24wm\" (UID: \"11a9f177-f213-4929-9a64-19065b49b3f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-7c192b-65886b4cf7-n24wm" Nov 03 13:21:21.016072 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:21.014616 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/11a9f177-f213-4929-9a64-19065b49b3f5-metrics-certs\") pod \"maas-api-e2e-extmod-b-7c192b-65886b4cf7-n24wm\" (UID: \"11a9f177-f213-4929-9a64-19065b49b3f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-7c192b-65886b4cf7-n24wm" Nov 03 13:21:21.016072 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:21.014744 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdnwq\" (UniqueName: \"kubernetes.io/projected/11a9f177-f213-4929-9a64-19065b49b3f5-kube-api-access-rdnwq\") pod \"maas-api-e2e-extmod-b-7c192b-65886b4cf7-n24wm\" (UID: \"11a9f177-f213-4929-9a64-19065b49b3f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-7c192b-65886b4cf7-n24wm" Nov 03 13:21:21.017650 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:21.017619 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/11a9f177-f213-4929-9a64-19065b49b3f5-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-7c192b-65886b4cf7-n24wm\" (UID: \"11a9f177-f213-4929-9a64-19065b49b3f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-7c192b-65886b4cf7-n24wm" Nov 03 13:21:21.028870 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:21.027023 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/11a9f177-f213-4929-9a64-19065b49b3f5-metrics-certs\") pod \"maas-api-e2e-extmod-b-7c192b-65886b4cf7-n24wm\" (UID: \"11a9f177-f213-4929-9a64-19065b49b3f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-7c192b-65886b4cf7-n24wm" Nov 03 13:21:21.028870 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:21.027455 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/11a9f177-f213-4929-9a64-19065b49b3f5-maas-api-tls\") pod \"maas-api-e2e-extmod-b-7c192b-65886b4cf7-n24wm\" (UID: \"11a9f177-f213-4929-9a64-19065b49b3f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-7c192b-65886b4cf7-n24wm" Nov 03 13:21:21.029198 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:21.029132 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rdnwq\" (UniqueName: \"kubernetes.io/projected/11a9f177-f213-4929-9a64-19065b49b3f5-kube-api-access-rdnwq\") pod \"maas-api-e2e-extmod-b-7c192b-65886b4cf7-n24wm\" (UID: \"11a9f177-f213-4929-9a64-19065b49b3f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-7c192b-65886b4cf7-n24wm" Nov 03 13:21:21.209671 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:21.209622 5024 patch_prober.go:29] interesting pod/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-wwvqr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Nov 03 13:21:21.209869 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:21.209696 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-wwvqr" podUID="fffd24a4-d55d-4661-a37d-c344d3343698" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Nov 03 13:21:21.319150 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:21.319040 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-7c192b-65886b4cf7-n24wm" Nov 03 13:21:21.812659 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:21.812621 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-7c192b-65886b4cf7-n24wm"] Nov 03 13:21:21.935665 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:21.935615 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-ded926-openshift-default-6bdf65b545-pjqfb" event={"ID":"736b02c5-9833-4f1b-9f9a-e4c22ab23ac9","Type":"ContainerStarted","Data":"d801f9bdd21d9a710d7f86b21b1822b6c6daf6b25709307c6052bfc6d103b5dc"} Nov 03 13:21:21.938583 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:21.938540 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-7c192b-65886b4cf7-n24wm" event={"ID":"11a9f177-f213-4929-9a64-19065b49b3f5","Type":"ContainerStarted","Data":"827c29fd3592cc5c7ab2b7d41cea228398586aac1eaf7d63b0402915abad59f0"} Nov 03 13:21:22.224403 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:22.224367 5024 patch_prober.go:29] interesting pod/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-wwvqr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Nov 03 13:21:22.224779 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:22.224747 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-wwvqr" podUID="fffd24a4-d55d-4661-a37d-c344d3343698" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Nov 03 13:21:22.969413 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:22.958560 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-ded926-openshift-default-6bdf65b545-pjqfb" Nov 03 13:21:22.969413 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:22.962401 5024 patch_prober.go:29] interesting pod/e2e-extmod-a-ded926-openshift-default-6bdf65b545-pjqfb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Nov 03 13:21:22.969413 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:22.962469 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-ded926-openshift-default-6bdf65b545-pjqfb" podUID="736b02c5-9833-4f1b-9f9a-e4c22ab23ac9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Nov 03 13:21:23.210031 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:23.210000 5024 patch_prober.go:29] interesting pod/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-wwvqr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Nov 03 13:21:23.210223 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:23.210196 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-wwvqr" podUID="fffd24a4-d55d-4661-a37d-c344d3343698" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Nov 03 13:21:23.928309 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:23.927683 5024 patch_prober.go:29] interesting pod/e2e-extmod-a-ded926-openshift-default-6bdf65b545-pjqfb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Nov 03 13:21:23.928309 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:23.927749 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-ded926-openshift-default-6bdf65b545-pjqfb" podUID="736b02c5-9833-4f1b-9f9a-e4c22ab23ac9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Nov 03 13:21:24.161924 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:24.161883 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-7c192b-65886b4cf7-n24wm" event={"ID":"11a9f177-f213-4929-9a64-19065b49b3f5","Type":"ContainerStarted","Data":"1b2fbb0821772430b070fdaef7983496a80f01d2782f460cf53575a44ba4fda7"} Nov 03 13:21:24.174617 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:24.174589 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-7c192b-65886b4cf7-n24wm" Nov 03 13:21:24.213364 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:24.213237 5024 patch_prober.go:29] interesting pod/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-wwvqr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Nov 03 13:21:24.213364 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:24.213317 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-wwvqr" podUID="fffd24a4-d55d-4661-a37d-c344d3343698" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Nov 03 13:21:24.803130 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:24.803089 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-db71f4/test-model-db71f4-kserve-6cf694c5f9-57cmd" podUID="029ef988-b71c-4de2-9b27-d7506a869ed8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 13:21:24.876902 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:24.876854 5024 patch_prober.go:29] interesting pod/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-d7gxh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 13:21:24.877163 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:24.877136 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-d7gxh" podUID="6ec0db42-ee6d-496c-8792-a17fe716e820" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 13:21:24.950326 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:24.935528 5024 patch_prober.go:29] interesting pod/e2e-extmod-a-ded926-openshift-default-6bdf65b545-pjqfb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Nov 03 13:21:24.950326 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:24.935592 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-ded926-openshift-default-6bdf65b545-pjqfb" podUID="736b02c5-9833-4f1b-9f9a-e4c22ab23ac9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Nov 03 13:21:25.212428 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:25.212380 5024 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-inf-b-db71f4/test-model-db71f4-kserve-6cf694c5f9-57cmd" podUID="029ef988-b71c-4de2-9b27-d7506a869ed8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 13:21:25.213606 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:25.213579 5024 patch_prober.go:29] interesting pod/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-wwvqr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Nov 03 13:21:25.213788 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:25.213767 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-wwvqr" podUID="fffd24a4-d55d-4661-a37d-c344d3343698" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Nov 03 13:21:25.645357 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:25.615893 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-ded926-openshift-default-6bdf65b545-pjqfb" Nov 03 13:21:25.698442 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:25.698398 5024 patch_prober.go:29] interesting pod/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-d2rlz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 13:21:25.703337 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:25.698716 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-d2rlz" podUID="af43a1c8-fad2-4b80-a0e9-65ab23c3612e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 13:21:25.876271 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:25.876217 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-db71f4/test-model-db71f4-kserve-6cf694c5f9-57cmd" podUID="029ef988-b71c-4de2-9b27-d7506a869ed8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded" Nov 03 13:21:25.890309 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:25.881675 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-db71f4/test-model-db71f4-kserve-6cf694c5f9-57cmd" Nov 03 13:21:25.930670 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:25.930322 5024 patch_prober.go:29] interesting pod/e2e-extmod-a-ded926-openshift-default-6bdf65b545-pjqfb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Nov 03 13:21:25.930670 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:25.930396 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-ded926-openshift-default-6bdf65b545-pjqfb" podUID="736b02c5-9833-4f1b-9f9a-e4c22ab23ac9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Nov 03 13:21:25.978117 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:25.978040 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-7c192b-65886b4cf7-n24wm" podStartSLOduration=5.089905841 podStartE2EDuration="5.978013453s" totalImagesPullingTime="888.107612ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:21:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:21:24.192704889 +0000 UTC m=+764.200896175" watchObservedRunningTime="2026-11-03 13:21:25.978013453 +0000 UTC m=+765.986204731" Nov 03 13:21:26.227146 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:26.227053 5024 patch_prober.go:29] interesting pod/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-wwvqr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Nov 03 13:21:26.250755 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:26.250710 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-wwvqr" podUID="fffd24a4-d55d-4661-a37d-c344d3343698" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Nov 03 13:21:26.751297 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:26.751245 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-db71f4/test-model-db71f4-kserve-6cf694c5f9-57cmd" Nov 03 13:21:26.853067 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:26.853018 5024 patch_prober.go:29] interesting pod/e2e-worker-w4-c381a7-openshift-default-78bcd4695-fglkk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 13:21:26.853426 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:26.853381 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-fglkk" podUID="e6d115b8-3b33-43c6-890b-a87bb608444b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 13:21:26.911667 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:26.911632 5024 patch_prober.go:29] interesting pod/e2e-extmod-a-ded926-openshift-default-6bdf65b545-pjqfb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Nov 03 13:21:26.959653 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:26.959610 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-ded926-openshift-default-6bdf65b545-pjqfb" podUID="736b02c5-9833-4f1b-9f9a-e4c22ab23ac9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Nov 03 13:21:27.217362 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:27.215276 5024 patch_prober.go:29] interesting pod/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-wwvqr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Nov 03 13:21:27.217662 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:27.217618 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-wwvqr" podUID="fffd24a4-d55d-4661-a37d-c344d3343698" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Nov 03 13:21:27.907859 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:27.907819 5024 patch_prober.go:29] interesting pod/e2e-extmod-a-ded926-openshift-default-6bdf65b545-pjqfb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Nov 03 13:21:27.909556 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:27.908510 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-ded926-openshift-default-6bdf65b545-pjqfb" podUID="736b02c5-9833-4f1b-9f9a-e4c22ab23ac9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Nov 03 13:21:28.643469 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:28.643436 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-wwvqr" Nov 03 13:21:28.643927 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:28.643869 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-wwvqr" Nov 03 13:21:28.645585 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:28.645561 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-wwvqr" Nov 03 13:21:28.903406 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:28.903310 5024 patch_prober.go:29] interesting pod/e2e-extmod-a-ded926-openshift-default-6bdf65b545-pjqfb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Nov 03 13:21:28.903671 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:28.903627 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-ded926-openshift-default-6bdf65b545-pjqfb" podUID="736b02c5-9833-4f1b-9f9a-e4c22ab23ac9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Nov 03 13:21:29.903425 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:29.903389 5024 patch_prober.go:29] interesting pod/e2e-extmod-a-ded926-openshift-default-6bdf65b545-pjqfb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Nov 03 13:21:29.903805 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:29.903446 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-ded926-openshift-default-6bdf65b545-pjqfb" podUID="736b02c5-9833-4f1b-9f9a-e4c22ab23ac9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Nov 03 13:21:30.907430 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:30.907395 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-ded926-openshift-default-6bdf65b545-pjqfb" Nov 03 13:21:30.908561 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:30.908538 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-ded926-openshift-default-6bdf65b545-pjqfb" Nov 03 13:21:31.635260 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:31.635228 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-7c192b-65886b4cf7-n24wm" Nov 03 13:21:38.588310 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:38.587463 5024 patch_prober.go:29] interesting pod/e2e-worker-w4-c381a7-openshift-default-78bcd4695-rddcl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 13:21:38.588310 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:38.587549 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-rddcl" podUID="77fc5206-8873-4bcc-89b1-fb637422962b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 13:21:39.532343 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:21:39.532301 5024 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d7mwz" containerName="extractor" Nov 03 13:21:46.432515 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:46.327095 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-a2ccd8-openshift-default-798b597bf7-5ljc2"] Nov 03 13:21:46.552909 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:46.552874 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-a2ccd8-openshift-default-798b597bf7-5ljc2"] Nov 03 13:21:46.553136 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:46.552997 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-a2ccd8-openshift-default-798b597bf7-5ljc2" Nov 03 13:21:46.560248 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:46.560168 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-a2ccd8-openshift-default-dockercfg-sc96f\"" Nov 03 13:21:46.716878 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:46.716794 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c08c13a6-4853-40b2-9496-ac50f59b1bf8-workload-certs\") pod \"e2e-praxis-a-a2ccd8-openshift-default-798b597bf7-5ljc2\" (UID: \"c08c13a6-4853-40b2-9496-ac50f59b1bf8\") " pod="openshift-ingress/e2e-praxis-a-a2ccd8-openshift-default-798b597bf7-5ljc2" Nov 03 13:21:46.716878 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:46.716848 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c08c13a6-4853-40b2-9496-ac50f59b1bf8-istio-podinfo\") pod \"e2e-praxis-a-a2ccd8-openshift-default-798b597bf7-5ljc2\" (UID: \"c08c13a6-4853-40b2-9496-ac50f59b1bf8\") " pod="openshift-ingress/e2e-praxis-a-a2ccd8-openshift-default-798b597bf7-5ljc2" Nov 03 13:21:46.717059 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:46.716949 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c08c13a6-4853-40b2-9496-ac50f59b1bf8-istio-data\") pod \"e2e-praxis-a-a2ccd8-openshift-default-798b597bf7-5ljc2\" (UID: \"c08c13a6-4853-40b2-9496-ac50f59b1bf8\") " pod="openshift-ingress/e2e-praxis-a-a2ccd8-openshift-default-798b597bf7-5ljc2" Nov 03 13:21:46.717059 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:46.717000 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c08c13a6-4853-40b2-9496-ac50f59b1bf8-workload-socket\") pod \"e2e-praxis-a-a2ccd8-openshift-default-798b597bf7-5ljc2\" (UID: \"c08c13a6-4853-40b2-9496-ac50f59b1bf8\") " pod="openshift-ingress/e2e-praxis-a-a2ccd8-openshift-default-798b597bf7-5ljc2" Nov 03 13:21:46.717059 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:46.717036 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l9q6t\" (UniqueName: \"kubernetes.io/projected/c08c13a6-4853-40b2-9496-ac50f59b1bf8-kube-api-access-l9q6t\") pod \"e2e-praxis-a-a2ccd8-openshift-default-798b597bf7-5ljc2\" (UID: \"c08c13a6-4853-40b2-9496-ac50f59b1bf8\") " pod="openshift-ingress/e2e-praxis-a-a2ccd8-openshift-default-798b597bf7-5ljc2" Nov 03 13:21:46.717216 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:46.717175 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c08c13a6-4853-40b2-9496-ac50f59b1bf8-istio-envoy\") pod \"e2e-praxis-a-a2ccd8-openshift-default-798b597bf7-5ljc2\" (UID: \"c08c13a6-4853-40b2-9496-ac50f59b1bf8\") " pod="openshift-ingress/e2e-praxis-a-a2ccd8-openshift-default-798b597bf7-5ljc2" Nov 03 13:21:46.717273 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:46.717240 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c08c13a6-4853-40b2-9496-ac50f59b1bf8-istiod-ca-cert\") pod \"e2e-praxis-a-a2ccd8-openshift-default-798b597bf7-5ljc2\" (UID: \"c08c13a6-4853-40b2-9496-ac50f59b1bf8\") " pod="openshift-ingress/e2e-praxis-a-a2ccd8-openshift-default-798b597bf7-5ljc2" Nov 03 13:21:46.717336 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:46.717310 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c08c13a6-4853-40b2-9496-ac50f59b1bf8-istio-token\") pod \"e2e-praxis-a-a2ccd8-openshift-default-798b597bf7-5ljc2\" (UID: \"c08c13a6-4853-40b2-9496-ac50f59b1bf8\") " pod="openshift-ingress/e2e-praxis-a-a2ccd8-openshift-default-798b597bf7-5ljc2" Nov 03 13:21:46.717374 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:46.717334 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c08c13a6-4853-40b2-9496-ac50f59b1bf8-credential-socket\") pod \"e2e-praxis-a-a2ccd8-openshift-default-798b597bf7-5ljc2\" (UID: \"c08c13a6-4853-40b2-9496-ac50f59b1bf8\") " pod="openshift-ingress/e2e-praxis-a-a2ccd8-openshift-default-798b597bf7-5ljc2" Nov 03 13:21:46.822308 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:46.822021 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c08c13a6-4853-40b2-9496-ac50f59b1bf8-workload-certs\") pod \"e2e-praxis-a-a2ccd8-openshift-default-798b597bf7-5ljc2\" (UID: \"c08c13a6-4853-40b2-9496-ac50f59b1bf8\") " pod="openshift-ingress/e2e-praxis-a-a2ccd8-openshift-default-798b597bf7-5ljc2" Nov 03 13:21:46.822308 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:46.822120 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c08c13a6-4853-40b2-9496-ac50f59b1bf8-istio-podinfo\") pod \"e2e-praxis-a-a2ccd8-openshift-default-798b597bf7-5ljc2\" (UID: \"c08c13a6-4853-40b2-9496-ac50f59b1bf8\") " pod="openshift-ingress/e2e-praxis-a-a2ccd8-openshift-default-798b597bf7-5ljc2" Nov 03 13:21:46.822308 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:46.822162 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c08c13a6-4853-40b2-9496-ac50f59b1bf8-istio-data\") pod \"e2e-praxis-a-a2ccd8-openshift-default-798b597bf7-5ljc2\" (UID: \"c08c13a6-4853-40b2-9496-ac50f59b1bf8\") " pod="openshift-ingress/e2e-praxis-a-a2ccd8-openshift-default-798b597bf7-5ljc2" Nov 03 13:21:46.822308 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:46.822206 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c08c13a6-4853-40b2-9496-ac50f59b1bf8-workload-socket\") pod \"e2e-praxis-a-a2ccd8-openshift-default-798b597bf7-5ljc2\" (UID: \"c08c13a6-4853-40b2-9496-ac50f59b1bf8\") " pod="openshift-ingress/e2e-praxis-a-a2ccd8-openshift-default-798b597bf7-5ljc2" Nov 03 13:21:46.822308 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:46.822244 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9q6t\" (UniqueName: \"kubernetes.io/projected/c08c13a6-4853-40b2-9496-ac50f59b1bf8-kube-api-access-l9q6t\") pod \"e2e-praxis-a-a2ccd8-openshift-default-798b597bf7-5ljc2\" (UID: \"c08c13a6-4853-40b2-9496-ac50f59b1bf8\") " pod="openshift-ingress/e2e-praxis-a-a2ccd8-openshift-default-798b597bf7-5ljc2" Nov 03 13:21:46.822308 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:46.822301 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c08c13a6-4853-40b2-9496-ac50f59b1bf8-istio-envoy\") pod \"e2e-praxis-a-a2ccd8-openshift-default-798b597bf7-5ljc2\" (UID: \"c08c13a6-4853-40b2-9496-ac50f59b1bf8\") " pod="openshift-ingress/e2e-praxis-a-a2ccd8-openshift-default-798b597bf7-5ljc2" Nov 03 13:21:46.822810 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:46.822363 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c08c13a6-4853-40b2-9496-ac50f59b1bf8-istiod-ca-cert\") pod \"e2e-praxis-a-a2ccd8-openshift-default-798b597bf7-5ljc2\" (UID: \"c08c13a6-4853-40b2-9496-ac50f59b1bf8\") " pod="openshift-ingress/e2e-praxis-a-a2ccd8-openshift-default-798b597bf7-5ljc2" Nov 03 13:21:46.822810 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:46.822432 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c08c13a6-4853-40b2-9496-ac50f59b1bf8-istio-token\") pod \"e2e-praxis-a-a2ccd8-openshift-default-798b597bf7-5ljc2\" (UID: \"c08c13a6-4853-40b2-9496-ac50f59b1bf8\") " pod="openshift-ingress/e2e-praxis-a-a2ccd8-openshift-default-798b597bf7-5ljc2" Nov 03 13:21:46.822810 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:46.822474 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c08c13a6-4853-40b2-9496-ac50f59b1bf8-credential-socket\") pod \"e2e-praxis-a-a2ccd8-openshift-default-798b597bf7-5ljc2\" (UID: \"c08c13a6-4853-40b2-9496-ac50f59b1bf8\") " pod="openshift-ingress/e2e-praxis-a-a2ccd8-openshift-default-798b597bf7-5ljc2" Nov 03 13:21:46.822810 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:46.822690 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c08c13a6-4853-40b2-9496-ac50f59b1bf8-istio-data\") pod \"e2e-praxis-a-a2ccd8-openshift-default-798b597bf7-5ljc2\" (UID: \"c08c13a6-4853-40b2-9496-ac50f59b1bf8\") " pod="openshift-ingress/e2e-praxis-a-a2ccd8-openshift-default-798b597bf7-5ljc2" Nov 03 13:21:46.826334 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:46.823056 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c08c13a6-4853-40b2-9496-ac50f59b1bf8-credential-socket\") pod \"e2e-praxis-a-a2ccd8-openshift-default-798b597bf7-5ljc2\" (UID: \"c08c13a6-4853-40b2-9496-ac50f59b1bf8\") " pod="openshift-ingress/e2e-praxis-a-a2ccd8-openshift-default-798b597bf7-5ljc2" Nov 03 13:21:46.826334 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:46.823339 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c08c13a6-4853-40b2-9496-ac50f59b1bf8-workload-socket\") pod \"e2e-praxis-a-a2ccd8-openshift-default-798b597bf7-5ljc2\" (UID: \"c08c13a6-4853-40b2-9496-ac50f59b1bf8\") " pod="openshift-ingress/e2e-praxis-a-a2ccd8-openshift-default-798b597bf7-5ljc2" Nov 03 13:21:46.826334 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:46.824657 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c08c13a6-4853-40b2-9496-ac50f59b1bf8-istiod-ca-cert\") pod \"e2e-praxis-a-a2ccd8-openshift-default-798b597bf7-5ljc2\" (UID: \"c08c13a6-4853-40b2-9496-ac50f59b1bf8\") " pod="openshift-ingress/e2e-praxis-a-a2ccd8-openshift-default-798b597bf7-5ljc2" Nov 03 13:21:46.840748 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:46.829101 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c08c13a6-4853-40b2-9496-ac50f59b1bf8-istio-envoy\") pod \"e2e-praxis-a-a2ccd8-openshift-default-798b597bf7-5ljc2\" (UID: \"c08c13a6-4853-40b2-9496-ac50f59b1bf8\") " pod="openshift-ingress/e2e-praxis-a-a2ccd8-openshift-default-798b597bf7-5ljc2" Nov 03 13:21:46.840986 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:46.838033 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c08c13a6-4853-40b2-9496-ac50f59b1bf8-workload-certs\") pod \"e2e-praxis-a-a2ccd8-openshift-default-798b597bf7-5ljc2\" (UID: \"c08c13a6-4853-40b2-9496-ac50f59b1bf8\") " pod="openshift-ingress/e2e-praxis-a-a2ccd8-openshift-default-798b597bf7-5ljc2" Nov 03 13:21:46.847253 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:46.847214 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l9q6t\" (UniqueName: \"kubernetes.io/projected/c08c13a6-4853-40b2-9496-ac50f59b1bf8-kube-api-access-l9q6t\") pod \"e2e-praxis-a-a2ccd8-openshift-default-798b597bf7-5ljc2\" (UID: \"c08c13a6-4853-40b2-9496-ac50f59b1bf8\") " pod="openshift-ingress/e2e-praxis-a-a2ccd8-openshift-default-798b597bf7-5ljc2" Nov 03 13:21:46.856728 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:46.856696 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c08c13a6-4853-40b2-9496-ac50f59b1bf8-istio-podinfo\") pod \"e2e-praxis-a-a2ccd8-openshift-default-798b597bf7-5ljc2\" (UID: \"c08c13a6-4853-40b2-9496-ac50f59b1bf8\") " pod="openshift-ingress/e2e-praxis-a-a2ccd8-openshift-default-798b597bf7-5ljc2" Nov 03 13:21:46.871349 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:46.870040 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c08c13a6-4853-40b2-9496-ac50f59b1bf8-istio-token\") pod \"e2e-praxis-a-a2ccd8-openshift-default-798b597bf7-5ljc2\" (UID: \"c08c13a6-4853-40b2-9496-ac50f59b1bf8\") " pod="openshift-ingress/e2e-praxis-a-a2ccd8-openshift-default-798b597bf7-5ljc2" Nov 03 13:21:46.927306 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:46.923512 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-a2ccd8-openshift-default-798b597bf7-5ljc2" Nov 03 13:21:47.250323 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:47.250293 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-a2ccd8-openshift-default-798b597bf7-5ljc2"] Nov 03 13:21:47.250848 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:21:47.250824 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc08c13a6_4853_40b2_9496_ac50f59b1bf8.slice/crio-8fe4ccbbaee08675f98957a9fcf7cde2108c8e10ce0b4ec320b0dad2aae41786 WatchSource:0}: Error finding container 8fe4ccbbaee08675f98957a9fcf7cde2108c8e10ce0b4ec320b0dad2aae41786: Status 404 returned error can't find the container with id 8fe4ccbbaee08675f98957a9fcf7cde2108c8e10ce0b4ec320b0dad2aae41786 Nov 03 13:21:47.253131 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:47.253099 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:21:47.253225 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:47.253175 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:21:47.253317 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:47.253223 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:21:48.122438 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:48.122400 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-a2ccd8-openshift-default-798b597bf7-5ljc2" event={"ID":"c08c13a6-4853-40b2-9496-ac50f59b1bf8","Type":"ContainerStarted","Data":"13962ec05066f1acad8d5e943e5ed0f14c8b9d9fa9c29908e2223765a48932d4"} Nov 03 13:21:48.122804 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:48.122464 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-a2ccd8-openshift-default-798b597bf7-5ljc2" event={"ID":"c08c13a6-4853-40b2-9496-ac50f59b1bf8","Type":"ContainerStarted","Data":"8fe4ccbbaee08675f98957a9fcf7cde2108c8e10ce0b4ec320b0dad2aae41786"} Nov 03 13:21:48.139689 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:48.139639 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-a2ccd8-openshift-default-798b597bf7-5ljc2" podStartSLOduration=2.139625443 podStartE2EDuration="2.139625443s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:21:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:21:48.137317304 +0000 UTC m=+788.145508580" watchObservedRunningTime="2026-11-03 13:21:48.139625443 +0000 UTC m=+788.147816719" Nov 03 13:21:48.925761 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:48.925720 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-a2ccd8-openshift-default-798b597bf7-5ljc2" Nov 03 13:21:48.931499 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:48.931470 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-a2ccd8-openshift-default-798b597bf7-5ljc2" Nov 03 13:21:49.121952 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:49.121918 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-a2ccd8-openshift-default-798b597bf7-5ljc2" Nov 03 13:21:49.123003 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:49.122983 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-a2ccd8-openshift-default-798b597bf7-5ljc2" Nov 03 13:21:51.755306 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:51.730399 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-db71f4/test-model-db71f4-kserve-6cf694c5f9-57cmd" podUID="029ef988-b71c-4de2-9b27-d7506a869ed8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded" Nov 03 13:21:54.245207 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:54.245167 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-zzmwf"] Nov 03 13:21:54.321071 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:54.302506 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-zzmwf"] Nov 03 13:21:54.321071 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:54.302648 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-zzmwf" Nov 03 13:21:54.367925 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:54.367889 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e65de701-9327-4af6-a7f7-e0d6adf6ffcd-workload-socket\") pod \"e2e-extmod-b-7c192b-openshift-default-5c496b978f-zzmwf\" (UID: \"e65de701-9327-4af6-a7f7-e0d6adf6ffcd\") " pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-zzmwf" Nov 03 13:21:54.368344 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:54.368272 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e65de701-9327-4af6-a7f7-e0d6adf6ffcd-istio-data\") pod \"e2e-extmod-b-7c192b-openshift-default-5c496b978f-zzmwf\" (UID: \"e65de701-9327-4af6-a7f7-e0d6adf6ffcd\") " pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-zzmwf" Nov 03 13:21:54.368509 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:54.368492 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e65de701-9327-4af6-a7f7-e0d6adf6ffcd-istio-podinfo\") pod \"e2e-extmod-b-7c192b-openshift-default-5c496b978f-zzmwf\" (UID: \"e65de701-9327-4af6-a7f7-e0d6adf6ffcd\") " pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-zzmwf" Nov 03 13:21:54.368628 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:54.368611 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e65de701-9327-4af6-a7f7-e0d6adf6ffcd-credential-socket\") pod \"e2e-extmod-b-7c192b-openshift-default-5c496b978f-zzmwf\" (UID: \"e65de701-9327-4af6-a7f7-e0d6adf6ffcd\") " pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-zzmwf" Nov 03 13:21:54.369207 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:54.369185 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e65de701-9327-4af6-a7f7-e0d6adf6ffcd-istiod-ca-cert\") pod \"e2e-extmod-b-7c192b-openshift-default-5c496b978f-zzmwf\" (UID: \"e65de701-9327-4af6-a7f7-e0d6adf6ffcd\") " pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-zzmwf" Nov 03 13:21:54.369426 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:54.369406 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e65de701-9327-4af6-a7f7-e0d6adf6ffcd-istio-envoy\") pod \"e2e-extmod-b-7c192b-openshift-default-5c496b978f-zzmwf\" (UID: \"e65de701-9327-4af6-a7f7-e0d6adf6ffcd\") " pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-zzmwf" Nov 03 13:21:54.369566 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:54.369551 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e65de701-9327-4af6-a7f7-e0d6adf6ffcd-workload-certs\") pod \"e2e-extmod-b-7c192b-openshift-default-5c496b978f-zzmwf\" (UID: \"e65de701-9327-4af6-a7f7-e0d6adf6ffcd\") " pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-zzmwf" Nov 03 13:21:54.369679 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:54.369664 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cdzt6\" (UniqueName: \"kubernetes.io/projected/e65de701-9327-4af6-a7f7-e0d6adf6ffcd-kube-api-access-cdzt6\") pod \"e2e-extmod-b-7c192b-openshift-default-5c496b978f-zzmwf\" (UID: \"e65de701-9327-4af6-a7f7-e0d6adf6ffcd\") " pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-zzmwf" Nov 03 13:21:54.369816 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:54.369802 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e65de701-9327-4af6-a7f7-e0d6adf6ffcd-istio-token\") pod \"e2e-extmod-b-7c192b-openshift-default-5c496b978f-zzmwf\" (UID: \"e65de701-9327-4af6-a7f7-e0d6adf6ffcd\") " pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-zzmwf" Nov 03 13:21:54.471592 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:54.471559 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e65de701-9327-4af6-a7f7-e0d6adf6ffcd-workload-socket\") pod \"e2e-extmod-b-7c192b-openshift-default-5c496b978f-zzmwf\" (UID: \"e65de701-9327-4af6-a7f7-e0d6adf6ffcd\") " pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-zzmwf" Nov 03 13:21:54.471782 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:54.471680 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e65de701-9327-4af6-a7f7-e0d6adf6ffcd-istio-data\") pod \"e2e-extmod-b-7c192b-openshift-default-5c496b978f-zzmwf\" (UID: \"e65de701-9327-4af6-a7f7-e0d6adf6ffcd\") " pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-zzmwf" Nov 03 13:21:54.471782 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:54.471699 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e65de701-9327-4af6-a7f7-e0d6adf6ffcd-istio-podinfo\") pod \"e2e-extmod-b-7c192b-openshift-default-5c496b978f-zzmwf\" (UID: \"e65de701-9327-4af6-a7f7-e0d6adf6ffcd\") " pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-zzmwf" Nov 03 13:21:54.471782 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:54.471719 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e65de701-9327-4af6-a7f7-e0d6adf6ffcd-credential-socket\") pod \"e2e-extmod-b-7c192b-openshift-default-5c496b978f-zzmwf\" (UID: \"e65de701-9327-4af6-a7f7-e0d6adf6ffcd\") " pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-zzmwf" Nov 03 13:21:54.471782 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:54.471743 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e65de701-9327-4af6-a7f7-e0d6adf6ffcd-istiod-ca-cert\") pod \"e2e-extmod-b-7c192b-openshift-default-5c496b978f-zzmwf\" (UID: \"e65de701-9327-4af6-a7f7-e0d6adf6ffcd\") " pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-zzmwf" Nov 03 13:21:54.471782 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:54.471774 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e65de701-9327-4af6-a7f7-e0d6adf6ffcd-istio-envoy\") pod \"e2e-extmod-b-7c192b-openshift-default-5c496b978f-zzmwf\" (UID: \"e65de701-9327-4af6-a7f7-e0d6adf6ffcd\") " pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-zzmwf" Nov 03 13:21:54.472055 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:54.471808 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e65de701-9327-4af6-a7f7-e0d6adf6ffcd-workload-certs\") pod \"e2e-extmod-b-7c192b-openshift-default-5c496b978f-zzmwf\" (UID: \"e65de701-9327-4af6-a7f7-e0d6adf6ffcd\") " pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-zzmwf" Nov 03 13:21:54.472055 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:54.471837 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cdzt6\" (UniqueName: \"kubernetes.io/projected/e65de701-9327-4af6-a7f7-e0d6adf6ffcd-kube-api-access-cdzt6\") pod \"e2e-extmod-b-7c192b-openshift-default-5c496b978f-zzmwf\" (UID: \"e65de701-9327-4af6-a7f7-e0d6adf6ffcd\") " pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-zzmwf" Nov 03 13:21:54.472055 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:54.471890 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e65de701-9327-4af6-a7f7-e0d6adf6ffcd-istio-token\") pod \"e2e-extmod-b-7c192b-openshift-default-5c496b978f-zzmwf\" (UID: \"e65de701-9327-4af6-a7f7-e0d6adf6ffcd\") " pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-zzmwf" Nov 03 13:21:54.472055 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:54.471995 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e65de701-9327-4af6-a7f7-e0d6adf6ffcd-workload-socket\") pod \"e2e-extmod-b-7c192b-openshift-default-5c496b978f-zzmwf\" (UID: \"e65de701-9327-4af6-a7f7-e0d6adf6ffcd\") " pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-zzmwf" Nov 03 13:21:54.472055 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:54.472041 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e65de701-9327-4af6-a7f7-e0d6adf6ffcd-istio-data\") pod \"e2e-extmod-b-7c192b-openshift-default-5c496b978f-zzmwf\" (UID: \"e65de701-9327-4af6-a7f7-e0d6adf6ffcd\") " pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-zzmwf" Nov 03 13:21:54.472382 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:54.472301 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e65de701-9327-4af6-a7f7-e0d6adf6ffcd-workload-certs\") pod \"e2e-extmod-b-7c192b-openshift-default-5c496b978f-zzmwf\" (UID: \"e65de701-9327-4af6-a7f7-e0d6adf6ffcd\") " pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-zzmwf" Nov 03 13:21:54.472382 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:54.472331 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e65de701-9327-4af6-a7f7-e0d6adf6ffcd-credential-socket\") pod \"e2e-extmod-b-7c192b-openshift-default-5c496b978f-zzmwf\" (UID: \"e65de701-9327-4af6-a7f7-e0d6adf6ffcd\") " pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-zzmwf" Nov 03 13:21:54.472894 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:54.472864 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e65de701-9327-4af6-a7f7-e0d6adf6ffcd-istiod-ca-cert\") pod \"e2e-extmod-b-7c192b-openshift-default-5c496b978f-zzmwf\" (UID: \"e65de701-9327-4af6-a7f7-e0d6adf6ffcd\") " pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-zzmwf" Nov 03 13:21:54.476549 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:54.476511 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e65de701-9327-4af6-a7f7-e0d6adf6ffcd-istio-envoy\") pod \"e2e-extmod-b-7c192b-openshift-default-5c496b978f-zzmwf\" (UID: \"e65de701-9327-4af6-a7f7-e0d6adf6ffcd\") " pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-zzmwf" Nov 03 13:21:54.527349 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:54.524630 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e65de701-9327-4af6-a7f7-e0d6adf6ffcd-istio-podinfo\") pod \"e2e-extmod-b-7c192b-openshift-default-5c496b978f-zzmwf\" (UID: \"e65de701-9327-4af6-a7f7-e0d6adf6ffcd\") " pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-zzmwf" Nov 03 13:21:54.532041 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:54.532006 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e65de701-9327-4af6-a7f7-e0d6adf6ffcd-istio-token\") pod \"e2e-extmod-b-7c192b-openshift-default-5c496b978f-zzmwf\" (UID: \"e65de701-9327-4af6-a7f7-e0d6adf6ffcd\") " pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-zzmwf" Nov 03 13:21:54.567816 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:54.567748 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cdzt6\" (UniqueName: \"kubernetes.io/projected/e65de701-9327-4af6-a7f7-e0d6adf6ffcd-kube-api-access-cdzt6\") pod \"e2e-extmod-b-7c192b-openshift-default-5c496b978f-zzmwf\" (UID: \"e65de701-9327-4af6-a7f7-e0d6adf6ffcd\") " pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-zzmwf" Nov 03 13:21:54.649449 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:54.649403 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-zzmwf" Nov 03 13:21:54.940168 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:54.940139 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-zzmwf"] Nov 03 13:21:54.941338 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:21:54.941304 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode65de701_9327_4af6_a7f7_e0d6adf6ffcd.slice/crio-8b4aa5d2e99cb6411b5939e6fd157f5262bc7e1d5ed8b15e3ee684c25a19a950 WatchSource:0}: Error finding container 8b4aa5d2e99cb6411b5939e6fd157f5262bc7e1d5ed8b15e3ee684c25a19a950: Status 404 returned error can't find the container with id 8b4aa5d2e99cb6411b5939e6fd157f5262bc7e1d5ed8b15e3ee684c25a19a950 Nov 03 13:21:54.944038 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:54.944002 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:21:54.944149 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:54.944076 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:21:54.944149 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:54.944109 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:21:55.559402 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:55.559355 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-zzmwf" event={"ID":"e65de701-9327-4af6-a7f7-e0d6adf6ffcd","Type":"ContainerStarted","Data":"e3ff084f3e19c0fb25357ff26d959383a28737282564937224b3eb7cb3ef14d6"} Nov 03 13:21:55.559825 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:55.559424 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-zzmwf" event={"ID":"e65de701-9327-4af6-a7f7-e0d6adf6ffcd","Type":"ContainerStarted","Data":"8b4aa5d2e99cb6411b5939e6fd157f5262bc7e1d5ed8b15e3ee684c25a19a950"} Nov 03 13:21:55.650147 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:55.650104 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-zzmwf" Nov 03 13:21:56.651123 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:56.651090 5024 patch_prober.go:29] interesting pod/e2e-extmod-b-7c192b-openshift-default-5c496b978f-zzmwf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Nov 03 13:21:56.651681 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:56.651650 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-zzmwf" podUID="e65de701-9327-4af6-a7f7-e0d6adf6ffcd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Nov 03 13:21:57.224549 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:57.224475 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-zzmwf" podStartSLOduration=3.224455009 podStartE2EDuration="3.224455009s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:21:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:21:55.595664727 +0000 UTC m=+795.603856006" watchObservedRunningTime="2026-11-03 13:21:57.224455009 +0000 UTC m=+797.232646289" Nov 03 13:21:57.228635 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:57.228595 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-a2ccd8-6bfb4ddf8f-vqss2"] Nov 03 13:21:57.234679 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:57.234657 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-a2ccd8-6bfb4ddf8f-vqss2" Nov 03 13:21:57.242276 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:57.242241 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-a2ccd8-serving-cert\"" Nov 03 13:21:57.242673 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:57.242650 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-a2ccd8-6bfb4ddf8f-vqss2"] Nov 03 13:21:57.323928 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:57.323893 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/41f9ddad-1f63-4f31-86e4-831fd43b8f44-metrics-certs\") pod \"maas-api-e2e-praxis-a-a2ccd8-6bfb4ddf8f-vqss2\" (UID: \"41f9ddad-1f63-4f31-86e4-831fd43b8f44\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-a2ccd8-6bfb4ddf8f-vqss2" Nov 03 13:21:57.324100 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:57.323963 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/41f9ddad-1f63-4f31-86e4-831fd43b8f44-maas-api-tls\") pod \"maas-api-e2e-praxis-a-a2ccd8-6bfb4ddf8f-vqss2\" (UID: \"41f9ddad-1f63-4f31-86e4-831fd43b8f44\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-a2ccd8-6bfb4ddf8f-vqss2" Nov 03 13:21:57.324100 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:57.324062 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ggxr8\" (UniqueName: \"kubernetes.io/projected/41f9ddad-1f63-4f31-86e4-831fd43b8f44-kube-api-access-ggxr8\") pod \"maas-api-e2e-praxis-a-a2ccd8-6bfb4ddf8f-vqss2\" (UID: \"41f9ddad-1f63-4f31-86e4-831fd43b8f44\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-a2ccd8-6bfb4ddf8f-vqss2" Nov 03 13:21:57.324179 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:57.324114 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/41f9ddad-1f63-4f31-86e4-831fd43b8f44-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-a2ccd8-6bfb4ddf8f-vqss2\" (UID: \"41f9ddad-1f63-4f31-86e4-831fd43b8f44\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-a2ccd8-6bfb4ddf8f-vqss2" Nov 03 13:21:57.492396 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:57.426089 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/41f9ddad-1f63-4f31-86e4-831fd43b8f44-metrics-certs\") pod \"maas-api-e2e-praxis-a-a2ccd8-6bfb4ddf8f-vqss2\" (UID: \"41f9ddad-1f63-4f31-86e4-831fd43b8f44\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-a2ccd8-6bfb4ddf8f-vqss2" Nov 03 13:21:57.492396 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:57.426201 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/41f9ddad-1f63-4f31-86e4-831fd43b8f44-maas-api-tls\") pod \"maas-api-e2e-praxis-a-a2ccd8-6bfb4ddf8f-vqss2\" (UID: \"41f9ddad-1f63-4f31-86e4-831fd43b8f44\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-a2ccd8-6bfb4ddf8f-vqss2" Nov 03 13:21:57.492396 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:57.426231 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ggxr8\" (UniqueName: \"kubernetes.io/projected/41f9ddad-1f63-4f31-86e4-831fd43b8f44-kube-api-access-ggxr8\") pod \"maas-api-e2e-praxis-a-a2ccd8-6bfb4ddf8f-vqss2\" (UID: \"41f9ddad-1f63-4f31-86e4-831fd43b8f44\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-a2ccd8-6bfb4ddf8f-vqss2" Nov 03 13:21:57.492396 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:57.426268 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/41f9ddad-1f63-4f31-86e4-831fd43b8f44-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-a2ccd8-6bfb4ddf8f-vqss2\" (UID: \"41f9ddad-1f63-4f31-86e4-831fd43b8f44\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-a2ccd8-6bfb4ddf8f-vqss2" Nov 03 13:21:57.492396 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:21:57.439063 5024 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-a-a2ccd8-serving-cert: secret "maas-api-e2e-praxis-a-a2ccd8-serving-cert" not found Nov 03 13:21:57.492396 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:21:57.439141 5024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/41f9ddad-1f63-4f31-86e4-831fd43b8f44-maas-api-tls podName:41f9ddad-1f63-4f31-86e4-831fd43b8f44 nodeName:}" failed. No retries permitted until 2026-11-03 13:21:57.939120731 +0000 UTC m=+797.947311988 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/41f9ddad-1f63-4f31-86e4-831fd43b8f44-maas-api-tls") pod "maas-api-e2e-praxis-a-a2ccd8-6bfb4ddf8f-vqss2" (UID: "41f9ddad-1f63-4f31-86e4-831fd43b8f44") : secret "maas-api-e2e-praxis-a-a2ccd8-serving-cert" not found Nov 03 13:21:57.492396 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:57.443423 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/41f9ddad-1f63-4f31-86e4-831fd43b8f44-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-a2ccd8-6bfb4ddf8f-vqss2\" (UID: \"41f9ddad-1f63-4f31-86e4-831fd43b8f44\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-a2ccd8-6bfb4ddf8f-vqss2" Nov 03 13:21:57.492396 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:57.448826 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/41f9ddad-1f63-4f31-86e4-831fd43b8f44-metrics-certs\") pod \"maas-api-e2e-praxis-a-a2ccd8-6bfb4ddf8f-vqss2\" (UID: \"41f9ddad-1f63-4f31-86e4-831fd43b8f44\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-a2ccd8-6bfb4ddf8f-vqss2" Nov 03 13:21:57.517240 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:57.517206 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ggxr8\" (UniqueName: \"kubernetes.io/projected/41f9ddad-1f63-4f31-86e4-831fd43b8f44-kube-api-access-ggxr8\") pod \"maas-api-e2e-praxis-a-a2ccd8-6bfb4ddf8f-vqss2\" (UID: \"41f9ddad-1f63-4f31-86e4-831fd43b8f44\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-a2ccd8-6bfb4ddf8f-vqss2" Nov 03 13:21:57.650322 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:57.650268 5024 patch_prober.go:29] interesting pod/e2e-extmod-b-7c192b-openshift-default-5c496b978f-zzmwf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Nov 03 13:21:57.650492 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:57.650347 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-zzmwf" podUID="e65de701-9327-4af6-a7f7-e0d6adf6ffcd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Nov 03 13:21:57.952323 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:57.951705 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/41f9ddad-1f63-4f31-86e4-831fd43b8f44-maas-api-tls\") pod \"maas-api-e2e-praxis-a-a2ccd8-6bfb4ddf8f-vqss2\" (UID: \"41f9ddad-1f63-4f31-86e4-831fd43b8f44\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-a2ccd8-6bfb4ddf8f-vqss2" Nov 03 13:21:57.975315 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:57.970386 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/41f9ddad-1f63-4f31-86e4-831fd43b8f44-maas-api-tls\") pod \"maas-api-e2e-praxis-a-a2ccd8-6bfb4ddf8f-vqss2\" (UID: \"41f9ddad-1f63-4f31-86e4-831fd43b8f44\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-a2ccd8-6bfb4ddf8f-vqss2" Nov 03 13:21:58.163527 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:58.163485 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-a2ccd8-6bfb4ddf8f-vqss2" Nov 03 13:21:58.649746 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:58.649718 5024 patch_prober.go:29] interesting pod/e2e-extmod-b-7c192b-openshift-default-5c496b978f-zzmwf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Nov 03 13:21:58.649908 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:58.649813 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-zzmwf" podUID="e65de701-9327-4af6-a7f7-e0d6adf6ffcd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Nov 03 13:21:58.665195 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:58.665174 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-a2ccd8-6bfb4ddf8f-vqss2"] Nov 03 13:21:58.727455 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:58.727426 5024 patch_prober.go:29] interesting pod/e2e-extmod-b-7c192b-openshift-default-5c496b978f-wkz2r container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body= Nov 03 13:21:58.727594 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:58.727474 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-wkz2r" podUID="c49c4c7b-3182-4699-898a-5407d9a3fea1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" Nov 03 13:21:59.635084 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:59.635032 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-a2ccd8-6bfb4ddf8f-vqss2" event={"ID":"41f9ddad-1f63-4f31-86e4-831fd43b8f44","Type":"ContainerStarted","Data":"20fbeb4a902e176d63d2d0df1e0dac6336c8f5ae18aaaecc9c4f0489d5020cf9"} Nov 03 13:21:59.635584 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:59.635110 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-a2ccd8-6bfb4ddf8f-vqss2" Nov 03 13:21:59.635584 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:59.635142 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-a2ccd8-6bfb4ddf8f-vqss2" event={"ID":"41f9ddad-1f63-4f31-86e4-831fd43b8f44","Type":"ContainerStarted","Data":"40055e76b30d95d034de48082ba72a552b97e76f15c343eb66571614357f6346"} Nov 03 13:21:59.638238 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:59.638214 5024 generic.go:356] "Generic (PLEG): container finished" podID="c49c4c7b-3182-4699-898a-5407d9a3fea1" containerID="fed93113530f4bcfe3c23ec37252c5c1ca8e3331bd12050e56155ad271ef7d79" exitCode=0 Nov 03 13:21:59.638380 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:59.638294 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-wkz2r" event={"ID":"c49c4c7b-3182-4699-898a-5407d9a3fea1","Type":"ContainerDied","Data":"fed93113530f4bcfe3c23ec37252c5c1ca8e3331bd12050e56155ad271ef7d79"} Nov 03 13:21:59.638685 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:59.638670 5024 scope.go:122] "RemoveContainer" containerID="fed93113530f4bcfe3c23ec37252c5c1ca8e3331bd12050e56155ad271ef7d79" Nov 03 13:21:59.640115 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:59.640087 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:21:59.640203 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:59.640154 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:21:59.640203 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:59.640185 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:21:59.649681 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:59.649655 5024 patch_prober.go:29] interesting pod/e2e-extmod-b-7c192b-openshift-default-5c496b978f-zzmwf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Nov 03 13:21:59.649762 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:59.649707 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-zzmwf" podUID="e65de701-9327-4af6-a7f7-e0d6adf6ffcd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Nov 03 13:21:59.657177 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:21:59.657136 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-a2ccd8-6bfb4ddf8f-vqss2" podStartSLOduration=2.149775869 podStartE2EDuration="2.657125727s" totalImagesPullingTime="507.349858ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:21:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:21:59.654094222 +0000 UTC m=+799.662285624" watchObservedRunningTime="2026-11-03 13:21:59.657125727 +0000 UTC m=+799.665317015" Nov 03 13:22:00.653332 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:00.652514 5024 patch_prober.go:29] interesting pod/e2e-extmod-b-7c192b-openshift-default-5c496b978f-zzmwf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Nov 03 13:22:00.653332 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:00.652572 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-zzmwf" podUID="e65de701-9327-4af6-a7f7-e0d6adf6ffcd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Nov 03 13:22:00.665244 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:00.664331 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-wkz2r" event={"ID":"c49c4c7b-3182-4699-898a-5407d9a3fea1","Type":"ContainerStarted","Data":"a83ddae672e505fbb2fd4641a2fc8481b8c2a939713bcb369cde5647ec1eaae5"} Nov 03 13:22:01.164010 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:01.163972 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-wkz2r" Nov 03 13:22:01.164208 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:01.164169 5024 patch_prober.go:29] interesting pod/e2e-extmod-b-7c192b-openshift-default-5c496b978f-wkz2r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body= Nov 03 13:22:01.164280 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:01.164238 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-wkz2r" podUID="c49c4c7b-3182-4699-898a-5407d9a3fea1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" Nov 03 13:22:01.654483 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:01.654442 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-zzmwf" Nov 03 13:22:01.654988 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:01.654819 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-zzmwf" Nov 03 13:22:01.655743 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:01.655722 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-zzmwf" Nov 03 13:22:01.759068 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:01.759037 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-n4t7b"] Nov 03 13:22:01.766241 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:01.766199 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-n4t7b" Nov 03 13:22:01.769816 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:01.769725 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-n4t7b"] Nov 03 13:22:01.807048 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:01.807014 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0da6e9d5-8520-418d-bbd6-525f6adfa582-istio-token\") pod \"e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-n4t7b\" (UID: \"0da6e9d5-8520-418d-bbd6-525f6adfa582\") " pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-n4t7b" Nov 03 13:22:01.807347 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:01.807323 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0da6e9d5-8520-418d-bbd6-525f6adfa582-workload-socket\") pod \"e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-n4t7b\" (UID: \"0da6e9d5-8520-418d-bbd6-525f6adfa582\") " pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-n4t7b" Nov 03 13:22:01.807572 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:01.807538 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0da6e9d5-8520-418d-bbd6-525f6adfa582-istio-podinfo\") pod \"e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-n4t7b\" (UID: \"0da6e9d5-8520-418d-bbd6-525f6adfa582\") " pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-n4t7b" Nov 03 13:22:01.807686 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:01.807592 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0da6e9d5-8520-418d-bbd6-525f6adfa582-credential-socket\") pod \"e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-n4t7b\" (UID: \"0da6e9d5-8520-418d-bbd6-525f6adfa582\") " pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-n4t7b" Nov 03 13:22:01.807751 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:01.807702 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rptbx\" (UniqueName: \"kubernetes.io/projected/0da6e9d5-8520-418d-bbd6-525f6adfa582-kube-api-access-rptbx\") pod \"e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-n4t7b\" (UID: \"0da6e9d5-8520-418d-bbd6-525f6adfa582\") " pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-n4t7b" Nov 03 13:22:01.807815 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:01.807781 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0da6e9d5-8520-418d-bbd6-525f6adfa582-workload-certs\") pod \"e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-n4t7b\" (UID: \"0da6e9d5-8520-418d-bbd6-525f6adfa582\") " pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-n4t7b" Nov 03 13:22:01.807876 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:01.807812 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0da6e9d5-8520-418d-bbd6-525f6adfa582-istiod-ca-cert\") pod \"e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-n4t7b\" (UID: \"0da6e9d5-8520-418d-bbd6-525f6adfa582\") " pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-n4t7b" Nov 03 13:22:01.807930 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:01.807911 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0da6e9d5-8520-418d-bbd6-525f6adfa582-istio-envoy\") pod \"e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-n4t7b\" (UID: \"0da6e9d5-8520-418d-bbd6-525f6adfa582\") " pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-n4t7b" Nov 03 13:22:01.807986 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:01.807945 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0da6e9d5-8520-418d-bbd6-525f6adfa582-istio-data\") pod \"e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-n4t7b\" (UID: \"0da6e9d5-8520-418d-bbd6-525f6adfa582\") " pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-n4t7b" Nov 03 13:22:01.908812 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:01.908712 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rptbx\" (UniqueName: \"kubernetes.io/projected/0da6e9d5-8520-418d-bbd6-525f6adfa582-kube-api-access-rptbx\") pod \"e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-n4t7b\" (UID: \"0da6e9d5-8520-418d-bbd6-525f6adfa582\") " pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-n4t7b" Nov 03 13:22:01.909184 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:01.909155 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0da6e9d5-8520-418d-bbd6-525f6adfa582-workload-certs\") pod \"e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-n4t7b\" (UID: \"0da6e9d5-8520-418d-bbd6-525f6adfa582\") " pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-n4t7b" Nov 03 13:22:01.909393 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:01.909370 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0da6e9d5-8520-418d-bbd6-525f6adfa582-istiod-ca-cert\") pod \"e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-n4t7b\" (UID: \"0da6e9d5-8520-418d-bbd6-525f6adfa582\") " pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-n4t7b" Nov 03 13:22:01.909747 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:01.909604 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0da6e9d5-8520-418d-bbd6-525f6adfa582-workload-certs\") pod \"e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-n4t7b\" (UID: \"0da6e9d5-8520-418d-bbd6-525f6adfa582\") " pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-n4t7b" Nov 03 13:22:01.909845 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:01.909810 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0da6e9d5-8520-418d-bbd6-525f6adfa582-istio-envoy\") pod \"e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-n4t7b\" (UID: \"0da6e9d5-8520-418d-bbd6-525f6adfa582\") " pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-n4t7b" Nov 03 13:22:01.910184 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:01.910159 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0da6e9d5-8520-418d-bbd6-525f6adfa582-istio-data\") pod \"e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-n4t7b\" (UID: \"0da6e9d5-8520-418d-bbd6-525f6adfa582\") " pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-n4t7b" Nov 03 13:22:01.910492 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:01.909842 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0da6e9d5-8520-418d-bbd6-525f6adfa582-istio-data\") pod \"e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-n4t7b\" (UID: \"0da6e9d5-8520-418d-bbd6-525f6adfa582\") " pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-n4t7b" Nov 03 13:22:01.910598 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:01.910566 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0da6e9d5-8520-418d-bbd6-525f6adfa582-istiod-ca-cert\") pod \"e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-n4t7b\" (UID: \"0da6e9d5-8520-418d-bbd6-525f6adfa582\") " pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-n4t7b" Nov 03 13:22:01.910670 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:01.910596 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0da6e9d5-8520-418d-bbd6-525f6adfa582-istio-token\") pod \"e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-n4t7b\" (UID: \"0da6e9d5-8520-418d-bbd6-525f6adfa582\") " pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-n4t7b" Nov 03 13:22:01.910788 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:01.910771 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0da6e9d5-8520-418d-bbd6-525f6adfa582-workload-socket\") pod \"e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-n4t7b\" (UID: \"0da6e9d5-8520-418d-bbd6-525f6adfa582\") " pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-n4t7b" Nov 03 13:22:01.911052 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:01.911032 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0da6e9d5-8520-418d-bbd6-525f6adfa582-workload-socket\") pod \"e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-n4t7b\" (UID: \"0da6e9d5-8520-418d-bbd6-525f6adfa582\") " pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-n4t7b" Nov 03 13:22:01.911159 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:01.911115 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0da6e9d5-8520-418d-bbd6-525f6adfa582-istio-podinfo\") pod \"e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-n4t7b\" (UID: \"0da6e9d5-8520-418d-bbd6-525f6adfa582\") " pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-n4t7b" Nov 03 13:22:01.911347 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:01.911277 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0da6e9d5-8520-418d-bbd6-525f6adfa582-credential-socket\") pod \"e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-n4t7b\" (UID: \"0da6e9d5-8520-418d-bbd6-525f6adfa582\") " pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-n4t7b" Nov 03 13:22:01.911617 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:01.911594 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0da6e9d5-8520-418d-bbd6-525f6adfa582-credential-socket\") pod \"e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-n4t7b\" (UID: \"0da6e9d5-8520-418d-bbd6-525f6adfa582\") " pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-n4t7b" Nov 03 13:22:01.913311 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:01.912339 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0da6e9d5-8520-418d-bbd6-525f6adfa582-istio-envoy\") pod \"e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-n4t7b\" (UID: \"0da6e9d5-8520-418d-bbd6-525f6adfa582\") " pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-n4t7b" Nov 03 13:22:01.914048 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:01.914024 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0da6e9d5-8520-418d-bbd6-525f6adfa582-istio-podinfo\") pod \"e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-n4t7b\" (UID: \"0da6e9d5-8520-418d-bbd6-525f6adfa582\") " pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-n4t7b" Nov 03 13:22:01.918376 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:01.918351 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rptbx\" (UniqueName: \"kubernetes.io/projected/0da6e9d5-8520-418d-bbd6-525f6adfa582-kube-api-access-rptbx\") pod \"e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-n4t7b\" (UID: \"0da6e9d5-8520-418d-bbd6-525f6adfa582\") " pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-n4t7b" Nov 03 13:22:01.919857 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:01.919832 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0da6e9d5-8520-418d-bbd6-525f6adfa582-istio-token\") pod \"e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-n4t7b\" (UID: \"0da6e9d5-8520-418d-bbd6-525f6adfa582\") " pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-n4t7b" Nov 03 13:22:02.130844 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:02.130807 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-n4t7b" Nov 03 13:22:02.164480 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:02.164414 5024 patch_prober.go:29] interesting pod/e2e-extmod-b-7c192b-openshift-default-5c496b978f-wkz2r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body= Nov 03 13:22:02.164480 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:02.164467 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-wkz2r" podUID="c49c4c7b-3182-4699-898a-5407d9a3fea1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" Nov 03 13:22:02.453414 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:02.453385 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-n4t7b"] Nov 03 13:22:02.454249 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:22:02.454226 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0da6e9d5_8520_418d_bbd6_525f6adfa582.slice/crio-7f517fea1a6a7687790a4689e9b253dc8db3493a533459686171373e1531c173 WatchSource:0}: Error finding container 7f517fea1a6a7687790a4689e9b253dc8db3493a533459686171373e1531c173: Status 404 returned error can't find the container with id 7f517fea1a6a7687790a4689e9b253dc8db3493a533459686171373e1531c173 Nov 03 13:22:02.456850 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:02.456819 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:22:02.456931 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:02.456880 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:22:02.456931 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:02.456909 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:22:02.756370 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:02.756100 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-n4t7b" event={"ID":"0da6e9d5-8520-418d-bbd6-525f6adfa582","Type":"ContainerStarted","Data":"cc197516ce142f9ee3b6d09e57c19dbcdfb6b544edf0ca7a9d2bd425436491d7"} Nov 03 13:22:02.756370 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:02.756164 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-n4t7b" event={"ID":"0da6e9d5-8520-418d-bbd6-525f6adfa582","Type":"ContainerStarted","Data":"7f517fea1a6a7687790a4689e9b253dc8db3493a533459686171373e1531c173"} Nov 03 13:22:02.781305 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:02.781134 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-n4t7b" podStartSLOduration=1.781112195 podStartE2EDuration="1.781112195s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:22:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:22:02.780769985 +0000 UTC m=+802.788961261" watchObservedRunningTime="2026-11-03 13:22:02.781112195 +0000 UTC m=+802.789303477" Nov 03 13:22:03.132010 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:03.131974 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-n4t7b" Nov 03 13:22:03.133246 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:03.133224 5024 patch_prober.go:29] interesting pod/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-n4t7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Nov 03 13:22:03.133367 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:03.133270 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-n4t7b" podUID="0da6e9d5-8520-418d-bbd6-525f6adfa582" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Nov 03 13:22:03.164266 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:03.164240 5024 patch_prober.go:29] interesting pod/e2e-extmod-b-7c192b-openshift-default-5c496b978f-wkz2r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body= Nov 03 13:22:03.164433 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:03.164314 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-wkz2r" podUID="c49c4c7b-3182-4699-898a-5407d9a3fea1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" Nov 03 13:22:04.135935 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:04.135895 5024 patch_prober.go:29] interesting pod/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-n4t7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Nov 03 13:22:04.136567 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:04.136529 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-n4t7b" podUID="0da6e9d5-8520-418d-bbd6-525f6adfa582" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Nov 03 13:22:04.179363 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:04.165639 5024 patch_prober.go:29] interesting pod/e2e-extmod-b-7c192b-openshift-default-5c496b978f-wkz2r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body= Nov 03 13:22:04.179363 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:04.165692 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-wkz2r" podUID="c49c4c7b-3182-4699-898a-5407d9a3fea1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" Nov 03 13:22:05.138314 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:05.131693 5024 patch_prober.go:29] interesting pod/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-n4t7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Nov 03 13:22:05.151349 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:05.138889 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-n4t7b" podUID="0da6e9d5-8520-418d-bbd6-525f6adfa582" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Nov 03 13:22:05.164123 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:05.164042 5024 patch_prober.go:29] interesting pod/e2e-extmod-b-7c192b-openshift-default-5c496b978f-wkz2r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body= Nov 03 13:22:05.164123 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:05.164086 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-wkz2r" podUID="c49c4c7b-3182-4699-898a-5407d9a3fea1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" Nov 03 13:22:05.708310 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:05.700465 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-a2ccd8-6bfb4ddf8f-vqss2" Nov 03 13:22:06.142439 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:06.142401 5024 patch_prober.go:29] interesting pod/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-n4t7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Nov 03 13:22:06.143223 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:06.143192 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-n4t7b" podUID="0da6e9d5-8520-418d-bbd6-525f6adfa582" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Nov 03 13:22:06.165734 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:06.165697 5024 patch_prober.go:29] interesting pod/e2e-extmod-b-7c192b-openshift-default-5c496b978f-wkz2r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body= Nov 03 13:22:06.166016 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:06.165985 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-wkz2r" podUID="c49c4c7b-3182-4699-898a-5407d9a3fea1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" Nov 03 13:22:07.134311 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:07.133735 5024 patch_prober.go:29] interesting pod/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-n4t7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Nov 03 13:22:07.134311 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:07.133801 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-n4t7b" podUID="0da6e9d5-8520-418d-bbd6-525f6adfa582" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Nov 03 13:22:07.183784 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:07.183741 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-wkz2r" Nov 03 13:22:07.187436 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:07.187280 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-wkz2r" Nov 03 13:22:07.190704 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:07.190312 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-wkz2r" Nov 03 13:22:08.134765 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:08.134726 5024 patch_prober.go:29] interesting pod/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-n4t7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Nov 03 13:22:08.135052 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:08.135021 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-n4t7b" podUID="0da6e9d5-8520-418d-bbd6-525f6adfa582" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Nov 03 13:22:08.740523 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:08.740451 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-db71f4/test-model-db71f4-kserve-6cf694c5f9-57cmd" podUID="029ef988-b71c-4de2-9b27-d7506a869ed8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 13:22:09.172444 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:09.172409 5024 patch_prober.go:29] interesting pod/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-n4t7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Nov 03 13:22:09.172744 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:09.172701 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-n4t7b" podUID="0da6e9d5-8520-418d-bbd6-525f6adfa582" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Nov 03 13:22:09.772914 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:22:09.772877 5024 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d7mwz" containerName="extractor" Nov 03 13:22:09.907399 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:09.906715 5024 patch_prober.go:29] interesting pod/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-d7gxh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 13:22:09.907399 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:09.906862 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-d7gxh" podUID="6ec0db42-ee6d-496c-8792-a17fe716e820" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 13:22:10.197056 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:10.197016 5024 patch_prober.go:29] interesting pod/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-n4t7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Nov 03 13:22:10.197404 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:10.197365 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-n4t7b" podUID="0da6e9d5-8520-418d-bbd6-525f6adfa582" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Nov 03 13:22:11.193330 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:11.191862 5024 patch_prober.go:29] interesting pod/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-n4t7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Nov 03 13:22:11.193330 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:11.191968 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-n4t7b" podUID="0da6e9d5-8520-418d-bbd6-525f6adfa582" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Nov 03 13:22:12.138575 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:12.138542 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-n4t7b" Nov 03 13:22:12.139268 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:12.139242 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-n4t7b" Nov 03 13:22:12.140498 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:12.140457 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-n4t7b" Nov 03 13:22:19.123298 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:19.123245 5024 patch_prober.go:29] interesting pod/e2e-praxis-a-a2ccd8-openshift-default-798b597bf7-5ljc2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Nov 03 13:22:19.123733 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:19.123328 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-a-a2ccd8-openshift-default-798b597bf7-5ljc2" podUID="c08c13a6-4853-40b2-9496-ac50f59b1bf8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Nov 03 13:22:19.313346 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:19.313310 5024 generic.go:356] "Generic (PLEG): container finished" podID="c08c13a6-4853-40b2-9496-ac50f59b1bf8" containerID="13962ec05066f1acad8d5e943e5ed0f14c8b9d9fa9c29908e2223765a48932d4" exitCode=0 Nov 03 13:22:19.313346 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:19.313331 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-a2ccd8-openshift-default-798b597bf7-5ljc2" event={"ID":"c08c13a6-4853-40b2-9496-ac50f59b1bf8","Type":"ContainerDied","Data":"13962ec05066f1acad8d5e943e5ed0f14c8b9d9fa9c29908e2223765a48932d4"} Nov 03 13:22:19.313737 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:19.313718 5024 scope.go:122] "RemoveContainer" containerID="13962ec05066f1acad8d5e943e5ed0f14c8b9d9fa9c29908e2223765a48932d4" Nov 03 13:22:19.315120 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:19.315089 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:22:19.315185 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:19.315160 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:22:19.315224 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:19.315193 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:22:20.324160 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:20.324117 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-a2ccd8-openshift-default-798b597bf7-5ljc2" event={"ID":"c08c13a6-4853-40b2-9496-ac50f59b1bf8","Type":"ContainerStarted","Data":"f7f63080bf0fb5b2a59903eb4d3bf9d86a386f978c8bbc70fee460d73433df0b"} Nov 03 13:22:20.928311 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:20.927755 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-a2ccd8-openshift-default-798b597bf7-5ljc2" Nov 03 13:22:20.928311 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:20.927937 5024 patch_prober.go:29] interesting pod/e2e-praxis-a-a2ccd8-openshift-default-798b597bf7-5ljc2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Nov 03 13:22:20.928311 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:20.927997 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-a2ccd8-openshift-default-798b597bf7-5ljc2" podUID="c08c13a6-4853-40b2-9496-ac50f59b1bf8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Nov 03 13:22:21.906595 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:21.906559 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-b26c22-cc548f947-ddsxv"] Nov 03 13:22:21.911370 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:21.911347 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-b26c22-cc548f947-ddsxv" Nov 03 13:22:21.917944 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:21.917923 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-b26c22\"" Nov 03 13:22:21.918311 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:21.918268 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-b26c22-dockercfg-jf68r\"" Nov 03 13:22:21.921476 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:21.921449 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-b26c22-cc548f947-ddsxv"] Nov 03 13:22:21.925756 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:21.925730 5024 patch_prober.go:29] interesting pod/e2e-praxis-a-a2ccd8-openshift-default-798b597bf7-5ljc2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Nov 03 13:22:21.925859 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:21.925808 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-a2ccd8-openshift-default-798b597bf7-5ljc2" podUID="c08c13a6-4853-40b2-9496-ac50f59b1bf8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Nov 03 13:22:21.947296 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:21.947255 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c05ee6f-6019-4020-8d8e-a8fb87a42565-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-b26c22-cc548f947-ddsxv\" (UID: \"1c05ee6f-6019-4020-8d8e-a8fb87a42565\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-b26c22-cc548f947-ddsxv" Nov 03 13:22:21.947544 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:21.947523 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cnvkp\" (UniqueName: \"kubernetes.io/projected/1c05ee6f-6019-4020-8d8e-a8fb87a42565-kube-api-access-cnvkp\") pod \"payload-pre-processing-e2e-praxis-b-b26c22-cc548f947-ddsxv\" (UID: \"1c05ee6f-6019-4020-8d8e-a8fb87a42565\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-b26c22-cc548f947-ddsxv" Nov 03 13:22:22.077309 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:22.053069 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cnvkp\" (UniqueName: \"kubernetes.io/projected/1c05ee6f-6019-4020-8d8e-a8fb87a42565-kube-api-access-cnvkp\") pod \"payload-pre-processing-e2e-praxis-b-b26c22-cc548f947-ddsxv\" (UID: \"1c05ee6f-6019-4020-8d8e-a8fb87a42565\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-b26c22-cc548f947-ddsxv" Nov 03 13:22:22.099310 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:22.078031 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c05ee6f-6019-4020-8d8e-a8fb87a42565-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-b26c22-cc548f947-ddsxv\" (UID: \"1c05ee6f-6019-4020-8d8e-a8fb87a42565\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-b26c22-cc548f947-ddsxv" Nov 03 13:22:22.239264 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:22.239175 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cnvkp\" (UniqueName: \"kubernetes.io/projected/1c05ee6f-6019-4020-8d8e-a8fb87a42565-kube-api-access-cnvkp\") pod \"payload-pre-processing-e2e-praxis-b-b26c22-cc548f947-ddsxv\" (UID: \"1c05ee6f-6019-4020-8d8e-a8fb87a42565\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-b26c22-cc548f947-ddsxv" Nov 03 13:22:22.244826 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:22.244794 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c05ee6f-6019-4020-8d8e-a8fb87a42565-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-b26c22-cc548f947-ddsxv\" (UID: \"1c05ee6f-6019-4020-8d8e-a8fb87a42565\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-b26c22-cc548f947-ddsxv" Nov 03 13:22:22.530828 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:22.530751 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-b26c22-cc548f947-ddsxv" Nov 03 13:22:22.845332 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:22.845251 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-b26c22-cc548f947-ddsxv"] Nov 03 13:22:22.845332 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:22:22.845279 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c05ee6f_6019_4020_8d8e_a8fb87a42565.slice/crio-3541b29ccdf91aab6698b43d6f05bada689c6a209c4cc54352797c62851c052e WatchSource:0}: Error finding container 3541b29ccdf91aab6698b43d6f05bada689c6a209c4cc54352797c62851c052e: Status 404 returned error can't find the container with id 3541b29ccdf91aab6698b43d6f05bada689c6a209c4cc54352797c62851c052e Nov 03 13:22:22.935309 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:22.929919 5024 patch_prober.go:29] interesting pod/e2e-praxis-a-a2ccd8-openshift-default-798b597bf7-5ljc2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Nov 03 13:22:22.935309 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:22.929995 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-a2ccd8-openshift-default-798b597bf7-5ljc2" podUID="c08c13a6-4853-40b2-9496-ac50f59b1bf8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Nov 03 13:22:23.354805 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:23.354767 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-b26c22-cc548f947-ddsxv" event={"ID":"1c05ee6f-6019-4020-8d8e-a8fb87a42565","Type":"ContainerStarted","Data":"a06338f012595ad3e22ac5ef1424f2485da6402cc32aef7393ac6a5741fcf380"} Nov 03 13:22:23.354988 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:23.354831 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-b26c22-cc548f947-ddsxv" Nov 03 13:22:23.354988 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:23.354852 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-b26c22-cc548f947-ddsxv" event={"ID":"1c05ee6f-6019-4020-8d8e-a8fb87a42565","Type":"ContainerStarted","Data":"3541b29ccdf91aab6698b43d6f05bada689c6a209c4cc54352797c62851c052e"} Nov 03 13:22:23.394317 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:23.392110 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-b26c22-cc548f947-ddsxv" podStartSLOduration=2.392091203 podStartE2EDuration="2.392091203s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:22:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:22:23.388868516 +0000 UTC m=+823.397059803" watchObservedRunningTime="2026-11-03 13:22:23.392091203 +0000 UTC m=+823.400282483" Nov 03 13:22:23.925006 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:23.924965 5024 patch_prober.go:29] interesting pod/e2e-praxis-a-a2ccd8-openshift-default-798b597bf7-5ljc2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Nov 03 13:22:23.925188 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:23.925023 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-a2ccd8-openshift-default-798b597bf7-5ljc2" podUID="c08c13a6-4853-40b2-9496-ac50f59b1bf8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Nov 03 13:22:25.000305 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:24.984579 5024 patch_prober.go:29] interesting pod/e2e-praxis-a-a2ccd8-openshift-default-798b597bf7-5ljc2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Nov 03 13:22:25.000305 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:24.984645 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-a2ccd8-openshift-default-798b597bf7-5ljc2" podUID="c08c13a6-4853-40b2-9496-ac50f59b1bf8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Nov 03 13:22:25.927309 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:25.926651 5024 patch_prober.go:29] interesting pod/e2e-praxis-a-a2ccd8-openshift-default-798b597bf7-5ljc2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Nov 03 13:22:25.927309 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:25.926712 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-a2ccd8-openshift-default-798b597bf7-5ljc2" podUID="c08c13a6-4853-40b2-9496-ac50f59b1bf8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Nov 03 13:22:26.928311 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:26.927720 5024 patch_prober.go:29] interesting pod/e2e-praxis-a-a2ccd8-openshift-default-798b597bf7-5ljc2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Nov 03 13:22:26.928311 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:26.927788 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-a2ccd8-openshift-default-798b597bf7-5ljc2" podUID="c08c13a6-4853-40b2-9496-ac50f59b1bf8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Nov 03 13:22:27.269006 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:27.268915 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-b26c22-77d4fd9fb-wtf7j"] Nov 03 13:22:27.338305 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:27.331829 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-b26c22-77d4fd9fb-wtf7j"] Nov 03 13:22:27.338305 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:27.331951 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b26c22-77d4fd9fb-wtf7j" Nov 03 13:22:27.377654 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:27.377616 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-b26c22-serving-cert\"" Nov 03 13:22:27.474307 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:27.440520 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/24b840c8-af1d-41da-8b95-b70f13c93125-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-b26c22-77d4fd9fb-wtf7j\" (UID: \"24b840c8-af1d-41da-8b95-b70f13c93125\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b26c22-77d4fd9fb-wtf7j" Nov 03 13:22:27.474307 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:27.440753 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/24b840c8-af1d-41da-8b95-b70f13c93125-metrics-certs\") pod \"maas-api-e2e-praxis-b-b26c22-77d4fd9fb-wtf7j\" (UID: \"24b840c8-af1d-41da-8b95-b70f13c93125\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b26c22-77d4fd9fb-wtf7j" Nov 03 13:22:27.474307 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:27.440857 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/24b840c8-af1d-41da-8b95-b70f13c93125-maas-api-tls\") pod \"maas-api-e2e-praxis-b-b26c22-77d4fd9fb-wtf7j\" (UID: \"24b840c8-af1d-41da-8b95-b70f13c93125\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b26c22-77d4fd9fb-wtf7j" Nov 03 13:22:27.474307 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:27.440970 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ntp67\" (UniqueName: \"kubernetes.io/projected/24b840c8-af1d-41da-8b95-b70f13c93125-kube-api-access-ntp67\") pod \"maas-api-e2e-praxis-b-b26c22-77d4fd9fb-wtf7j\" (UID: \"24b840c8-af1d-41da-8b95-b70f13c93125\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b26c22-77d4fd9fb-wtf7j" Nov 03 13:22:27.562884 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:27.562792 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/24b840c8-af1d-41da-8b95-b70f13c93125-maas-api-tls\") pod \"maas-api-e2e-praxis-b-b26c22-77d4fd9fb-wtf7j\" (UID: \"24b840c8-af1d-41da-8b95-b70f13c93125\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b26c22-77d4fd9fb-wtf7j" Nov 03 13:22:27.564675 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:27.564647 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ntp67\" (UniqueName: \"kubernetes.io/projected/24b840c8-af1d-41da-8b95-b70f13c93125-kube-api-access-ntp67\") pod \"maas-api-e2e-praxis-b-b26c22-77d4fd9fb-wtf7j\" (UID: \"24b840c8-af1d-41da-8b95-b70f13c93125\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b26c22-77d4fd9fb-wtf7j" Nov 03 13:22:27.565129 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:27.565095 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/24b840c8-af1d-41da-8b95-b70f13c93125-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-b26c22-77d4fd9fb-wtf7j\" (UID: \"24b840c8-af1d-41da-8b95-b70f13c93125\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b26c22-77d4fd9fb-wtf7j" Nov 03 13:22:27.566379 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:27.566006 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/24b840c8-af1d-41da-8b95-b70f13c93125-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-b26c22-77d4fd9fb-wtf7j\" (UID: \"24b840c8-af1d-41da-8b95-b70f13c93125\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b26c22-77d4fd9fb-wtf7j" Nov 03 13:22:27.567116 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:27.567093 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/24b840c8-af1d-41da-8b95-b70f13c93125-metrics-certs\") pod \"maas-api-e2e-praxis-b-b26c22-77d4fd9fb-wtf7j\" (UID: \"24b840c8-af1d-41da-8b95-b70f13c93125\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b26c22-77d4fd9fb-wtf7j" Nov 03 13:22:27.594886 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:27.594852 5024 patch_prober.go:29] interesting pod/e2e-worker-w4-c381a7-openshift-default-78bcd4695-kjbs5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 13:22:27.595122 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:27.595097 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-kjbs5" podUID="4eeaf0ad-8a27-44c2-8641-88d4d64e7969" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 13:22:27.783281 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:27.783241 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ntp67\" (UniqueName: \"kubernetes.io/projected/24b840c8-af1d-41da-8b95-b70f13c93125-kube-api-access-ntp67\") pod \"maas-api-e2e-praxis-b-b26c22-77d4fd9fb-wtf7j\" (UID: \"24b840c8-af1d-41da-8b95-b70f13c93125\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b26c22-77d4fd9fb-wtf7j" Nov 03 13:22:27.804629 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:27.804557 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/24b840c8-af1d-41da-8b95-b70f13c93125-maas-api-tls\") pod \"maas-api-e2e-praxis-b-b26c22-77d4fd9fb-wtf7j\" (UID: \"24b840c8-af1d-41da-8b95-b70f13c93125\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b26c22-77d4fd9fb-wtf7j" Nov 03 13:22:27.815340 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:27.814527 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/24b840c8-af1d-41da-8b95-b70f13c93125-metrics-certs\") pod \"maas-api-e2e-praxis-b-b26c22-77d4fd9fb-wtf7j\" (UID: \"24b840c8-af1d-41da-8b95-b70f13c93125\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b26c22-77d4fd9fb-wtf7j" Nov 03 13:22:27.865993 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:27.865952 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b26c22-77d4fd9fb-wtf7j" Nov 03 13:22:27.994307 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:27.970582 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-a2ccd8-openshift-default-798b597bf7-5ljc2" Nov 03 13:22:27.995683 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:27.995325 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-a2ccd8-openshift-default-798b597bf7-5ljc2" Nov 03 13:22:28.034305 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:28.011669 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-a2ccd8-openshift-default-798b597bf7-5ljc2" Nov 03 13:22:28.642222 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:28.634380 5024 patch_prober.go:29] interesting pod/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-d66hs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 13:22:28.642222 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:28.634446 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-d66hs" podUID="14bfce12-7ae7-4866-aa46-bce174d28ca6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 13:22:29.216417 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:29.210202 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-b26c22-77d4fd9fb-wtf7j"] Nov 03 13:22:29.231593 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:22:29.231558 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24b840c8_af1d_41da_8b95_b70f13c93125.slice/crio-2e33395710ecb85eba7dc1e71e3372696162361c777a91f161fa9a2020d3c3ca WatchSource:0}: Error finding container 2e33395710ecb85eba7dc1e71e3372696162361c777a91f161fa9a2020d3c3ca: Status 404 returned error can't find the container with id 2e33395710ecb85eba7dc1e71e3372696162361c777a91f161fa9a2020d3c3ca Nov 03 13:22:30.000276 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:30.000174 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b26c22-77d4fd9fb-wtf7j" event={"ID":"24b840c8-af1d-41da-8b95-b70f13c93125","Type":"ContainerStarted","Data":"00924c985904874527910172f45cf969c1681be901fdf724d8c36ab39065d551"} Nov 03 13:22:30.000276 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:30.000244 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b26c22-77d4fd9fb-wtf7j" event={"ID":"24b840c8-af1d-41da-8b95-b70f13c93125","Type":"ContainerStarted","Data":"2e33395710ecb85eba7dc1e71e3372696162361c777a91f161fa9a2020d3c3ca"} Nov 03 13:22:30.000858 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:30.000832 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b26c22-77d4fd9fb-wtf7j" Nov 03 13:22:30.026343 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:30.026268 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b26c22-77d4fd9fb-wtf7j" podStartSLOduration=2.585322808 podStartE2EDuration="3.026247155s" totalImagesPullingTime="440.924347ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:22:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:22:30.025471662 +0000 UTC m=+830.033662989" watchObservedRunningTime="2026-11-03 13:22:30.026247155 +0000 UTC m=+830.034438433" Nov 03 13:22:34.363802 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:34.363770 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-b26c22-cc548f947-ddsxv" Nov 03 13:22:36.116381 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:36.116333 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b26c22-77d4fd9fb-wtf7j" Nov 03 13:22:39.323470 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:39.323382 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-k2fbp"] Nov 03 13:22:39.413536 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:39.397402 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-k2fbp"] Nov 03 13:22:39.421907 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:39.421877 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-k2fbp" Nov 03 13:22:39.477505 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:39.476872 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd2442f9-1e68-4d0d-b363-42f2f6a03f07-istio-data\") pod \"e2e-extmod-b-7c192b-openshift-default-5c496b978f-k2fbp\" (UID: \"fd2442f9-1e68-4d0d-b363-42f2f6a03f07\") " pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-k2fbp" Nov 03 13:22:39.477505 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:39.476922 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd2442f9-1e68-4d0d-b363-42f2f6a03f07-istiod-ca-cert\") pod \"e2e-extmod-b-7c192b-openshift-default-5c496b978f-k2fbp\" (UID: \"fd2442f9-1e68-4d0d-b363-42f2f6a03f07\") " pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-k2fbp" Nov 03 13:22:39.477505 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:39.476987 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd2442f9-1e68-4d0d-b363-42f2f6a03f07-credential-socket\") pod \"e2e-extmod-b-7c192b-openshift-default-5c496b978f-k2fbp\" (UID: \"fd2442f9-1e68-4d0d-b363-42f2f6a03f07\") " pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-k2fbp" Nov 03 13:22:39.477505 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:39.477029 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd2442f9-1e68-4d0d-b363-42f2f6a03f07-istio-token\") pod \"e2e-extmod-b-7c192b-openshift-default-5c496b978f-k2fbp\" (UID: \"fd2442f9-1e68-4d0d-b363-42f2f6a03f07\") " pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-k2fbp" Nov 03 13:22:39.477505 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:39.477160 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9mgw\" (UniqueName: \"kubernetes.io/projected/fd2442f9-1e68-4d0d-b363-42f2f6a03f07-kube-api-access-p9mgw\") pod \"e2e-extmod-b-7c192b-openshift-default-5c496b978f-k2fbp\" (UID: \"fd2442f9-1e68-4d0d-b363-42f2f6a03f07\") " pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-k2fbp" Nov 03 13:22:39.477505 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:39.477216 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd2442f9-1e68-4d0d-b363-42f2f6a03f07-workload-socket\") pod \"e2e-extmod-b-7c192b-openshift-default-5c496b978f-k2fbp\" (UID: \"fd2442f9-1e68-4d0d-b363-42f2f6a03f07\") " pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-k2fbp" Nov 03 13:22:39.481302 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:39.477996 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd2442f9-1e68-4d0d-b363-42f2f6a03f07-workload-certs\") pod \"e2e-extmod-b-7c192b-openshift-default-5c496b978f-k2fbp\" (UID: \"fd2442f9-1e68-4d0d-b363-42f2f6a03f07\") " pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-k2fbp" Nov 03 13:22:39.481302 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:39.478066 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd2442f9-1e68-4d0d-b363-42f2f6a03f07-istio-envoy\") pod \"e2e-extmod-b-7c192b-openshift-default-5c496b978f-k2fbp\" (UID: \"fd2442f9-1e68-4d0d-b363-42f2f6a03f07\") " pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-k2fbp" Nov 03 13:22:39.481302 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:39.478199 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd2442f9-1e68-4d0d-b363-42f2f6a03f07-istio-podinfo\") pod \"e2e-extmod-b-7c192b-openshift-default-5c496b978f-k2fbp\" (UID: \"fd2442f9-1e68-4d0d-b363-42f2f6a03f07\") " pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-k2fbp" Nov 03 13:22:39.569647 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:22:39.568527 5024 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d7mwz" containerName="extractor" Nov 03 13:22:39.579723 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:39.579635 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd2442f9-1e68-4d0d-b363-42f2f6a03f07-credential-socket\") pod \"e2e-extmod-b-7c192b-openshift-default-5c496b978f-k2fbp\" (UID: \"fd2442f9-1e68-4d0d-b363-42f2f6a03f07\") " pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-k2fbp" Nov 03 13:22:39.579723 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:39.579694 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd2442f9-1e68-4d0d-b363-42f2f6a03f07-istio-token\") pod \"e2e-extmod-b-7c192b-openshift-default-5c496b978f-k2fbp\" (UID: \"fd2442f9-1e68-4d0d-b363-42f2f6a03f07\") " pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-k2fbp" Nov 03 13:22:39.579932 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:39.579825 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9mgw\" (UniqueName: \"kubernetes.io/projected/fd2442f9-1e68-4d0d-b363-42f2f6a03f07-kube-api-access-p9mgw\") pod \"e2e-extmod-b-7c192b-openshift-default-5c496b978f-k2fbp\" (UID: \"fd2442f9-1e68-4d0d-b363-42f2f6a03f07\") " pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-k2fbp" Nov 03 13:22:39.579932 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:39.579877 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd2442f9-1e68-4d0d-b363-42f2f6a03f07-workload-socket\") pod \"e2e-extmod-b-7c192b-openshift-default-5c496b978f-k2fbp\" (UID: \"fd2442f9-1e68-4d0d-b363-42f2f6a03f07\") " pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-k2fbp" Nov 03 13:22:39.580049 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:39.579937 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd2442f9-1e68-4d0d-b363-42f2f6a03f07-workload-certs\") pod \"e2e-extmod-b-7c192b-openshift-default-5c496b978f-k2fbp\" (UID: \"fd2442f9-1e68-4d0d-b363-42f2f6a03f07\") " pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-k2fbp" Nov 03 13:22:39.580049 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:39.579968 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd2442f9-1e68-4d0d-b363-42f2f6a03f07-istio-envoy\") pod \"e2e-extmod-b-7c192b-openshift-default-5c496b978f-k2fbp\" (UID: \"fd2442f9-1e68-4d0d-b363-42f2f6a03f07\") " pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-k2fbp" Nov 03 13:22:39.580153 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:39.580079 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd2442f9-1e68-4d0d-b363-42f2f6a03f07-istio-podinfo\") pod \"e2e-extmod-b-7c192b-openshift-default-5c496b978f-k2fbp\" (UID: \"fd2442f9-1e68-4d0d-b363-42f2f6a03f07\") " pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-k2fbp" Nov 03 13:22:39.580153 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:39.580118 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd2442f9-1e68-4d0d-b363-42f2f6a03f07-istio-data\") pod \"e2e-extmod-b-7c192b-openshift-default-5c496b978f-k2fbp\" (UID: \"fd2442f9-1e68-4d0d-b363-42f2f6a03f07\") " pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-k2fbp" Nov 03 13:22:39.580153 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:39.580146 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd2442f9-1e68-4d0d-b363-42f2f6a03f07-istiod-ca-cert\") pod \"e2e-extmod-b-7c192b-openshift-default-5c496b978f-k2fbp\" (UID: \"fd2442f9-1e68-4d0d-b363-42f2f6a03f07\") " pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-k2fbp" Nov 03 13:22:39.580850 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:39.580822 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd2442f9-1e68-4d0d-b363-42f2f6a03f07-credential-socket\") pod \"e2e-extmod-b-7c192b-openshift-default-5c496b978f-k2fbp\" (UID: \"fd2442f9-1e68-4d0d-b363-42f2f6a03f07\") " pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-k2fbp" Nov 03 13:22:39.582071 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:39.581435 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd2442f9-1e68-4d0d-b363-42f2f6a03f07-workload-socket\") pod \"e2e-extmod-b-7c192b-openshift-default-5c496b978f-k2fbp\" (UID: \"fd2442f9-1e68-4d0d-b363-42f2f6a03f07\") " pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-k2fbp" Nov 03 13:22:39.582071 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:39.581740 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd2442f9-1e68-4d0d-b363-42f2f6a03f07-workload-certs\") pod \"e2e-extmod-b-7c192b-openshift-default-5c496b978f-k2fbp\" (UID: \"fd2442f9-1e68-4d0d-b363-42f2f6a03f07\") " pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-k2fbp" Nov 03 13:22:39.582071 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:39.581977 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd2442f9-1e68-4d0d-b363-42f2f6a03f07-istio-data\") pod \"e2e-extmod-b-7c192b-openshift-default-5c496b978f-k2fbp\" (UID: \"fd2442f9-1e68-4d0d-b363-42f2f6a03f07\") " pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-k2fbp" Nov 03 13:22:39.582644 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:39.582620 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd2442f9-1e68-4d0d-b363-42f2f6a03f07-istiod-ca-cert\") pod \"e2e-extmod-b-7c192b-openshift-default-5c496b978f-k2fbp\" (UID: \"fd2442f9-1e68-4d0d-b363-42f2f6a03f07\") " pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-k2fbp" Nov 03 13:22:39.586722 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:39.586124 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd2442f9-1e68-4d0d-b363-42f2f6a03f07-istio-podinfo\") pod \"e2e-extmod-b-7c192b-openshift-default-5c496b978f-k2fbp\" (UID: \"fd2442f9-1e68-4d0d-b363-42f2f6a03f07\") " pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-k2fbp" Nov 03 13:22:39.586722 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:39.586498 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd2442f9-1e68-4d0d-b363-42f2f6a03f07-istio-envoy\") pod \"e2e-extmod-b-7c192b-openshift-default-5c496b978f-k2fbp\" (UID: \"fd2442f9-1e68-4d0d-b363-42f2f6a03f07\") " pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-k2fbp" Nov 03 13:22:39.604308 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:39.603047 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd2442f9-1e68-4d0d-b363-42f2f6a03f07-istio-token\") pod \"e2e-extmod-b-7c192b-openshift-default-5c496b978f-k2fbp\" (UID: \"fd2442f9-1e68-4d0d-b363-42f2f6a03f07\") " pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-k2fbp" Nov 03 13:22:39.661297 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:39.651136 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9mgw\" (UniqueName: \"kubernetes.io/projected/fd2442f9-1e68-4d0d-b363-42f2f6a03f07-kube-api-access-p9mgw\") pod \"e2e-extmod-b-7c192b-openshift-default-5c496b978f-k2fbp\" (UID: \"fd2442f9-1e68-4d0d-b363-42f2f6a03f07\") " pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-k2fbp" Nov 03 13:22:39.792314 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:39.790462 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-k2fbp" Nov 03 13:22:40.592631 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:22:40.592595 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd2442f9_1e68_4d0d_b363_42f2f6a03f07.slice/crio-17bbcc3dce28a3572b44a5bd2c82f97ac7e3ec60c09279432b1379e6ecce76aa WatchSource:0}: Error finding container 17bbcc3dce28a3572b44a5bd2c82f97ac7e3ec60c09279432b1379e6ecce76aa: Status 404 returned error can't find the container with id 17bbcc3dce28a3572b44a5bd2c82f97ac7e3ec60c09279432b1379e6ecce76aa Nov 03 13:22:40.645211 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:40.645151 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:22:40.645650 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:40.645546 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:22:40.645969 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:40.645946 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:22:40.897987 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:40.897952 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-k2fbp"] Nov 03 13:22:41.457147 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:41.457056 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-k2fbp" event={"ID":"fd2442f9-1e68-4d0d-b363-42f2f6a03f07","Type":"ContainerStarted","Data":"a80377780a05717214347004f3699ecbbae085d484755ef7e8087c9a859e5b2a"} Nov 03 13:22:41.457147 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:41.457122 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-k2fbp" event={"ID":"fd2442f9-1e68-4d0d-b363-42f2f6a03f07","Type":"ContainerStarted","Data":"17bbcc3dce28a3572b44a5bd2c82f97ac7e3ec60c09279432b1379e6ecce76aa"} Nov 03 13:22:41.485367 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:41.485304 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-k2fbp" podStartSLOduration=2.485270829 podStartE2EDuration="2.485270829s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:22:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:22:41.479383003 +0000 UTC m=+841.487574279" watchObservedRunningTime="2026-11-03 13:22:41.485270829 +0000 UTC m=+841.493462106" Nov 03 13:22:41.808463 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:41.808365 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-k2fbp" Nov 03 13:22:42.741024 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:42.740968 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-db71f4/test-model-db71f4-kserve-6cf694c5f9-57cmd"] Nov 03 13:22:42.798306 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:42.797399 5024 patch_prober.go:29] interesting pod/e2e-extmod-b-7c192b-openshift-default-5c496b978f-k2fbp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Nov 03 13:22:42.798306 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:42.797462 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-k2fbp" podUID="fd2442f9-1e68-4d0d-b363-42f2f6a03f07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Nov 03 13:22:43.792309 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:43.791371 5024 patch_prober.go:29] interesting pod/e2e-extmod-b-7c192b-openshift-default-5c496b978f-k2fbp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Nov 03 13:22:43.792309 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:43.791440 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-k2fbp" podUID="fd2442f9-1e68-4d0d-b363-42f2f6a03f07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Nov 03 13:22:44.796743 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:44.796702 5024 patch_prober.go:29] interesting pod/e2e-extmod-b-7c192b-openshift-default-5c496b978f-k2fbp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Nov 03 13:22:44.797341 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:44.797274 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-k2fbp" podUID="fd2442f9-1e68-4d0d-b363-42f2f6a03f07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Nov 03 13:22:45.795744 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:45.795707 5024 patch_prober.go:29] interesting pod/e2e-extmod-b-7c192b-openshift-default-5c496b978f-k2fbp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Nov 03 13:22:45.795965 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:45.795777 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-k2fbp" podUID="fd2442f9-1e68-4d0d-b363-42f2f6a03f07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Nov 03 13:22:46.796308 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:46.795741 5024 patch_prober.go:29] interesting pod/e2e-extmod-b-7c192b-openshift-default-5c496b978f-k2fbp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Nov 03 13:22:46.796308 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:46.795815 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-k2fbp" podUID="fd2442f9-1e68-4d0d-b363-42f2f6a03f07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Nov 03 13:22:47.797583 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:47.797516 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-k2fbp" Nov 03 13:22:47.799671 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:47.799647 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-k2fbp" Nov 03 13:22:47.800540 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:47.800521 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-k2fbp" Nov 03 13:22:49.614771 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:49.614739 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-17adbf-696b5d5d7f-9fqx5"] Nov 03 13:22:49.751499 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:49.751464 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-17adbf-5965f57bff-h2rr9"] Nov 03 13:22:49.752978 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:49.752747 5024 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-17adbf-5965f57bff-h2rr9" podUID="a00c1abc-3c3b-4829-94ce-f0c1bcc6d862" containerName="maas-api" containerID="cri-o://73645f59350e35fbe8726e1d43475ffab44d1b96d91a9a3eb8a981c678205530" gracePeriod=30 Nov 03 13:22:50.460788 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:50.460763 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-17adbf-5965f57bff-h2rr9" Nov 03 13:22:50.514304 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:50.513787 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a00c1abc-3c3b-4829-94ce-f0c1bcc6d862-kube-api-access-d5ltd\" (UniqueName: \"kubernetes.io/projected/a00c1abc-3c3b-4829-94ce-f0c1bcc6d862-kube-api-access-d5ltd\") pod \"a00c1abc-3c3b-4829-94ce-f0c1bcc6d862\" (UID: \"a00c1abc-3c3b-4829-94ce-f0c1bcc6d862\") " Nov 03 13:22:50.514304 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:50.513878 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a00c1abc-3c3b-4829-94ce-f0c1bcc6d862-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a00c1abc-3c3b-4829-94ce-f0c1bcc6d862-maas-api-tls\") pod \"a00c1abc-3c3b-4829-94ce-f0c1bcc6d862\" (UID: \"a00c1abc-3c3b-4829-94ce-f0c1bcc6d862\") " Nov 03 13:22:50.514304 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:50.514077 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a00c1abc-3c3b-4829-94ce-f0c1bcc6d862-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a00c1abc-3c3b-4829-94ce-f0c1bcc6d862-metrics-certs\") pod \"a00c1abc-3c3b-4829-94ce-f0c1bcc6d862\" (UID: \"a00c1abc-3c3b-4829-94ce-f0c1bcc6d862\") " Nov 03 13:22:50.514304 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:50.514122 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a00c1abc-3c3b-4829-94ce-f0c1bcc6d862-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a00c1abc-3c3b-4829-94ce-f0c1bcc6d862-openshift-service-ca\") pod \"a00c1abc-3c3b-4829-94ce-f0c1bcc6d862\" (UID: \"a00c1abc-3c3b-4829-94ce-f0c1bcc6d862\") " Nov 03 13:22:50.514661 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:50.514570 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a00c1abc-3c3b-4829-94ce-f0c1bcc6d862-openshift-service-ca" pod "a00c1abc-3c3b-4829-94ce-f0c1bcc6d862" (UID: "a00c1abc-3c3b-4829-94ce-f0c1bcc6d862"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:22:50.517097 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:50.517070 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a00c1abc-3c3b-4829-94ce-f0c1bcc6d862-metrics-certs" pod "a00c1abc-3c3b-4829-94ce-f0c1bcc6d862" (UID: "a00c1abc-3c3b-4829-94ce-f0c1bcc6d862"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 13:22:50.522100 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:50.522060 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a00c1abc-3c3b-4829-94ce-f0c1bcc6d862-kube-api-access-d5ltd" pod "a00c1abc-3c3b-4829-94ce-f0c1bcc6d862" (UID: "a00c1abc-3c3b-4829-94ce-f0c1bcc6d862"). InnerVolumeSpecName "kube-api-access-d5ltd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:22:50.523475 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:50.523446 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a00c1abc-3c3b-4829-94ce-f0c1bcc6d862-maas-api-tls" pod "a00c1abc-3c3b-4829-94ce-f0c1bcc6d862" (UID: "a00c1abc-3c3b-4829-94ce-f0c1bcc6d862"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 13:22:50.583001 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:50.582965 5024 generic.go:356] "Generic (PLEG): container finished" podID="a00c1abc-3c3b-4829-94ce-f0c1bcc6d862" containerID="73645f59350e35fbe8726e1d43475ffab44d1b96d91a9a3eb8a981c678205530" exitCode=0 Nov 03 13:22:50.583177 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:50.583031 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-17adbf-5965f57bff-h2rr9" Nov 03 13:22:50.583177 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:50.583054 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-17adbf-5965f57bff-h2rr9" event={"ID":"a00c1abc-3c3b-4829-94ce-f0c1bcc6d862","Type":"ContainerDied","Data":"73645f59350e35fbe8726e1d43475ffab44d1b96d91a9a3eb8a981c678205530"} Nov 03 13:22:50.583177 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:50.583118 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-17adbf-5965f57bff-h2rr9" event={"ID":"a00c1abc-3c3b-4829-94ce-f0c1bcc6d862","Type":"ContainerDied","Data":"14940865b97448433e901b8177af7b1316ee6a631847799b2f78def427cfaa53"} Nov 03 13:22:50.583177 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:50.583149 5024 scope.go:122] "RemoveContainer" containerID="73645f59350e35fbe8726e1d43475ffab44d1b96d91a9a3eb8a981c678205530" Nov 03 13:22:50.595432 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:50.594062 5024 scope.go:122] "RemoveContainer" containerID="73645f59350e35fbe8726e1d43475ffab44d1b96d91a9a3eb8a981c678205530" Nov 03 13:22:50.596951 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:22:50.596570 5024 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"73645f59350e35fbe8726e1d43475ffab44d1b96d91a9a3eb8a981c678205530\": container with ID starting with 73645f59350e35fbe8726e1d43475ffab44d1b96d91a9a3eb8a981c678205530 not found: ID does not exist" containerID="73645f59350e35fbe8726e1d43475ffab44d1b96d91a9a3eb8a981c678205530" Nov 03 13:22:50.596951 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:50.596611 5024 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73645f59350e35fbe8726e1d43475ffab44d1b96d91a9a3eb8a981c678205530"} err="failed to get container status \"73645f59350e35fbe8726e1d43475ffab44d1b96d91a9a3eb8a981c678205530\": rpc error: code = NotFound desc = could not find container \"73645f59350e35fbe8726e1d43475ffab44d1b96d91a9a3eb8a981c678205530\": container with ID starting with 73645f59350e35fbe8726e1d43475ffab44d1b96d91a9a3eb8a981c678205530 not found: ID does not exist" Nov 03 13:22:50.626339 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:50.617100 5024 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a00c1abc-3c3b-4829-94ce-f0c1bcc6d862-maas-api-tls\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:22:50.626339 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:50.617134 5024 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a00c1abc-3c3b-4829-94ce-f0c1bcc6d862-metrics-certs\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:22:50.626339 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:50.617150 5024 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a00c1abc-3c3b-4829-94ce-f0c1bcc6d862-openshift-service-ca\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:22:50.626339 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:50.617165 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d5ltd\" (UniqueName: \"kubernetes.io/projected/a00c1abc-3c3b-4829-94ce-f0c1bcc6d862-kube-api-access-d5ltd\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:22:50.634113 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:50.634083 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-17adbf-5965f57bff-h2rr9"] Nov 03 13:22:50.636818 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:50.636797 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-17adbf-5965f57bff-h2rr9"] Nov 03 13:22:52.108821 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:52.108786 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a00c1abc-3c3b-4829-94ce-f0c1bcc6d862" path="/var/lib/kubelet/pods/a00c1abc-3c3b-4829-94ce-f0c1bcc6d862/volumes" Nov 03 13:22:53.440308 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:53.435377 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-praxis-a-a2ccd8/praxis-route-a2ccd8-kserve-748544bd55-8rmdm"] Nov 03 13:22:53.447710 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:53.447371 5024 cpu_manager.go:414] "containerMap: removing container" podUID="a00c1abc-3c3b-4829-94ce-f0c1bcc6d862" containerName="maas-api" Nov 03 13:22:53.447710 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:53.447406 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a00c1abc-3c3b-4829-94ce-f0c1bcc6d862" containerName="maas-api" Nov 03 13:22:53.448180 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:53.448132 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="a00c1abc-3c3b-4829-94ce-f0c1bcc6d862" containerName="maas-api" Nov 03 13:22:53.469475 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:53.468816 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-a2ccd8/praxis-route-a2ccd8-kserve-748544bd55-8rmdm" Nov 03 13:22:53.497671 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:53.497628 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-praxis-a-a2ccd8\"/\"default-dockercfg-xbpsq\"" Nov 03 13:22:53.498059 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:53.498036 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-praxis-a-a2ccd8\"/\"praxis-route-a2ccd8-kserve-self-signed-certs\"" Nov 03 13:22:53.505917 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:53.503736 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-praxis-a-a2ccd8\"/\"kube-root-ca.crt\"" Nov 03 13:22:53.505917 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:53.503914 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-praxis-a-a2ccd8\"/\"openshift-service-ca.crt\"" Nov 03 13:22:53.524347 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:53.524241 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-praxis-a-a2ccd8/praxis-route-a2ccd8-kserve-748544bd55-8rmdm"] Nov 03 13:22:53.604310 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:53.569185 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/11621285-51db-4e67-903d-311f39837fc9-tmp-dir\") pod \"praxis-route-a2ccd8-kserve-748544bd55-8rmdm\" (UID: \"11621285-51db-4e67-903d-311f39837fc9\") " pod="ai-tenant-e2e-praxis-a-a2ccd8/praxis-route-a2ccd8-kserve-748544bd55-8rmdm" Nov 03 13:22:53.604310 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:53.569259 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/11621285-51db-4e67-903d-311f39837fc9-home\") pod \"praxis-route-a2ccd8-kserve-748544bd55-8rmdm\" (UID: \"11621285-51db-4e67-903d-311f39837fc9\") " pod="ai-tenant-e2e-praxis-a-a2ccd8/praxis-route-a2ccd8-kserve-748544bd55-8rmdm" Nov 03 13:22:53.604310 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:53.569327 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/11621285-51db-4e67-903d-311f39837fc9-tls-certs\") pod \"praxis-route-a2ccd8-kserve-748544bd55-8rmdm\" (UID: \"11621285-51db-4e67-903d-311f39837fc9\") " pod="ai-tenant-e2e-praxis-a-a2ccd8/praxis-route-a2ccd8-kserve-748544bd55-8rmdm" Nov 03 13:22:53.604310 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:53.569461 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/11621285-51db-4e67-903d-311f39837fc9-model-cache\") pod \"praxis-route-a2ccd8-kserve-748544bd55-8rmdm\" (UID: \"11621285-51db-4e67-903d-311f39837fc9\") " pod="ai-tenant-e2e-praxis-a-a2ccd8/praxis-route-a2ccd8-kserve-748544bd55-8rmdm" Nov 03 13:22:53.604310 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:53.569524 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wxvsc\" (UniqueName: \"kubernetes.io/projected/11621285-51db-4e67-903d-311f39837fc9-kube-api-access-wxvsc\") pod \"praxis-route-a2ccd8-kserve-748544bd55-8rmdm\" (UID: \"11621285-51db-4e67-903d-311f39837fc9\") " pod="ai-tenant-e2e-praxis-a-a2ccd8/praxis-route-a2ccd8-kserve-748544bd55-8rmdm" Nov 03 13:22:53.604310 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:53.569577 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/11621285-51db-4e67-903d-311f39837fc9-dshm\") pod \"praxis-route-a2ccd8-kserve-748544bd55-8rmdm\" (UID: \"11621285-51db-4e67-903d-311f39837fc9\") " pod="ai-tenant-e2e-praxis-a-a2ccd8/praxis-route-a2ccd8-kserve-748544bd55-8rmdm" Nov 03 13:22:53.732373 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:53.705915 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/11621285-51db-4e67-903d-311f39837fc9-model-cache\") pod \"praxis-route-a2ccd8-kserve-748544bd55-8rmdm\" (UID: \"11621285-51db-4e67-903d-311f39837fc9\") " pod="ai-tenant-e2e-praxis-a-a2ccd8/praxis-route-a2ccd8-kserve-748544bd55-8rmdm" Nov 03 13:22:53.732712 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:53.732676 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxvsc\" (UniqueName: \"kubernetes.io/projected/11621285-51db-4e67-903d-311f39837fc9-kube-api-access-wxvsc\") pod \"praxis-route-a2ccd8-kserve-748544bd55-8rmdm\" (UID: \"11621285-51db-4e67-903d-311f39837fc9\") " pod="ai-tenant-e2e-praxis-a-a2ccd8/praxis-route-a2ccd8-kserve-748544bd55-8rmdm" Nov 03 13:22:53.736335 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:53.732880 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/11621285-51db-4e67-903d-311f39837fc9-dshm\") pod \"praxis-route-a2ccd8-kserve-748544bd55-8rmdm\" (UID: \"11621285-51db-4e67-903d-311f39837fc9\") " pod="ai-tenant-e2e-praxis-a-a2ccd8/praxis-route-a2ccd8-kserve-748544bd55-8rmdm" Nov 03 13:22:53.740312 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:53.736675 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/11621285-51db-4e67-903d-311f39837fc9-tmp-dir\") pod \"praxis-route-a2ccd8-kserve-748544bd55-8rmdm\" (UID: \"11621285-51db-4e67-903d-311f39837fc9\") " pod="ai-tenant-e2e-praxis-a-a2ccd8/praxis-route-a2ccd8-kserve-748544bd55-8rmdm" Nov 03 13:22:53.749305 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:53.740528 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/11621285-51db-4e67-903d-311f39837fc9-home\") pod \"praxis-route-a2ccd8-kserve-748544bd55-8rmdm\" (UID: \"11621285-51db-4e67-903d-311f39837fc9\") " pod="ai-tenant-e2e-praxis-a-a2ccd8/praxis-route-a2ccd8-kserve-748544bd55-8rmdm" Nov 03 13:22:53.749653 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:53.749592 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/11621285-51db-4e67-903d-311f39837fc9-tls-certs\") pod \"praxis-route-a2ccd8-kserve-748544bd55-8rmdm\" (UID: \"11621285-51db-4e67-903d-311f39837fc9\") " pod="ai-tenant-e2e-praxis-a-a2ccd8/praxis-route-a2ccd8-kserve-748544bd55-8rmdm" Nov 03 13:22:53.749919 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:53.741160 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/11621285-51db-4e67-903d-311f39837fc9-home\") pod \"praxis-route-a2ccd8-kserve-748544bd55-8rmdm\" (UID: \"11621285-51db-4e67-903d-311f39837fc9\") " pod="ai-tenant-e2e-praxis-a-a2ccd8/praxis-route-a2ccd8-kserve-748544bd55-8rmdm" Nov 03 13:22:53.767429 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:53.741478 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/11621285-51db-4e67-903d-311f39837fc9-model-cache\") pod \"praxis-route-a2ccd8-kserve-748544bd55-8rmdm\" (UID: \"11621285-51db-4e67-903d-311f39837fc9\") " pod="ai-tenant-e2e-praxis-a-a2ccd8/praxis-route-a2ccd8-kserve-748544bd55-8rmdm" Nov 03 13:22:53.859209 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:53.837833 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/11621285-51db-4e67-903d-311f39837fc9-tmp-dir\") pod \"praxis-route-a2ccd8-kserve-748544bd55-8rmdm\" (UID: \"11621285-51db-4e67-903d-311f39837fc9\") " pod="ai-tenant-e2e-praxis-a-a2ccd8/praxis-route-a2ccd8-kserve-748544bd55-8rmdm" Nov 03 13:22:53.895883 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:53.874086 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxvsc\" (UniqueName: \"kubernetes.io/projected/11621285-51db-4e67-903d-311f39837fc9-kube-api-access-wxvsc\") pod \"praxis-route-a2ccd8-kserve-748544bd55-8rmdm\" (UID: \"11621285-51db-4e67-903d-311f39837fc9\") " pod="ai-tenant-e2e-praxis-a-a2ccd8/praxis-route-a2ccd8-kserve-748544bd55-8rmdm" Nov 03 13:22:53.895883 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:53.882649 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/11621285-51db-4e67-903d-311f39837fc9-dshm\") pod \"praxis-route-a2ccd8-kserve-748544bd55-8rmdm\" (UID: \"11621285-51db-4e67-903d-311f39837fc9\") " pod="ai-tenant-e2e-praxis-a-a2ccd8/praxis-route-a2ccd8-kserve-748544bd55-8rmdm" Nov 03 13:22:53.899632 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:53.899589 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/11621285-51db-4e67-903d-311f39837fc9-tls-certs\") pod \"praxis-route-a2ccd8-kserve-748544bd55-8rmdm\" (UID: \"11621285-51db-4e67-903d-311f39837fc9\") " pod="ai-tenant-e2e-praxis-a-a2ccd8/praxis-route-a2ccd8-kserve-748544bd55-8rmdm" Nov 03 13:22:54.127882 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:54.126639 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-a2ccd8/praxis-route-a2ccd8-kserve-748544bd55-8rmdm" Nov 03 13:22:54.617156 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:54.616348 5024 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-17adbf-696b5d5d7f-9fqx5" podUID="a046d933-94e8-4244-96c5-0ac0665bf579" containerName="payload-pre-processing" containerID="cri-o://f552a78cc62f8af156914e5e918b93737ffc4ddc4280d3982fd1e9e123f83560" gracePeriod=25 Nov 03 13:22:54.944168 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:54.944129 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-praxis-a-a2ccd8/praxis-route-a2ccd8-kserve-748544bd55-8rmdm"] Nov 03 13:22:55.048141 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:55.048072 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-a2ccd8/praxis-route-a2ccd8-kserve-748544bd55-8rmdm" event={"ID":"11621285-51db-4e67-903d-311f39837fc9","Type":"ContainerStarted","Data":"6fbf154178ea0d77da4bbbe1cba5d93c70ec359a3c6939c3d2e8a49adbb61fd2"} Nov 03 13:22:56.055995 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:56.055959 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-a2ccd8/praxis-route-a2ccd8-kserve-748544bd55-8rmdm" event={"ID":"11621285-51db-4e67-903d-311f39837fc9","Type":"ContainerStarted","Data":"0710966ec288a0c4283931918050baf6fc2e1f6800be5fe45fd4b4d0886378d5"} Nov 03 13:22:56.081048 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:56.080995 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-praxis-a-a2ccd8/praxis-route-a2ccd8-kserve-748544bd55-8rmdm" podStartSLOduration=3.080975542 podStartE2EDuration="3.080975542s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:22:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:22:56.076359125 +0000 UTC m=+856.084550419" watchObservedRunningTime="2026-11-03 13:22:56.080975542 +0000 UTC m=+856.089166818" Nov 03 13:22:56.127952 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:56.127923 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-praxis-a-a2ccd8/praxis-route-a2ccd8-kserve-748544bd55-8rmdm" Nov 03 13:22:57.270122 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:57.270084 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-17adbf-696b5d5d7f-9fqx5" podUID="a046d933-94e8-4244-96c5-0ac0665bf579" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 13:22:57.294416 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:57.294379 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-ded926/payload-processing-external-model-e2e-extmod-a-ded926-5c948r5dr"] Nov 03 13:22:57.299241 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:57.299216 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-ded926/payload-processing-external-model-e2e-extmod-a-ded926-5c948r5dr" Nov 03 13:22:57.306742 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:57.306719 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-ded926\"/\"agc-cred-ecad53a0492b\"" Nov 03 13:22:57.306876 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:57.306789 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-ded926\"/\"openshift-service-ca.crt\"" Nov 03 13:22:57.306949 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:57.306889 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-ded926\"/\"kube-root-ca.crt\"" Nov 03 13:22:57.307989 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:57.307971 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-ded926/payload-processing-external-model-e2e-extmod-a-ded926-5c948r5dr"] Nov 03 13:22:57.308097 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:57.308084 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-ded926\"/\"routing-overlay\"" Nov 03 13:22:57.308167 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:57.308129 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-ded926\"/\"payload-processing-external-model-plugins-e2e-extmod-a-ded926\"" Nov 03 13:22:57.465014 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:57.464984 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/23c40e8d-557b-43c0-a050-9d39181d67de-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-ded926-5c948r5dr\" (UID: \"23c40e8d-557b-43c0-a050-9d39181d67de\") " pod="ai-tenant-e2e-extmod-a-ded926/payload-processing-external-model-e2e-extmod-a-ded926-5c948r5dr" Nov 03 13:22:57.465183 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:57.465024 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pkldx\" (UniqueName: \"kubernetes.io/projected/23c40e8d-557b-43c0-a050-9d39181d67de-kube-api-access-pkldx\") pod \"payload-processing-external-model-e2e-extmod-a-ded926-5c948r5dr\" (UID: \"23c40e8d-557b-43c0-a050-9d39181d67de\") " pod="ai-tenant-e2e-extmod-a-ded926/payload-processing-external-model-e2e-extmod-a-ded926-5c948r5dr" Nov 03 13:22:57.465183 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:57.465134 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/23c40e8d-557b-43c0-a050-9d39181d67de-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-ded926-5c948r5dr\" (UID: \"23c40e8d-557b-43c0-a050-9d39181d67de\") " pod="ai-tenant-e2e-extmod-a-ded926/payload-processing-external-model-e2e-extmod-a-ded926-5c948r5dr" Nov 03 13:22:57.465183 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:57.465177 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23c40e8d-557b-43c0-a050-9d39181d67de-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-ded926-5c948r5dr\" (UID: \"23c40e8d-557b-43c0-a050-9d39181d67de\") " pod="ai-tenant-e2e-extmod-a-ded926/payload-processing-external-model-e2e-extmod-a-ded926-5c948r5dr" Nov 03 13:22:57.575387 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:57.571668 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/23c40e8d-557b-43c0-a050-9d39181d67de-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-ded926-5c948r5dr\" (UID: \"23c40e8d-557b-43c0-a050-9d39181d67de\") " pod="ai-tenant-e2e-extmod-a-ded926/payload-processing-external-model-e2e-extmod-a-ded926-5c948r5dr" Nov 03 13:22:57.575387 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:57.571736 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pkldx\" (UniqueName: \"kubernetes.io/projected/23c40e8d-557b-43c0-a050-9d39181d67de-kube-api-access-pkldx\") pod \"payload-processing-external-model-e2e-extmod-a-ded926-5c948r5dr\" (UID: \"23c40e8d-557b-43c0-a050-9d39181d67de\") " pod="ai-tenant-e2e-extmod-a-ded926/payload-processing-external-model-e2e-extmod-a-ded926-5c948r5dr" Nov 03 13:22:57.575387 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:57.571804 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/23c40e8d-557b-43c0-a050-9d39181d67de-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-ded926-5c948r5dr\" (UID: \"23c40e8d-557b-43c0-a050-9d39181d67de\") " pod="ai-tenant-e2e-extmod-a-ded926/payload-processing-external-model-e2e-extmod-a-ded926-5c948r5dr" Nov 03 13:22:57.575387 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:57.571842 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23c40e8d-557b-43c0-a050-9d39181d67de-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-ded926-5c948r5dr\" (UID: \"23c40e8d-557b-43c0-a050-9d39181d67de\") " pod="ai-tenant-e2e-extmod-a-ded926/payload-processing-external-model-e2e-extmod-a-ded926-5c948r5dr" Nov 03 13:22:57.575387 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:57.573218 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23c40e8d-557b-43c0-a050-9d39181d67de-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-ded926-5c948r5dr\" (UID: \"23c40e8d-557b-43c0-a050-9d39181d67de\") " pod="ai-tenant-e2e-extmod-a-ded926/payload-processing-external-model-e2e-extmod-a-ded926-5c948r5dr" Nov 03 13:22:57.575387 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:57.574179 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/23c40e8d-557b-43c0-a050-9d39181d67de-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-ded926-5c948r5dr\" (UID: \"23c40e8d-557b-43c0-a050-9d39181d67de\") " pod="ai-tenant-e2e-extmod-a-ded926/payload-processing-external-model-e2e-extmod-a-ded926-5c948r5dr" Nov 03 13:22:57.614306 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:57.607164 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/23c40e8d-557b-43c0-a050-9d39181d67de-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-ded926-5c948r5dr\" (UID: \"23c40e8d-557b-43c0-a050-9d39181d67de\") " pod="ai-tenant-e2e-extmod-a-ded926/payload-processing-external-model-e2e-extmod-a-ded926-5c948r5dr" Nov 03 13:22:57.614306 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:57.609802 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pkldx\" (UniqueName: \"kubernetes.io/projected/23c40e8d-557b-43c0-a050-9d39181d67de-kube-api-access-pkldx\") pod \"payload-processing-external-model-e2e-extmod-a-ded926-5c948r5dr\" (UID: \"23c40e8d-557b-43c0-a050-9d39181d67de\") " pod="ai-tenant-e2e-extmod-a-ded926/payload-processing-external-model-e2e-extmod-a-ded926-5c948r5dr" Nov 03 13:22:57.617783 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:57.617566 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-ded926/payload-processing-external-model-e2e-extmod-a-ded926-5c948r5dr" Nov 03 13:22:57.792942 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:57.775734 5024 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-db71f4/test-model-db71f4-kserve-6cf694c5f9-57cmd" podUID="029ef988-b71c-4de2-9b27-d7506a869ed8" containerName="main" containerID="cri-o://a1780f1a3eaddf562f65c244668df9e32e9062c2e94496e40f382672af4a332e" gracePeriod=45 Nov 03 13:22:57.893680 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:57.890126 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-db71f4/test-model-db71f4-kserve-6cf694c5f9-57cmd" podUID="029ef988-b71c-4de2-9b27-d7506a869ed8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": EOF" Nov 03 13:22:58.080124 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:58.077200 5024 generic.go:356] "Generic (PLEG): container finished" podID="029ef988-b71c-4de2-9b27-d7506a869ed8" containerID="a1780f1a3eaddf562f65c244668df9e32e9062c2e94496e40f382672af4a332e" exitCode=0 Nov 03 13:22:58.080124 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:58.079767 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-db71f4/test-model-db71f4-kserve-6cf694c5f9-57cmd" event={"ID":"029ef988-b71c-4de2-9b27-d7506a869ed8","Type":"ContainerDied","Data":"a1780f1a3eaddf562f65c244668df9e32e9062c2e94496e40f382672af4a332e"} Nov 03 13:22:58.168528 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:58.167669 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-ded926/payload-processing-external-model-e2e-extmod-a-ded926-5c948r5dr"] Nov 03 13:22:58.168854 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:22:58.168797 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23c40e8d_557b_43c0_a050_9d39181d67de.slice/crio-7898906663d101669e1c77deb7bcb45bdffbfa32ee1f52368c0be9bf1036301e WatchSource:0}: Error finding container 7898906663d101669e1c77deb7bcb45bdffbfa32ee1f52368c0be9bf1036301e: Status 404 returned error can't find the container with id 7898906663d101669e1c77deb7bcb45bdffbfa32ee1f52368c0be9bf1036301e Nov 03 13:22:58.214370 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:58.214336 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-db71f4/test-model-db71f4-kserve-6cf694c5f9-57cmd" Nov 03 13:22:58.398370 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:58.397592 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/029ef988-b71c-4de2-9b27-d7506a869ed8-model-cache" pod "029ef988-b71c-4de2-9b27-d7506a869ed8" (UID: "029ef988-b71c-4de2-9b27-d7506a869ed8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:22:58.399265 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:58.399222 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/029ef988-b71c-4de2-9b27-d7506a869ed8-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/029ef988-b71c-4de2-9b27-d7506a869ed8-model-cache\") pod \"029ef988-b71c-4de2-9b27-d7506a869ed8\" (UID: \"029ef988-b71c-4de2-9b27-d7506a869ed8\") " Nov 03 13:22:58.399502 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:58.399482 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/029ef988-b71c-4de2-9b27-d7506a869ed8-home\" (UniqueName: \"kubernetes.io/empty-dir/029ef988-b71c-4de2-9b27-d7506a869ed8-home\") pod \"029ef988-b71c-4de2-9b27-d7506a869ed8\" (UID: \"029ef988-b71c-4de2-9b27-d7506a869ed8\") " Nov 03 13:22:58.399770 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:58.399746 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/029ef988-b71c-4de2-9b27-d7506a869ed8-kube-api-access-jrjct\" (UniqueName: \"kubernetes.io/projected/029ef988-b71c-4de2-9b27-d7506a869ed8-kube-api-access-jrjct\") pod \"029ef988-b71c-4de2-9b27-d7506a869ed8\" (UID: \"029ef988-b71c-4de2-9b27-d7506a869ed8\") " Nov 03 13:22:58.399938 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:58.399914 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/029ef988-b71c-4de2-9b27-d7506a869ed8-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/029ef988-b71c-4de2-9b27-d7506a869ed8-tmp-dir\") pod \"029ef988-b71c-4de2-9b27-d7506a869ed8\" (UID: \"029ef988-b71c-4de2-9b27-d7506a869ed8\") " Nov 03 13:22:58.400168 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:58.400145 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/029ef988-b71c-4de2-9b27-d7506a869ed8-dshm\" (UniqueName: \"kubernetes.io/empty-dir/029ef988-b71c-4de2-9b27-d7506a869ed8-dshm\") pod \"029ef988-b71c-4de2-9b27-d7506a869ed8\" (UID: \"029ef988-b71c-4de2-9b27-d7506a869ed8\") " Nov 03 13:22:58.400367 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:58.400351 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/029ef988-b71c-4de2-9b27-d7506a869ed8-tls-certs\" (UniqueName: \"kubernetes.io/secret/029ef988-b71c-4de2-9b27-d7506a869ed8-tls-certs\") pod \"029ef988-b71c-4de2-9b27-d7506a869ed8\" (UID: \"029ef988-b71c-4de2-9b27-d7506a869ed8\") " Nov 03 13:22:58.430352 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:58.408837 5024 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/029ef988-b71c-4de2-9b27-d7506a869ed8-model-cache\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:22:58.430552 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:58.402161 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/029ef988-b71c-4de2-9b27-d7506a869ed8-home" pod "029ef988-b71c-4de2-9b27-d7506a869ed8" (UID: "029ef988-b71c-4de2-9b27-d7506a869ed8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:22:58.430654 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:58.402386 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/029ef988-b71c-4de2-9b27-d7506a869ed8-tmp-dir" pod "029ef988-b71c-4de2-9b27-d7506a869ed8" (UID: "029ef988-b71c-4de2-9b27-d7506a869ed8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:22:58.430733 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:58.406225 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/029ef988-b71c-4de2-9b27-d7506a869ed8-dshm" pod "029ef988-b71c-4de2-9b27-d7506a869ed8" (UID: "029ef988-b71c-4de2-9b27-d7506a869ed8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:22:58.430805 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:58.426779 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/029ef988-b71c-4de2-9b27-d7506a869ed8-tls-certs" pod "029ef988-b71c-4de2-9b27-d7506a869ed8" (UID: "029ef988-b71c-4de2-9b27-d7506a869ed8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 13:22:58.433778 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:58.433032 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/029ef988-b71c-4de2-9b27-d7506a869ed8-kube-api-access-jrjct" pod "029ef988-b71c-4de2-9b27-d7506a869ed8" (UID: "029ef988-b71c-4de2-9b27-d7506a869ed8"). InnerVolumeSpecName "kube-api-access-jrjct". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:22:58.532515 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:58.532487 5024 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/029ef988-b71c-4de2-9b27-d7506a869ed8-tmp-dir\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:22:58.532515 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:58.532514 5024 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/029ef988-b71c-4de2-9b27-d7506a869ed8-dshm\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:22:58.532701 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:58.532526 5024 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/029ef988-b71c-4de2-9b27-d7506a869ed8-tls-certs\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:22:58.532701 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:58.532537 5024 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/029ef988-b71c-4de2-9b27-d7506a869ed8-home\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:22:58.532701 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:58.532549 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jrjct\" (UniqueName: \"kubernetes.io/projected/029ef988-b71c-4de2-9b27-d7506a869ed8-kube-api-access-jrjct\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:22:58.748360 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:58.746273 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-ded926/payload-processing-external-model-e2e-extmod-a-ded926-5c948r5dr"] Nov 03 13:22:59.157316 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:59.154464 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-db71f4/test-model-db71f4-kserve-6cf694c5f9-57cmd" event={"ID":"029ef988-b71c-4de2-9b27-d7506a869ed8","Type":"ContainerDied","Data":"2af654e55d22a1a25e3cc42d6a33de0bdb59e9e583c07de791cdc299a096a530"} Nov 03 13:22:59.157316 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:59.154538 5024 scope.go:122] "RemoveContainer" containerID="a1780f1a3eaddf562f65c244668df9e32e9062c2e94496e40f382672af4a332e" Nov 03 13:22:59.157316 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:59.154693 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-db71f4/test-model-db71f4-kserve-6cf694c5f9-57cmd" Nov 03 13:22:59.163110 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:59.163068 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-ded926/payload-processing-external-model-e2e-extmod-a-ded926-5c948r5dr" event={"ID":"23c40e8d-557b-43c0-a050-9d39181d67de","Type":"ContainerStarted","Data":"268a5859df5b46863cbd451a20f792638b1c9bfd085f4dde7670ace4c3855155"} Nov 03 13:22:59.163437 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:59.163396 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-ded926/payload-processing-external-model-e2e-extmod-a-ded926-5c948r5dr" event={"ID":"23c40e8d-557b-43c0-a050-9d39181d67de","Type":"ContainerStarted","Data":"7898906663d101669e1c77deb7bcb45bdffbfa32ee1f52368c0be9bf1036301e"} Nov 03 13:22:59.164835 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:59.164815 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-ded926/payload-processing-external-model-e2e-extmod-a-ded926-5c948r5dr" Nov 03 13:22:59.200739 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:59.198332 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-ded926/payload-processing-external-model-e2e-extmod-a-ded926-5c948r5dr" podStartSLOduration=2.198312525 podStartE2EDuration="2.198312525s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:22:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:22:59.183828063 +0000 UTC m=+859.192019341" watchObservedRunningTime="2026-11-03 13:22:59.198312525 +0000 UTC m=+859.206503798" Nov 03 13:22:59.279310 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:22:59.274626 5024 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-a-ded926/payload-processing-external-model-plugins-e2e-extmod-a-ded926: configmap "payload-processing-external-model-plugins-e2e-extmod-a-ded926" not found Nov 03 13:22:59.279310 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:22:59.274718 5024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/23c40e8d-557b-43c0-a050-9d39181d67de-plugins-config-volume podName:23c40e8d-557b-43c0-a050-9d39181d67de nodeName:}" failed. No retries permitted until 2026-11-03 13:22:59.774696134 +0000 UTC m=+859.782887514 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/23c40e8d-557b-43c0-a050-9d39181d67de-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-a-ded926-5c948r5dr" (UID: "23c40e8d-557b-43c0-a050-9d39181d67de") : configmap "payload-processing-external-model-plugins-e2e-extmod-a-ded926" not found Nov 03 13:22:59.279310 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:22:59.275694 5024 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-ded926/agc-cred-ecad53a0492b: secret "agc-cred-ecad53a0492b" not found Nov 03 13:22:59.279310 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:22:59.275714 5024 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-ded926/payload-processing-external-model-e2e-extmod-a-ded926-5c948r5dr: secret "agc-cred-ecad53a0492b" not found Nov 03 13:22:59.279310 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:22:59.275760 5024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/23c40e8d-557b-43c0-a050-9d39181d67de-provider-credentials podName:23c40e8d-557b-43c0-a050-9d39181d67de nodeName:}" failed. No retries permitted until 2026-11-03 13:22:59.775744893 +0000 UTC m=+859.783936161 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/23c40e8d-557b-43c0-a050-9d39181d67de-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-ded926-5c948r5dr" (UID: "23c40e8d-557b-43c0-a050-9d39181d67de") : secret "agc-cred-ecad53a0492b" not found Nov 03 13:22:59.288099 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:59.281546 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-db71f4/test-model-db71f4-kserve-6cf694c5f9-57cmd"] Nov 03 13:22:59.288099 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:22:59.287973 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-db71f4/test-model-db71f4-kserve-6cf694c5f9-57cmd"] Nov 03 13:22:59.799665 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:22:59.799625 5024 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-a-ded926/payload-processing-external-model-plugins-e2e-extmod-a-ded926: configmap "payload-processing-external-model-plugins-e2e-extmod-a-ded926" not found Nov 03 13:22:59.800304 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:22:59.800262 5024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/23c40e8d-557b-43c0-a050-9d39181d67de-plugins-config-volume podName:23c40e8d-557b-43c0-a050-9d39181d67de nodeName:}" failed. No retries permitted until 2026-11-03 13:23:00.800235696 +0000 UTC m=+860.808426965 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/23c40e8d-557b-43c0-a050-9d39181d67de-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-a-ded926-5c948r5dr" (UID: "23c40e8d-557b-43c0-a050-9d39181d67de") : configmap "payload-processing-external-model-plugins-e2e-extmod-a-ded926" not found Nov 03 13:22:59.800956 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:22:59.800933 5024 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-ded926/agc-cred-ecad53a0492b: secret "agc-cred-ecad53a0492b" not found Nov 03 13:22:59.801068 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:22:59.801057 5024 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-ded926/payload-processing-external-model-e2e-extmod-a-ded926-5c948r5dr: secret "agc-cred-ecad53a0492b" not found Nov 03 13:22:59.801185 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:22:59.801176 5024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/23c40e8d-557b-43c0-a050-9d39181d67de-provider-credentials podName:23c40e8d-557b-43c0-a050-9d39181d67de nodeName:}" failed. No retries permitted until 2026-11-03 13:23:00.801157288 +0000 UTC m=+860.809348559 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/23c40e8d-557b-43c0-a050-9d39181d67de-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-ded926-5c948r5dr" (UID: "23c40e8d-557b-43c0-a050-9d39181d67de") : secret "agc-cred-ecad53a0492b" not found Nov 03 13:23:00.111720 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:23:00.111688 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="029ef988-b71c-4de2-9b27-d7506a869ed8" path="/var/lib/kubelet/pods/029ef988-b71c-4de2-9b27-d7506a869ed8/volumes" Nov 03 13:23:00.819944 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:23:00.819902 5024 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-ded926/agc-cred-ecad53a0492b: secret "agc-cred-ecad53a0492b" not found Nov 03 13:23:00.819944 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:23:00.819932 5024 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-a-ded926/payload-processing-external-model-plugins-e2e-extmod-a-ded926: configmap "payload-processing-external-model-plugins-e2e-extmod-a-ded926" not found Nov 03 13:23:00.820376 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:23:00.820012 5024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/23c40e8d-557b-43c0-a050-9d39181d67de-plugins-config-volume podName:23c40e8d-557b-43c0-a050-9d39181d67de nodeName:}" failed. No retries permitted until 2026-11-03 13:23:02.819992198 +0000 UTC m=+862.828183466 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/23c40e8d-557b-43c0-a050-9d39181d67de-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-a-ded926-5c948r5dr" (UID: "23c40e8d-557b-43c0-a050-9d39181d67de") : configmap "payload-processing-external-model-plugins-e2e-extmod-a-ded926" not found Nov 03 13:23:00.820376 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:23:00.819938 5024 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-ded926/payload-processing-external-model-e2e-extmod-a-ded926-5c948r5dr: secret "agc-cred-ecad53a0492b" not found Nov 03 13:23:00.820376 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:23:00.820094 5024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/23c40e8d-557b-43c0-a050-9d39181d67de-provider-credentials podName:23c40e8d-557b-43c0-a050-9d39181d67de nodeName:}" failed. No retries permitted until 2026-11-03 13:23:02.820076278 +0000 UTC m=+862.828267545 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/23c40e8d-557b-43c0-a050-9d39181d67de-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-ded926-5c948r5dr" (UID: "23c40e8d-557b-43c0-a050-9d39181d67de") : secret "agc-cred-ecad53a0492b" not found Nov 03 13:23:02.861176 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:23:02.861139 5024 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-ded926/agc-cred-ecad53a0492b: secret "agc-cred-ecad53a0492b" not found Nov 03 13:23:02.861176 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:23:02.861168 5024 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-ded926/payload-processing-external-model-e2e-extmod-a-ded926-5c948r5dr: secret "agc-cred-ecad53a0492b" not found Nov 03 13:23:02.861176 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:23:02.861165 5024 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-a-ded926/payload-processing-external-model-plugins-e2e-extmod-a-ded926: configmap "payload-processing-external-model-plugins-e2e-extmod-a-ded926" not found Nov 03 13:23:02.861646 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:23:02.861220 5024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/23c40e8d-557b-43c0-a050-9d39181d67de-provider-credentials podName:23c40e8d-557b-43c0-a050-9d39181d67de nodeName:}" failed. No retries permitted until 2026-11-03 13:23:06.861206393 +0000 UTC m=+866.869397661 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/23c40e8d-557b-43c0-a050-9d39181d67de-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-ded926-5c948r5dr" (UID: "23c40e8d-557b-43c0-a050-9d39181d67de") : secret "agc-cred-ecad53a0492b" not found Nov 03 13:23:02.861646 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:23:02.861238 5024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/23c40e8d-557b-43c0-a050-9d39181d67de-plugins-config-volume podName:23c40e8d-557b-43c0-a050-9d39181d67de nodeName:}" failed. No retries permitted until 2026-11-03 13:23:06.861225423 +0000 UTC m=+866.869416678 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/23c40e8d-557b-43c0-a050-9d39181d67de-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-a-ded926-5c948r5dr" (UID: "23c40e8d-557b-43c0-a050-9d39181d67de") : configmap "payload-processing-external-model-plugins-e2e-extmod-a-ded926" not found Nov 03 13:23:03.730013 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:23:03.729977 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-17adbf-openshift-default-75f4498d49-hsbdd"] Nov 03 13:23:03.730305 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:23:03.730262 5024 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-17adbf-openshift-default-75f4498d49-hsbdd" podUID="42d727c9-2a2b-4a3d-afb7-c0d2ee617b7f" containerName="istio-proxy" containerID="cri-o://6ab9d2c6cbd7023205c3c125b210bde795226750662dd659a173d9d75c5fdc6f" gracePeriod=30 Nov 03 13:23:04.143446 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:23:04.142705 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-praxis-a-a2ccd8/praxis-route-a2ccd8-kserve-748544bd55-8rmdm" Nov 03 13:23:04.203411 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:23:04.203370 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-praxis-a-a2ccd8/praxis-route-a2ccd8-kserve-748544bd55-8rmdm" Nov 03 13:23:04.264119 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:23:04.264087 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-praxis-a-a2ccd8/praxis-route-a2ccd8-kserve-748544bd55-8rmdm" Nov 03 13:23:05.171549 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:23:05.171503 5024 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-ded926/payload-processing-external-model-e2e-extmod-a-ded926-5c948r5dr" podUID="23c40e8d-557b-43c0-a050-9d39181d67de" containerName="payload-processing" containerID="cri-o://268a5859df5b46863cbd451a20f792638b1c9bfd085f4dde7670ace4c3855155" gracePeriod=25 Nov 03 13:23:05.483010 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:23:05.481991 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-ded926/payload-processing-external-model-e2e-extmod-a-ded926-5c948r5dr" Nov 03 13:23:05.561315 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:23:05.559740 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23c40e8d-557b-43c0-a050-9d39181d67de-provider-credentials\" (UniqueName: \"kubernetes.io/projected/23c40e8d-557b-43c0-a050-9d39181d67de-provider-credentials\") pod \"23c40e8d-557b-43c0-a050-9d39181d67de\" (UID: \"23c40e8d-557b-43c0-a050-9d39181d67de\") " Nov 03 13:23:05.561315 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:23:05.559902 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23c40e8d-557b-43c0-a050-9d39181d67de-kube-api-access-pkldx\" (UniqueName: \"kubernetes.io/projected/23c40e8d-557b-43c0-a050-9d39181d67de-kube-api-access-pkldx\") pod \"23c40e8d-557b-43c0-a050-9d39181d67de\" (UID: \"23c40e8d-557b-43c0-a050-9d39181d67de\") " Nov 03 13:23:05.561315 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:23:05.559969 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/23c40e8d-557b-43c0-a050-9d39181d67de-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23c40e8d-557b-43c0-a050-9d39181d67de-plugins-config-volume\") pod \"23c40e8d-557b-43c0-a050-9d39181d67de\" (UID: \"23c40e8d-557b-43c0-a050-9d39181d67de\") " Nov 03 13:23:05.561315 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:23:05.560181 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/23c40e8d-557b-43c0-a050-9d39181d67de-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/23c40e8d-557b-43c0-a050-9d39181d67de-routing-overlay-volume\") pod \"23c40e8d-557b-43c0-a050-9d39181d67de\" (UID: \"23c40e8d-557b-43c0-a050-9d39181d67de\") " Nov 03 13:23:05.561315 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:23:05.561160 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23c40e8d-557b-43c0-a050-9d39181d67de-routing-overlay-volume" pod "23c40e8d-557b-43c0-a050-9d39181d67de" (UID: "23c40e8d-557b-43c0-a050-9d39181d67de"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:23:05.566305 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:23:05.564627 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23c40e8d-557b-43c0-a050-9d39181d67de-plugins-config-volume" pod "23c40e8d-557b-43c0-a050-9d39181d67de" (UID: "23c40e8d-557b-43c0-a050-9d39181d67de"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:23:05.566305 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:23:05.565368 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23c40e8d-557b-43c0-a050-9d39181d67de-provider-credentials" pod "23c40e8d-557b-43c0-a050-9d39181d67de" (UID: "23c40e8d-557b-43c0-a050-9d39181d67de"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:23:05.569041 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:23:05.567564 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23c40e8d-557b-43c0-a050-9d39181d67de-kube-api-access-pkldx" pod "23c40e8d-557b-43c0-a050-9d39181d67de" (UID: "23c40e8d-557b-43c0-a050-9d39181d67de"). InnerVolumeSpecName "kube-api-access-pkldx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:23:05.708395 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:23:05.662412 5024 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/23c40e8d-557b-43c0-a050-9d39181d67de-routing-overlay-volume\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:23:05.708395 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:23:05.662454 5024 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/23c40e8d-557b-43c0-a050-9d39181d67de-provider-credentials\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:23:05.708395 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:23:05.662475 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pkldx\" (UniqueName: \"kubernetes.io/projected/23c40e8d-557b-43c0-a050-9d39181d67de-kube-api-access-pkldx\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:23:05.708395 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:23:05.662489 5024 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23c40e8d-557b-43c0-a050-9d39181d67de-plugins-config-volume\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:23:06.288353 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:23:06.278545 5024 generic.go:356] "Generic (PLEG): container finished" podID="23c40e8d-557b-43c0-a050-9d39181d67de" containerID="268a5859df5b46863cbd451a20f792638b1c9bfd085f4dde7670ace4c3855155" exitCode=0 Nov 03 13:23:06.288353 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:23:06.278695 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-ded926/payload-processing-external-model-e2e-extmod-a-ded926-5c948r5dr" event={"ID":"23c40e8d-557b-43c0-a050-9d39181d67de","Type":"ContainerDied","Data":"268a5859df5b46863cbd451a20f792638b1c9bfd085f4dde7670ace4c3855155"} Nov 03 13:23:06.288353 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:23:06.278745 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-ded926/payload-processing-external-model-e2e-extmod-a-ded926-5c948r5dr" event={"ID":"23c40e8d-557b-43c0-a050-9d39181d67de","Type":"ContainerDied","Data":"7898906663d101669e1c77deb7bcb45bdffbfa32ee1f52368c0be9bf1036301e"} Nov 03 13:23:06.288353 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:23:06.278774 5024 scope.go:122] "RemoveContainer" containerID="268a5859df5b46863cbd451a20f792638b1c9bfd085f4dde7670ace4c3855155" Nov 03 13:23:06.288353 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:23:06.278929 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-ded926/payload-processing-external-model-e2e-extmod-a-ded926-5c948r5dr" Nov 03 13:23:06.361336 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:23:06.357805 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-ded926/payload-processing-external-model-e2e-extmod-a-ded926-5c948r5dr"] Nov 03 13:23:06.368929 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:23:06.367583 5024 scope.go:122] "RemoveContainer" containerID="268a5859df5b46863cbd451a20f792638b1c9bfd085f4dde7670ace4c3855155" Nov 03 13:23:06.368929 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:23:06.368031 5024 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"268a5859df5b46863cbd451a20f792638b1c9bfd085f4dde7670ace4c3855155\": container with ID starting with 268a5859df5b46863cbd451a20f792638b1c9bfd085f4dde7670ace4c3855155 not found: ID does not exist" containerID="268a5859df5b46863cbd451a20f792638b1c9bfd085f4dde7670ace4c3855155" Nov 03 13:23:06.368929 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:23:06.368068 5024 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"268a5859df5b46863cbd451a20f792638b1c9bfd085f4dde7670ace4c3855155"} err="failed to get container status \"268a5859df5b46863cbd451a20f792638b1c9bfd085f4dde7670ace4c3855155\": rpc error: code = NotFound desc = could not find container \"268a5859df5b46863cbd451a20f792638b1c9bfd085f4dde7670ace4c3855155\": container with ID starting with 268a5859df5b46863cbd451a20f792638b1c9bfd085f4dde7670ace4c3855155 not found: ID does not exist" Nov 03 13:23:06.373007 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:23:06.372967 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-ded926/payload-processing-external-model-e2e-extmod-a-ded926-5c948r5dr"] Nov 03 13:23:07.283435 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:23:07.283395 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-17adbf-696b5d5d7f-9fqx5" podUID="a046d933-94e8-4244-96c5-0ac0665bf579" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 13:23:08.208314 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:23:08.207595 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="23c40e8d-557b-43c0-a050-9d39181d67de" path="/var/lib/kubelet/pods/23c40e8d-557b-43c0-a050-9d39181d67de/volumes" Nov 03 13:23:09.351720 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:23:09.351686 5024 generic.go:356] "Generic (PLEG): container finished" podID="42d727c9-2a2b-4a3d-afb7-c0d2ee617b7f" containerID="6ab9d2c6cbd7023205c3c125b210bde795226750662dd659a173d9d75c5fdc6f" exitCode=0 Nov 03 13:23:09.352087 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:23:09.351725 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-17adbf-openshift-default-75f4498d49-hsbdd" event={"ID":"42d727c9-2a2b-4a3d-afb7-c0d2ee617b7f","Type":"ContainerDied","Data":"6ab9d2c6cbd7023205c3c125b210bde795226750662dd659a173d9d75c5fdc6f"} Nov 03 13:23:09.352087 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:23:09.351827 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-17adbf-openshift-default-75f4498d49-hsbdd" event={"ID":"42d727c9-2a2b-4a3d-afb7-c0d2ee617b7f","Type":"ContainerDied","Data":"37e1c14208b20fe33c927d5d957d6d241387432b19232e1ec0ece853aae83afc"} Nov 03 13:23:09.352087 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:23:09.351853 5024 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="37e1c14208b20fe33c927d5d957d6d241387432b19232e1ec0ece853aae83afc" Nov 03 13:23:09.352087 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:23:09.351871 5024 scope.go:122] "RemoveContainer" containerID="60b1fa6c8cab7eb811c819276689b8f963f74a5ff0074beb66d743617ee6f9ba" Nov 03 13:23:09.362016 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:23:09.361999 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-17adbf-openshift-default-75f4498d49-hsbdd" Nov 03 13:23:09.503208 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:23:09.503149 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42d727c9-2a2b-4a3d-afb7-c0d2ee617b7f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42d727c9-2a2b-4a3d-afb7-c0d2ee617b7f-istio-envoy\") pod \"42d727c9-2a2b-4a3d-afb7-c0d2ee617b7f\" (UID: \"42d727c9-2a2b-4a3d-afb7-c0d2ee617b7f\") " Nov 03 13:23:09.503561 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:23:09.503536 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/42d727c9-2a2b-4a3d-afb7-c0d2ee617b7f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42d727c9-2a2b-4a3d-afb7-c0d2ee617b7f-istio-podinfo\") pod \"42d727c9-2a2b-4a3d-afb7-c0d2ee617b7f\" (UID: \"42d727c9-2a2b-4a3d-afb7-c0d2ee617b7f\") " Nov 03 13:23:09.503959 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:23:09.503938 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/42d727c9-2a2b-4a3d-afb7-c0d2ee617b7f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42d727c9-2a2b-4a3d-afb7-c0d2ee617b7f-istiod-ca-cert\") pod \"42d727c9-2a2b-4a3d-afb7-c0d2ee617b7f\" (UID: \"42d727c9-2a2b-4a3d-afb7-c0d2ee617b7f\") " Nov 03 13:23:09.504123 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:23:09.504107 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42d727c9-2a2b-4a3d-afb7-c0d2ee617b7f-kube-api-access-flws7\" (UniqueName: \"kubernetes.io/projected/42d727c9-2a2b-4a3d-afb7-c0d2ee617b7f-kube-api-access-flws7\") pod \"42d727c9-2a2b-4a3d-afb7-c0d2ee617b7f\" (UID: \"42d727c9-2a2b-4a3d-afb7-c0d2ee617b7f\") " Nov 03 13:23:09.504440 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:23:09.504419 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42d727c9-2a2b-4a3d-afb7-c0d2ee617b7f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42d727c9-2a2b-4a3d-afb7-c0d2ee617b7f-credential-socket\") pod \"42d727c9-2a2b-4a3d-afb7-c0d2ee617b7f\" (UID: \"42d727c9-2a2b-4a3d-afb7-c0d2ee617b7f\") " Nov 03 13:23:09.505330 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:23:09.505309 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42d727c9-2a2b-4a3d-afb7-c0d2ee617b7f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42d727c9-2a2b-4a3d-afb7-c0d2ee617b7f-workload-socket\") pod \"42d727c9-2a2b-4a3d-afb7-c0d2ee617b7f\" (UID: \"42d727c9-2a2b-4a3d-afb7-c0d2ee617b7f\") " Nov 03 13:23:09.505585 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:23:09.505561 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42d727c9-2a2b-4a3d-afb7-c0d2ee617b7f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42d727c9-2a2b-4a3d-afb7-c0d2ee617b7f-istio-data\") pod \"42d727c9-2a2b-4a3d-afb7-c0d2ee617b7f\" (UID: \"42d727c9-2a2b-4a3d-afb7-c0d2ee617b7f\") " Nov 03 13:23:09.505764 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:23:09.505748 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42d727c9-2a2b-4a3d-afb7-c0d2ee617b7f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42d727c9-2a2b-4a3d-afb7-c0d2ee617b7f-workload-certs\") pod \"42d727c9-2a2b-4a3d-afb7-c0d2ee617b7f\" (UID: \"42d727c9-2a2b-4a3d-afb7-c0d2ee617b7f\") " Nov 03 13:23:09.505951 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:23:09.505935 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42d727c9-2a2b-4a3d-afb7-c0d2ee617b7f-istio-token\" (UniqueName: \"kubernetes.io/projected/42d727c9-2a2b-4a3d-afb7-c0d2ee617b7f-istio-token\") pod \"42d727c9-2a2b-4a3d-afb7-c0d2ee617b7f\" (UID: \"42d727c9-2a2b-4a3d-afb7-c0d2ee617b7f\") " Nov 03 13:23:09.524327 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:23:09.514211 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42d727c9-2a2b-4a3d-afb7-c0d2ee617b7f-credential-socket" pod "42d727c9-2a2b-4a3d-afb7-c0d2ee617b7f" (UID: "42d727c9-2a2b-4a3d-afb7-c0d2ee617b7f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:23:09.524327 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:23:09.515404 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42d727c9-2a2b-4a3d-afb7-c0d2ee617b7f-istio-data" pod "42d727c9-2a2b-4a3d-afb7-c0d2ee617b7f" (UID: "42d727c9-2a2b-4a3d-afb7-c0d2ee617b7f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:23:09.524327 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:23:09.515712 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42d727c9-2a2b-4a3d-afb7-c0d2ee617b7f-workload-socket" pod "42d727c9-2a2b-4a3d-afb7-c0d2ee617b7f" (UID: "42d727c9-2a2b-4a3d-afb7-c0d2ee617b7f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:23:09.524327 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:23:09.515971 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42d727c9-2a2b-4a3d-afb7-c0d2ee617b7f-istiod-ca-cert" pod "42d727c9-2a2b-4a3d-afb7-c0d2ee617b7f" (UID: "42d727c9-2a2b-4a3d-afb7-c0d2ee617b7f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:23:09.524327 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:23:09.523495 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42d727c9-2a2b-4a3d-afb7-c0d2ee617b7f-workload-certs" pod "42d727c9-2a2b-4a3d-afb7-c0d2ee617b7f" (UID: "42d727c9-2a2b-4a3d-afb7-c0d2ee617b7f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:23:09.529191 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:23:09.527964 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42d727c9-2a2b-4a3d-afb7-c0d2ee617b7f-kube-api-access-flws7" pod "42d727c9-2a2b-4a3d-afb7-c0d2ee617b7f" (UID: "42d727c9-2a2b-4a3d-afb7-c0d2ee617b7f"). InnerVolumeSpecName "kube-api-access-flws7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:23:09.529191 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:23:09.528554 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42d727c9-2a2b-4a3d-afb7-c0d2ee617b7f-istio-token" pod "42d727c9-2a2b-4a3d-afb7-c0d2ee617b7f" (UID: "42d727c9-2a2b-4a3d-afb7-c0d2ee617b7f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:23:09.530787 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:23:09.530736 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/42d727c9-2a2b-4a3d-afb7-c0d2ee617b7f-istio-podinfo" pod "42d727c9-2a2b-4a3d-afb7-c0d2ee617b7f" (UID: "42d727c9-2a2b-4a3d-afb7-c0d2ee617b7f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 13:23:09.536398 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:23:09.536369 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42d727c9-2a2b-4a3d-afb7-c0d2ee617b7f-istio-envoy" pod "42d727c9-2a2b-4a3d-afb7-c0d2ee617b7f" (UID: "42d727c9-2a2b-4a3d-afb7-c0d2ee617b7f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:23:09.560734 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:23:09.560657 5024 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d7mwz" containerName="extractor" Nov 03 13:23:09.607936 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:23:09.607908 5024 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42d727c9-2a2b-4a3d-afb7-c0d2ee617b7f-workload-certs\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:23:09.607936 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:23:09.607935 5024 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42d727c9-2a2b-4a3d-afb7-c0d2ee617b7f-istio-token\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:23:09.608105 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:23:09.607944 5024 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42d727c9-2a2b-4a3d-afb7-c0d2ee617b7f-istio-envoy\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:23:09.608105 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:23:09.607954 5024 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42d727c9-2a2b-4a3d-afb7-c0d2ee617b7f-istio-podinfo\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:23:09.608105 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:23:09.607962 5024 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42d727c9-2a2b-4a3d-afb7-c0d2ee617b7f-istiod-ca-cert\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:23:09.608105 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:23:09.607971 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-flws7\" (UniqueName: \"kubernetes.io/projected/42d727c9-2a2b-4a3d-afb7-c0d2ee617b7f-kube-api-access-flws7\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:23:09.608105 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:23:09.607981 5024 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42d727c9-2a2b-4a3d-afb7-c0d2ee617b7f-credential-socket\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:23:09.608105 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:23:09.607990 5024 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42d727c9-2a2b-4a3d-afb7-c0d2ee617b7f-workload-socket\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:23:09.608105 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:23:09.607999 5024 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42d727c9-2a2b-4a3d-afb7-c0d2ee617b7f-istio-data\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:23:10.360040 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:23:10.360014 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-17adbf-openshift-default-75f4498d49-hsbdd" Nov 03 13:23:10.398138 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:23:10.398063 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-17adbf-openshift-default-75f4498d49-hsbdd"] Nov 03 13:23:10.402916 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:23:10.402891 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-17adbf-openshift-default-75f4498d49-hsbdd"] Nov 03 13:23:10.811990 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:23:10.811909 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-db71f4-6b76577dd7-cgpqq"] Nov 03 13:23:12.146107 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:23:12.141947 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="42d727c9-2a2b-4a3d-afb7-c0d2ee617b7f" path="/var/lib/kubelet/pods/42d727c9-2a2b-4a3d-afb7-c0d2ee617b7f/volumes" Nov 03 13:23:14.920179 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:23:14.920147 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-17adbf-696b5d5d7f-9fqx5" Nov 03 13:23:14.959586 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:23:14.959560 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a046d933-94e8-4244-96c5-0ac0665bf579-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a046d933-94e8-4244-96c5-0ac0665bf579-plugins-config-volume\") pod \"a046d933-94e8-4244-96c5-0ac0665bf579\" (UID: \"a046d933-94e8-4244-96c5-0ac0665bf579\") " Nov 03 13:23:14.959805 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:23:14.959604 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a046d933-94e8-4244-96c5-0ac0665bf579-kube-api-access-ptwwj\" (UniqueName: \"kubernetes.io/projected/a046d933-94e8-4244-96c5-0ac0665bf579-kube-api-access-ptwwj\") pod \"a046d933-94e8-4244-96c5-0ac0665bf579\" (UID: \"a046d933-94e8-4244-96c5-0ac0665bf579\") " Nov 03 13:23:14.959982 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:23:14.959960 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a046d933-94e8-4244-96c5-0ac0665bf579-plugins-config-volume" pod "a046d933-94e8-4244-96c5-0ac0665bf579" (UID: "a046d933-94e8-4244-96c5-0ac0665bf579"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:23:14.960142 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:23:14.960125 5024 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a046d933-94e8-4244-96c5-0ac0665bf579-plugins-config-volume\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:23:14.965659 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:23:14.965635 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a046d933-94e8-4244-96c5-0ac0665bf579-kube-api-access-ptwwj" pod "a046d933-94e8-4244-96c5-0ac0665bf579" (UID: "a046d933-94e8-4244-96c5-0ac0665bf579"). InnerVolumeSpecName "kube-api-access-ptwwj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:23:15.061840 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:23:15.061805 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ptwwj\" (UniqueName: \"kubernetes.io/projected/a046d933-94e8-4244-96c5-0ac0665bf579-kube-api-access-ptwwj\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:23:15.400597 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:23:15.400560 5024 generic.go:356] "Generic (PLEG): container finished" podID="a046d933-94e8-4244-96c5-0ac0665bf579" containerID="f552a78cc62f8af156914e5e918b93737ffc4ddc4280d3982fd1e9e123f83560" exitCode=0 Nov 03 13:23:15.400793 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:23:15.400629 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-17adbf-696b5d5d7f-9fqx5" Nov 03 13:23:15.400793 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:23:15.400641 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-17adbf-696b5d5d7f-9fqx5" event={"ID":"a046d933-94e8-4244-96c5-0ac0665bf579","Type":"ContainerDied","Data":"f552a78cc62f8af156914e5e918b93737ffc4ddc4280d3982fd1e9e123f83560"} Nov 03 13:23:15.400793 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:23:15.400720 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-17adbf-696b5d5d7f-9fqx5" event={"ID":"a046d933-94e8-4244-96c5-0ac0665bf579","Type":"ContainerDied","Data":"c8b83058508331593dd3ebcd39117b236209a197c3e0b8fc00390265de6c5cb3"} Nov 03 13:23:15.400793 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:23:15.400743 5024 scope.go:122] "RemoveContainer" containerID="f552a78cc62f8af156914e5e918b93737ffc4ddc4280d3982fd1e9e123f83560" Nov 03 13:23:15.410531 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:23:15.410420 5024 scope.go:122] "RemoveContainer" containerID="f552a78cc62f8af156914e5e918b93737ffc4ddc4280d3982fd1e9e123f83560" Nov 03 13:23:15.410811 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:23:15.410720 5024 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f552a78cc62f8af156914e5e918b93737ffc4ddc4280d3982fd1e9e123f83560\": container with ID starting with f552a78cc62f8af156914e5e918b93737ffc4ddc4280d3982fd1e9e123f83560 not found: ID does not exist" containerID="f552a78cc62f8af156914e5e918b93737ffc4ddc4280d3982fd1e9e123f83560" Nov 03 13:23:15.410811 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:23:15.410755 5024 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f552a78cc62f8af156914e5e918b93737ffc4ddc4280d3982fd1e9e123f83560"} err="failed to get container status \"f552a78cc62f8af156914e5e918b93737ffc4ddc4280d3982fd1e9e123f83560\": rpc error: code = NotFound desc = could not find container \"f552a78cc62f8af156914e5e918b93737ffc4ddc4280d3982fd1e9e123f83560\": container with ID starting with f552a78cc62f8af156914e5e918b93737ffc4ddc4280d3982fd1e9e123f83560 not found: ID does not exist" Nov 03 13:23:15.428697 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:23:15.428676 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-17adbf-696b5d5d7f-9fqx5"] Nov 03 13:23:15.432440 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:23:15.432419 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-17adbf-696b5d5d7f-9fqx5"] Nov 03 13:23:15.812475 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:23:15.812368 5024 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-db71f4-6b76577dd7-cgpqq" podUID="4d8f419b-6347-4761-a260-a35769a80f40" containerName="payload-pre-processing" containerID="cri-o://797bc8e0c83843263f9ae2e8fb172dc13dbfd01bd394d49896c72afdf7fb94ec" gracePeriod=25 Nov 03 13:23:16.115123 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:23:16.115089 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a046d933-94e8-4244-96c5-0ac0665bf579" path="/var/lib/kubelet/pods/a046d933-94e8-4244-96c5-0ac0665bf579/volumes" Nov 03 13:23:16.546355 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:23:16.544307 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-db71f4-6b76577dd7-cgpqq" podUID="4d8f419b-6347-4761-a260-a35769a80f40" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 13:23:20.804140 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:23:20.804103 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-xrw9k"] Nov 03 13:23:20.805601 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:23:20.805568 5024 cpu_manager.go:414] "containerMap: removing container" podUID="42d727c9-2a2b-4a3d-afb7-c0d2ee617b7f" containerName="istio-proxy" Nov 03 13:23:20.805775 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:23:20.805759 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42d727c9-2a2b-4a3d-afb7-c0d2ee617b7f" containerName="istio-proxy" Nov 03 13:23:20.805882 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:23:20.805868 5024 cpu_manager.go:414] "containerMap: removing container" podUID="a046d933-94e8-4244-96c5-0ac0665bf579" containerName="payload-pre-processing" Nov 03 13:23:20.805970 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:23:20.805957 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a046d933-94e8-4244-96c5-0ac0665bf579" containerName="payload-pre-processing" Nov 03 13:23:20.806059 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:23:20.806046 5024 cpu_manager.go:414] "containerMap: removing container" podUID="029ef988-b71c-4de2-9b27-d7506a869ed8" containerName="main" Nov 03 13:23:20.808433 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:23:20.808403 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="029ef988-b71c-4de2-9b27-d7506a869ed8" containerName="main" Nov 03 13:23:20.808591 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:23:20.808575 5024 cpu_manager.go:414] "containerMap: removing container" podUID="23c40e8d-557b-43c0-a050-9d39181d67de" containerName="payload-processing" Nov 03 13:23:20.808689 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:23:20.808677 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23c40e8d-557b-43c0-a050-9d39181d67de" containerName="payload-processing" Nov 03 13:23:20.808781 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:23:20.808771 5024 cpu_manager.go:414] "containerMap: removing container" podUID="42d727c9-2a2b-4a3d-afb7-c0d2ee617b7f" containerName="istio-proxy" Nov 03 13:23:20.808862 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:23:20.808851 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42d727c9-2a2b-4a3d-afb7-c0d2ee617b7f" containerName="istio-proxy" Nov 03 13:23:20.809148 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:23:20.809129 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="42d727c9-2a2b-4a3d-afb7-c0d2ee617b7f" containerName="istio-proxy" Nov 03 13:23:20.809255 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:23:20.809243 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="a046d933-94e8-4244-96c5-0ac0665bf579" containerName="payload-pre-processing" Nov 03 13:23:20.809362 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:23:20.809350 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="029ef988-b71c-4de2-9b27-d7506a869ed8" containerName="main" Nov 03 13:23:20.809457 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:23:20.809446 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="23c40e8d-557b-43c0-a050-9d39181d67de" containerName="payload-processing" Nov 03 13:23:20.815028 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:23:20.815008 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-xrw9k"] Nov 03 13:23:20.830418 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:23:20.818561 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-xrw9k" Nov 03 13:23:20.976361 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:23:20.974432 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e218218f-5605-485f-a27c-eb68f01c01a6-credential-socket\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-xrw9k\" (UID: \"e218218f-5605-485f-a27c-eb68f01c01a6\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-xrw9k" Nov 03 13:23:20.976361 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:23:20.974494 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e218218f-5605-485f-a27c-eb68f01c01a6-workload-certs\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-xrw9k\" (UID: \"e218218f-5605-485f-a27c-eb68f01c01a6\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-xrw9k" Nov 03 13:23:20.976361 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:23:20.974536 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e218218f-5605-485f-a27c-eb68f01c01a6-istio-podinfo\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-xrw9k\" (UID: \"e218218f-5605-485f-a27c-eb68f01c01a6\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-xrw9k" Nov 03 13:23:20.976361 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:23:20.974600 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e218218f-5605-485f-a27c-eb68f01c01a6-istio-token\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-xrw9k\" (UID: \"e218218f-5605-485f-a27c-eb68f01c01a6\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-xrw9k" Nov 03 13:23:20.976361 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:23:20.974693 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w9xsp\" (UniqueName: \"kubernetes.io/projected/e218218f-5605-485f-a27c-eb68f01c01a6-kube-api-access-w9xsp\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-xrw9k\" (UID: \"e218218f-5605-485f-a27c-eb68f01c01a6\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-xrw9k" Nov 03 13:23:20.976361 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:23:20.974742 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e218218f-5605-485f-a27c-eb68f01c01a6-workload-socket\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-xrw9k\" (UID: \"e218218f-5605-485f-a27c-eb68f01c01a6\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-xrw9k" Nov 03 13:23:20.976361 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:23:20.974799 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e218218f-5605-485f-a27c-eb68f01c01a6-istiod-ca-cert\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-xrw9k\" (UID: \"e218218f-5605-485f-a27c-eb68f01c01a6\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-xrw9k" Nov 03 13:23:20.976361 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:23:20.974830 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e218218f-5605-485f-a27c-eb68f01c01a6-istio-envoy\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-xrw9k\" (UID: \"e218218f-5605-485f-a27c-eb68f01c01a6\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-xrw9k" Nov 03 13:23:20.976361 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:23:20.974877 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e218218f-5605-485f-a27c-eb68f01c01a6-istio-data\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-xrw9k\" (UID: \"e218218f-5605-485f-a27c-eb68f01c01a6\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-xrw9k" Nov 03 13:23:21.086116 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:23:21.085191 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w9xsp\" (UniqueName: \"kubernetes.io/projected/e218218f-5605-485f-a27c-eb68f01c01a6-kube-api-access-w9xsp\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-xrw9k\" (UID: \"e218218f-5605-485f-a27c-eb68f01c01a6\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-xrw9k" Nov 03 13:23:21.086116 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:23:21.085265 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e218218f-5605-485f-a27c-eb68f01c01a6-workload-socket\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-xrw9k\" (UID: \"e218218f-5605-485f-a27c-eb68f01c01a6\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-xrw9k" Nov 03 13:23:21.086116 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:23:21.085342 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e218218f-5605-485f-a27c-eb68f01c01a6-istiod-ca-cert\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-xrw9k\" (UID: \"e218218f-5605-485f-a27c-eb68f01c01a6\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-xrw9k" Nov 03 13:23:21.086116 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:23:21.085376 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e218218f-5605-485f-a27c-eb68f01c01a6-istio-envoy\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-xrw9k\" (UID: \"e218218f-5605-485f-a27c-eb68f01c01a6\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-xrw9k" Nov 03 13:23:21.086116 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:23:21.085420 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e218218f-5605-485f-a27c-eb68f01c01a6-istio-data\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-xrw9k\" (UID: \"e218218f-5605-485f-a27c-eb68f01c01a6\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-xrw9k" Nov 03 13:23:21.086116 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:23:21.085562 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e218218f-5605-485f-a27c-eb68f01c01a6-credential-socket\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-xrw9k\" (UID: \"e218218f-5605-485f-a27c-eb68f01c01a6\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-xrw9k" Nov 03 13:23:21.086116 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:23:21.085592 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e218218f-5605-485f-a27c-eb68f01c01a6-workload-certs\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-xrw9k\" (UID: \"e218218f-5605-485f-a27c-eb68f01c01a6\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-xrw9k" Nov 03 13:23:21.086116 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:23:21.085637 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e218218f-5605-485f-a27c-eb68f01c01a6-istio-podinfo\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-xrw9k\" (UID: \"e218218f-5605-485f-a27c-eb68f01c01a6\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-xrw9k" Nov 03 13:23:21.086116 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:23:21.085699 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e218218f-5605-485f-a27c-eb68f01c01a6-istio-token\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-xrw9k\" (UID: \"e218218f-5605-485f-a27c-eb68f01c01a6\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-xrw9k" Nov 03 13:23:21.098815 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:23:21.098219 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e218218f-5605-485f-a27c-eb68f01c01a6-workload-socket\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-xrw9k\" (UID: \"e218218f-5605-485f-a27c-eb68f01c01a6\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-xrw9k" Nov 03 13:23:21.098981 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:23:21.098941 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e218218f-5605-485f-a27c-eb68f01c01a6-istiod-ca-cert\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-xrw9k\" (UID: \"e218218f-5605-485f-a27c-eb68f01c01a6\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-xrw9k" Nov 03 13:23:21.105317 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:23:21.099403 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e218218f-5605-485f-a27c-eb68f01c01a6-istio-data\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-xrw9k\" (UID: \"e218218f-5605-485f-a27c-eb68f01c01a6\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-xrw9k" Nov 03 13:23:21.126562 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:23:21.114462 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e218218f-5605-485f-a27c-eb68f01c01a6-credential-socket\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-xrw9k\" (UID: \"e218218f-5605-485f-a27c-eb68f01c01a6\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-xrw9k" Nov 03 13:23:21.140940 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:23:21.114791 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e218218f-5605-485f-a27c-eb68f01c01a6-workload-certs\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-xrw9k\" (UID: \"e218218f-5605-485f-a27c-eb68f01c01a6\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-xrw9k" Nov 03 13:23:21.166851 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:23:21.166817 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-d3ca48-openshift-default-dockercfg-lgz27\"" Nov 03 13:23:21.460260 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:23:21.460212 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e218218f-5605-485f-a27c-eb68f01c01a6-istio-envoy\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-xrw9k\" (UID: \"e218218f-5605-485f-a27c-eb68f01c01a6\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-xrw9k" Nov 03 13:23:21.494826 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:23:21.493437 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e218218f-5605-485f-a27c-eb68f01c01a6-istio-podinfo\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-xrw9k\" (UID: \"e218218f-5605-485f-a27c-eb68f01c01a6\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-xrw9k" Nov 03 13:23:21.494826 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:23:21.493897 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w9xsp\" (UniqueName: \"kubernetes.io/projected/e218218f-5605-485f-a27c-eb68f01c01a6-kube-api-access-w9xsp\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-xrw9k\" (UID: \"e218218f-5605-485f-a27c-eb68f01c01a6\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-xrw9k" Nov 03 13:23:21.495674 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:23:21.495628 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e218218f-5605-485f-a27c-eb68f01c01a6-istio-token\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-xrw9k\" (UID: \"e218218f-5605-485f-a27c-eb68f01c01a6\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-xrw9k" Nov 03 13:23:21.768035 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:23:21.767931 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-xrw9k" Nov 03 13:23:22.054793 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:23:22.054756 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode218218f_5605_485f_a27c_eb68f01c01a6.slice/crio-f8dca2598bf2af0366353e4c2091628ced1d134fc5f36cd02dd5554ca7819b25 WatchSource:0}: Error finding container f8dca2598bf2af0366353e4c2091628ced1d134fc5f36cd02dd5554ca7819b25: Status 404 returned error can't find the container with id f8dca2598bf2af0366353e4c2091628ced1d134fc5f36cd02dd5554ca7819b25 Nov 03 13:23:22.055794 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:23:22.055751 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-xrw9k"] Nov 03 13:23:22.057611 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:23:22.057572 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:23:22.057727 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:23:22.057644 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:23:22.057727 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:23:22.057679 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:23:22.713091 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:23:22.713054 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-xrw9k" event={"ID":"e218218f-5605-485f-a27c-eb68f01c01a6","Type":"ContainerStarted","Data":"16e134c028375600bf74362974e1500fece99126914878b251a8136f7788669c"} Nov 03 13:23:22.713315 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:23:22.713131 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-xrw9k" event={"ID":"e218218f-5605-485f-a27c-eb68f01c01a6","Type":"ContainerStarted","Data":"f8dca2598bf2af0366353e4c2091628ced1d134fc5f36cd02dd5554ca7819b25"} Nov 03 13:23:22.736651 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:23:22.736605 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-xrw9k" podStartSLOduration=2.736589912 podStartE2EDuration="2.736589912s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:23:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:23:22.735954905 +0000 UTC m=+882.744146182" watchObservedRunningTime="2026-11-03 13:23:22.736589912 +0000 UTC m=+882.744781188" Nov 03 13:23:22.768178 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:23:22.768153 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-xrw9k" Nov 03 13:23:23.773381 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:23:23.773354 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-xrw9k" Nov 03 13:23:24.727582 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:23:24.727551 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-xrw9k" Nov 03 13:23:24.728505 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:23:24.728481 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-xrw9k" Nov 03 13:23:26.541564 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:23:26.541522 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-db71f4-6b76577dd7-cgpqq" podUID="4d8f419b-6347-4761-a260-a35769a80f40" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 13:23:36.115347 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:23:36.114825 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-db71f4-6b76577dd7-cgpqq" Nov 03 13:23:36.208752 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:23:36.205130 5024 generic.go:356] "Generic (PLEG): container finished" podID="4d8f419b-6347-4761-a260-a35769a80f40" containerID="797bc8e0c83843263f9ae2e8fb172dc13dbfd01bd394d49896c72afdf7fb94ec" exitCode=0 Nov 03 13:23:36.209258 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:23:36.209222 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-db71f4-6b76577dd7-cgpqq" event={"ID":"4d8f419b-6347-4761-a260-a35769a80f40","Type":"ContainerDied","Data":"797bc8e0c83843263f9ae2e8fb172dc13dbfd01bd394d49896c72afdf7fb94ec"} Nov 03 13:23:36.211183 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:23:36.209613 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-db71f4-6b76577dd7-cgpqq" event={"ID":"4d8f419b-6347-4761-a260-a35769a80f40","Type":"ContainerDied","Data":"be9729557c89cb2710a5ff18b4c780c7646a5868e3f3162850a4ad1c22c286b2"} Nov 03 13:23:36.211415 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:23:36.211396 5024 scope.go:122] "RemoveContainer" containerID="797bc8e0c83843263f9ae2e8fb172dc13dbfd01bd394d49896c72afdf7fb94ec" Nov 03 13:23:36.211725 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:23:36.211706 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-db71f4-6b76577dd7-cgpqq" Nov 03 13:23:36.233387 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:23:36.232263 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d8f419b-6347-4761-a260-a35769a80f40-kube-api-access-4wr4c\" (UniqueName: \"kubernetes.io/projected/4d8f419b-6347-4761-a260-a35769a80f40-kube-api-access-4wr4c\") pod \"4d8f419b-6347-4761-a260-a35769a80f40\" (UID: \"4d8f419b-6347-4761-a260-a35769a80f40\") " Nov 03 13:23:36.233387 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:23:36.232417 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4d8f419b-6347-4761-a260-a35769a80f40-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d8f419b-6347-4761-a260-a35769a80f40-plugins-config-volume\") pod \"4d8f419b-6347-4761-a260-a35769a80f40\" (UID: \"4d8f419b-6347-4761-a260-a35769a80f40\") " Nov 03 13:23:36.233830 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:23:36.233804 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d8f419b-6347-4761-a260-a35769a80f40-plugins-config-volume" pod "4d8f419b-6347-4761-a260-a35769a80f40" (UID: "4d8f419b-6347-4761-a260-a35769a80f40"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:23:36.240457 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:23:36.236448 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d8f419b-6347-4761-a260-a35769a80f40-kube-api-access-4wr4c" pod "4d8f419b-6347-4761-a260-a35769a80f40" (UID: "4d8f419b-6347-4761-a260-a35769a80f40"). InnerVolumeSpecName "kube-api-access-4wr4c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:23:36.250527 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:23:36.250484 5024 scope.go:122] "RemoveContainer" containerID="797bc8e0c83843263f9ae2e8fb172dc13dbfd01bd394d49896c72afdf7fb94ec" Nov 03 13:23:36.250971 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:23:36.250945 5024 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"797bc8e0c83843263f9ae2e8fb172dc13dbfd01bd394d49896c72afdf7fb94ec\": container with ID starting with 797bc8e0c83843263f9ae2e8fb172dc13dbfd01bd394d49896c72afdf7fb94ec not found: ID does not exist" containerID="797bc8e0c83843263f9ae2e8fb172dc13dbfd01bd394d49896c72afdf7fb94ec" Nov 03 13:23:36.251045 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:23:36.250983 5024 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"797bc8e0c83843263f9ae2e8fb172dc13dbfd01bd394d49896c72afdf7fb94ec"} err="failed to get container status \"797bc8e0c83843263f9ae2e8fb172dc13dbfd01bd394d49896c72afdf7fb94ec\": rpc error: code = NotFound desc = could not find container \"797bc8e0c83843263f9ae2e8fb172dc13dbfd01bd394d49896c72afdf7fb94ec\": container with ID starting with 797bc8e0c83843263f9ae2e8fb172dc13dbfd01bd394d49896c72afdf7fb94ec not found: ID does not exist" Nov 03 13:23:36.334610 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:23:36.334502 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4wr4c\" (UniqueName: \"kubernetes.io/projected/4d8f419b-6347-4761-a260-a35769a80f40-kube-api-access-4wr4c\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:23:36.334610 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:23:36.334557 5024 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d8f419b-6347-4761-a260-a35769a80f40-plugins-config-volume\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:23:36.539365 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:23:36.539277 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-db71f4-6b76577dd7-cgpqq"] Nov 03 13:23:36.542396 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:23:36.542367 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-db71f4-6b76577dd7-cgpqq"] Nov 03 13:23:38.110113 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:23:38.110081 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4d8f419b-6347-4761-a260-a35769a80f40" path="/var/lib/kubelet/pods/4d8f419b-6347-4761-a260-a35769a80f40/volumes" Nov 03 13:23:39.507875 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:23:39.507830 5024 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d7mwz" containerName="extractor" Nov 03 13:23:40.794908 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:23:40.794880 5024 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6htht_cd91de08-084e-4667-b13f-2e07b3cff624/console-operator/1.log" Nov 03 13:23:40.798016 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:23:40.797994 5024 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6htht_cd91de08-084e-4667-b13f-2e07b3cff624/console-operator/1.log" Nov 03 13:23:48.468673 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:23:48.468637 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-ded926/payload-processing-external-model-e2e-extmod-a-ded926-7759ds2gl"] Nov 03 13:23:48.469817 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:23:48.469791 5024 cpu_manager.go:414] "containerMap: removing container" podUID="4d8f419b-6347-4761-a260-a35769a80f40" containerName="payload-pre-processing" Nov 03 13:23:48.469817 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:23:48.469814 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d8f419b-6347-4761-a260-a35769a80f40" containerName="payload-pre-processing" Nov 03 13:23:48.469984 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:23:48.469933 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="42d727c9-2a2b-4a3d-afb7-c0d2ee617b7f" containerName="istio-proxy" Nov 03 13:23:48.469984 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:23:48.469941 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d8f419b-6347-4761-a260-a35769a80f40" containerName="payload-pre-processing" Nov 03 13:23:48.525796 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:23:48.525760 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-ded926/payload-processing-external-model-e2e-extmod-a-ded926-7759ds2gl"] Nov 03 13:23:48.529975 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:23:48.525874 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-ded926/payload-processing-external-model-e2e-extmod-a-ded926-7759ds2gl" Nov 03 13:23:48.574724 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:23:48.574679 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bcff8324-c70b-49a7-80a0-8e4ebcf5b77e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-ded926-7759ds2gl\" (UID: \"bcff8324-c70b-49a7-80a0-8e4ebcf5b77e\") " pod="ai-tenant-e2e-extmod-a-ded926/payload-processing-external-model-e2e-extmod-a-ded926-7759ds2gl" Nov 03 13:23:48.594854 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:23:48.594817 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/bcff8324-c70b-49a7-80a0-8e4ebcf5b77e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-ded926-7759ds2gl\" (UID: \"bcff8324-c70b-49a7-80a0-8e4ebcf5b77e\") " pod="ai-tenant-e2e-extmod-a-ded926/payload-processing-external-model-e2e-extmod-a-ded926-7759ds2gl" Nov 03 13:23:48.595186 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:23:48.595165 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bcff8324-c70b-49a7-80a0-8e4ebcf5b77e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-ded926-7759ds2gl\" (UID: \"bcff8324-c70b-49a7-80a0-8e4ebcf5b77e\") " pod="ai-tenant-e2e-extmod-a-ded926/payload-processing-external-model-e2e-extmod-a-ded926-7759ds2gl" Nov 03 13:23:48.595362 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:23:48.595344 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wdppn\" (UniqueName: \"kubernetes.io/projected/bcff8324-c70b-49a7-80a0-8e4ebcf5b77e-kube-api-access-wdppn\") pod \"payload-processing-external-model-e2e-extmod-a-ded926-7759ds2gl\" (UID: \"bcff8324-c70b-49a7-80a0-8e4ebcf5b77e\") " pod="ai-tenant-e2e-extmod-a-ded926/payload-processing-external-model-e2e-extmod-a-ded926-7759ds2gl" Nov 03 13:23:48.596014 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:23:48.594518 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-ded926\"/\"openshift-service-ca.crt\"" Nov 03 13:23:48.596140 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:23:48.584951 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-ded926\"/\"payload-processing-external-model-plugins-e2e-extmod-a-ded926\"" Nov 03 13:23:48.596272 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:23:48.587510 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-ded926\"/\"agc-cred-58029127f63d\"" Nov 03 13:23:48.596397 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:23:48.587606 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-ded926\"/\"kube-root-ca.crt\"" Nov 03 13:23:48.596483 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:23:48.594598 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-ded926\"/\"payload-processing-external-model-e2e--61c7371d-dockercfg-slfl7\"" Nov 03 13:23:48.596555 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:23:48.594649 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-ded926\"/\"routing-overlay\"" Nov 03 13:23:48.703338 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:23:48.702577 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bcff8324-c70b-49a7-80a0-8e4ebcf5b77e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-ded926-7759ds2gl\" (UID: \"bcff8324-c70b-49a7-80a0-8e4ebcf5b77e\") " pod="ai-tenant-e2e-extmod-a-ded926/payload-processing-external-model-e2e-extmod-a-ded926-7759ds2gl" Nov 03 13:23:48.703338 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:23:48.702716 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/bcff8324-c70b-49a7-80a0-8e4ebcf5b77e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-ded926-7759ds2gl\" (UID: \"bcff8324-c70b-49a7-80a0-8e4ebcf5b77e\") " pod="ai-tenant-e2e-extmod-a-ded926/payload-processing-external-model-e2e-extmod-a-ded926-7759ds2gl" Nov 03 13:23:48.703338 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:23:48.702823 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bcff8324-c70b-49a7-80a0-8e4ebcf5b77e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-ded926-7759ds2gl\" (UID: \"bcff8324-c70b-49a7-80a0-8e4ebcf5b77e\") " pod="ai-tenant-e2e-extmod-a-ded926/payload-processing-external-model-e2e-extmod-a-ded926-7759ds2gl" Nov 03 13:23:48.703338 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:23:48.702884 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wdppn\" (UniqueName: \"kubernetes.io/projected/bcff8324-c70b-49a7-80a0-8e4ebcf5b77e-kube-api-access-wdppn\") pod \"payload-processing-external-model-e2e-extmod-a-ded926-7759ds2gl\" (UID: \"bcff8324-c70b-49a7-80a0-8e4ebcf5b77e\") " pod="ai-tenant-e2e-extmod-a-ded926/payload-processing-external-model-e2e-extmod-a-ded926-7759ds2gl" Nov 03 13:23:48.712805 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:23:48.712772 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bcff8324-c70b-49a7-80a0-8e4ebcf5b77e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-ded926-7759ds2gl\" (UID: \"bcff8324-c70b-49a7-80a0-8e4ebcf5b77e\") " pod="ai-tenant-e2e-extmod-a-ded926/payload-processing-external-model-e2e-extmod-a-ded926-7759ds2gl" Nov 03 13:23:48.713872 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:23:48.713845 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bcff8324-c70b-49a7-80a0-8e4ebcf5b77e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-ded926-7759ds2gl\" (UID: \"bcff8324-c70b-49a7-80a0-8e4ebcf5b77e\") " pod="ai-tenant-e2e-extmod-a-ded926/payload-processing-external-model-e2e-extmod-a-ded926-7759ds2gl" Nov 03 13:23:48.718992 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:23:48.718893 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/bcff8324-c70b-49a7-80a0-8e4ebcf5b77e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-ded926-7759ds2gl\" (UID: \"bcff8324-c70b-49a7-80a0-8e4ebcf5b77e\") " pod="ai-tenant-e2e-extmod-a-ded926/payload-processing-external-model-e2e-extmod-a-ded926-7759ds2gl" Nov 03 13:23:48.738317 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:23:48.738183 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wdppn\" (UniqueName: \"kubernetes.io/projected/bcff8324-c70b-49a7-80a0-8e4ebcf5b77e-kube-api-access-wdppn\") pod \"payload-processing-external-model-e2e-extmod-a-ded926-7759ds2gl\" (UID: \"bcff8324-c70b-49a7-80a0-8e4ebcf5b77e\") " pod="ai-tenant-e2e-extmod-a-ded926/payload-processing-external-model-e2e-extmod-a-ded926-7759ds2gl" Nov 03 13:23:48.983755 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:23:48.983665 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-ded926/payload-processing-external-model-e2e-extmod-a-ded926-7759ds2gl" Nov 03 13:23:49.355661 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:23:49.354582 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-ded926/payload-processing-external-model-e2e-extmod-a-ded926-7759ds2gl"] Nov 03 13:23:49.571496 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:23:49.571445 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-ded926/payload-processing-external-model-e2e-extmod-a-ded926-7759ds2gl" event={"ID":"bcff8324-c70b-49a7-80a0-8e4ebcf5b77e","Type":"ContainerStarted","Data":"d7a584cce6e2557951b6b1f7fa8bcf569ccda4ee5731c41472b35fa91b7582f8"} Nov 03 13:23:49.572073 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:23:49.572043 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-ded926/payload-processing-external-model-e2e-extmod-a-ded926-7759ds2gl" Nov 03 13:23:49.572203 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:23:49.572086 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-ded926/payload-processing-external-model-e2e-extmod-a-ded926-7759ds2gl" event={"ID":"bcff8324-c70b-49a7-80a0-8e4ebcf5b77e","Type":"ContainerStarted","Data":"3df03026f34c146846962fb5b5fdc08d4d28ec8c006e4acc10291e14370447c1"} Nov 03 13:23:49.594243 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:23:49.594154 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-ded926/payload-processing-external-model-e2e-extmod-a-ded926-7759ds2gl" podStartSLOduration=1.594131274 podStartE2EDuration="1.594131274s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:23:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:23:49.591555403 +0000 UTC m=+909.599746681" watchObservedRunningTime="2026-11-03 13:23:49.594131274 +0000 UTC m=+909.602322551" Nov 03 13:23:53.638061 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:23:53.637984 5024 generic.go:356] "Generic (PLEG): container finished" podID="e218218f-5605-485f-a27c-eb68f01c01a6" containerID="16e134c028375600bf74362974e1500fece99126914878b251a8136f7788669c" exitCode=0 Nov 03 13:23:53.638532 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:23:53.638054 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-xrw9k" event={"ID":"e218218f-5605-485f-a27c-eb68f01c01a6","Type":"ContainerDied","Data":"16e134c028375600bf74362974e1500fece99126914878b251a8136f7788669c"} Nov 03 13:23:53.638586 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:23:53.638531 5024 scope.go:122] "RemoveContainer" containerID="16e134c028375600bf74362974e1500fece99126914878b251a8136f7788669c" Nov 03 13:23:53.640096 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:23:53.640072 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:23:53.640156 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:23:53.640136 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:23:53.640191 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:23:53.640168 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:23:54.206840 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:23:54.206802 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-ded926/payload-processing-external-model-e2e-extmod-a-ded926-7759ds2gl"] Nov 03 13:23:54.210328 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:23:54.210268 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-ded926/payload-processing-external-model-e2e-extmod-a-ded926-7759ds2gl" Nov 03 13:23:54.735309 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:23:54.734427 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-xrw9k" Nov 03 13:23:54.766309 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:23:54.753955 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-xrw9k" event={"ID":"e218218f-5605-485f-a27c-eb68f01c01a6","Type":"ContainerStarted","Data":"cfc03a1da0578b9dd7718f73f17c1aed80b383d1cdfbe54a74c7fe51347d3896"} Nov 03 13:23:54.784310 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:23:54.778012 5024 patch_prober.go:29] interesting pod/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-xrw9k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Nov 03 13:23:54.784310 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:23:54.778120 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-xrw9k" podUID="e218218f-5605-485f-a27c-eb68f01c01a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Nov 03 13:23:54.790315 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:23:54.788669 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-xrw9k" Nov 03 13:23:55.780325 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:23:55.779717 5024 patch_prober.go:29] interesting pod/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-xrw9k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Nov 03 13:23:55.780325 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:23:55.779795 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-xrw9k" podUID="e218218f-5605-485f-a27c-eb68f01c01a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Nov 03 13:23:56.768360 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:23:56.768300 5024 patch_prober.go:29] interesting pod/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-xrw9k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Nov 03 13:23:56.768548 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:23:56.768368 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-xrw9k" podUID="e218218f-5605-485f-a27c-eb68f01c01a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Nov 03 13:23:57.217745 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:23:57.217705 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-praxis-a-a2ccd8/praxis-route-a2ccd8-kserve-748544bd55-8rmdm"] Nov 03 13:23:57.774307 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:23:57.768822 5024 patch_prober.go:29] interesting pod/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-xrw9k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Nov 03 13:23:57.774307 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:23:57.768929 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-xrw9k" podUID="e218218f-5605-485f-a27c-eb68f01c01a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Nov 03 13:23:58.768744 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:23:58.768702 5024 patch_prober.go:29] interesting pod/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-xrw9k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Nov 03 13:23:58.769414 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:23:58.769383 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-xrw9k" podUID="e218218f-5605-485f-a27c-eb68f01c01a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Nov 03 13:23:59.217312 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:23:59.214275 5024 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-ded926/payload-processing-external-model-e2e-extmod-a-ded926-7759ds2gl" podUID="bcff8324-c70b-49a7-80a0-8e4ebcf5b77e" containerName="payload-processing" containerID="cri-o://d7a584cce6e2557951b6b1f7fa8bcf569ccda4ee5731c41472b35fa91b7582f8" gracePeriod=25 Nov 03 13:23:59.263934 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:23:59.258766 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-praxis-a-a2ccd8/praxis-route-a2ccd8-kserve-748544bd55-8rmdm" podUID="11621285-51db-4e67-903d-311f39837fc9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded" Nov 03 13:23:59.772370 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:23:59.768730 5024 patch_prober.go:29] interesting pod/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-xrw9k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Nov 03 13:23:59.772370 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:23:59.768800 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-xrw9k" podUID="e218218f-5605-485f-a27c-eb68f01c01a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Nov 03 13:23:59.900305 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:23:59.895010 5024 generic.go:356] "Generic (PLEG): container finished" podID="bcff8324-c70b-49a7-80a0-8e4ebcf5b77e" containerID="d7a584cce6e2557951b6b1f7fa8bcf569ccda4ee5731c41472b35fa91b7582f8" exitCode=0 Nov 03 13:23:59.900305 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:23:59.895179 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-ded926/payload-processing-external-model-e2e-extmod-a-ded926-7759ds2gl" event={"ID":"bcff8324-c70b-49a7-80a0-8e4ebcf5b77e","Type":"ContainerDied","Data":"d7a584cce6e2557951b6b1f7fa8bcf569ccda4ee5731c41472b35fa91b7582f8"} Nov 03 13:24:00.321336 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:24:00.294026 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-praxis-a-a2ccd8/praxis-route-a2ccd8-kserve-748544bd55-8rmdm" podUID="11621285-51db-4e67-903d-311f39837fc9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded" Nov 03 13:24:00.855111 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:24:00.855072 5024 patch_prober.go:29] interesting pod/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-xrw9k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Nov 03 13:24:00.952638 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:24:00.952520 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-xrw9k" podUID="e218218f-5605-485f-a27c-eb68f01c01a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Nov 03 13:24:01.281390 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:24:01.235122 5024 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.112s" Nov 03 13:24:01.281390 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:24:01.235256 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-praxis-a-a2ccd8/praxis-route-a2ccd8-kserve-748544bd55-8rmdm" Nov 03 13:24:01.294577 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:24:01.294518 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-praxis-a-a2ccd8/praxis-route-a2ccd8-kserve-748544bd55-8rmdm" podUID="11621285-51db-4e67-903d-311f39837fc9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 13:24:01.334438 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:24:01.324491 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-ded926/payload-processing-external-model-e2e-extmod-a-ded926-7759ds2gl" Nov 03 13:24:01.380341 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:24:01.347488 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bcff8324-c70b-49a7-80a0-8e4ebcf5b77e-kube-api-access-wdppn\" (UniqueName: \"kubernetes.io/projected/bcff8324-c70b-49a7-80a0-8e4ebcf5b77e-kube-api-access-wdppn\") pod \"bcff8324-c70b-49a7-80a0-8e4ebcf5b77e\" (UID: \"bcff8324-c70b-49a7-80a0-8e4ebcf5b77e\") " Nov 03 13:24:01.380883 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:24:01.380842 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bcff8324-c70b-49a7-80a0-8e4ebcf5b77e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bcff8324-c70b-49a7-80a0-8e4ebcf5b77e-plugins-config-volume\") pod \"bcff8324-c70b-49a7-80a0-8e4ebcf5b77e\" (UID: \"bcff8324-c70b-49a7-80a0-8e4ebcf5b77e\") " Nov 03 13:24:01.382670 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:24:01.381548 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bcff8324-c70b-49a7-80a0-8e4ebcf5b77e-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bcff8324-c70b-49a7-80a0-8e4ebcf5b77e-routing-overlay-volume\") pod \"bcff8324-c70b-49a7-80a0-8e4ebcf5b77e\" (UID: \"bcff8324-c70b-49a7-80a0-8e4ebcf5b77e\") " Nov 03 13:24:01.382670 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:24:01.381614 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bcff8324-c70b-49a7-80a0-8e4ebcf5b77e-provider-credentials\" (UniqueName: \"kubernetes.io/projected/bcff8324-c70b-49a7-80a0-8e4ebcf5b77e-provider-credentials\") pod \"bcff8324-c70b-49a7-80a0-8e4ebcf5b77e\" (UID: \"bcff8324-c70b-49a7-80a0-8e4ebcf5b77e\") " Nov 03 13:24:01.442703 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:24:01.442633 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bcff8324-c70b-49a7-80a0-8e4ebcf5b77e-plugins-config-volume" pod "bcff8324-c70b-49a7-80a0-8e4ebcf5b77e" (UID: "bcff8324-c70b-49a7-80a0-8e4ebcf5b77e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:24:01.452354 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:24:01.452312 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bcff8324-c70b-49a7-80a0-8e4ebcf5b77e-routing-overlay-volume" pod "bcff8324-c70b-49a7-80a0-8e4ebcf5b77e" (UID: "bcff8324-c70b-49a7-80a0-8e4ebcf5b77e"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:24:01.488025 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:24:01.487956 5024 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bcff8324-c70b-49a7-80a0-8e4ebcf5b77e-plugins-config-volume\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:24:01.488025 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:24:01.487995 5024 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bcff8324-c70b-49a7-80a0-8e4ebcf5b77e-routing-overlay-volume\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:24:01.593200 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:24:01.593107 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-ded926/payload-processing-external-model-e2e-extmod-a-ded926-7759ds2gl" event={"ID":"bcff8324-c70b-49a7-80a0-8e4ebcf5b77e","Type":"ContainerDied","Data":"3df03026f34c146846962fb5b5fdc08d4d28ec8c006e4acc10291e14370447c1"} Nov 03 13:24:01.594867 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:24:01.594834 5024 scope.go:122] "RemoveContainer" containerID="d7a584cce6e2557951b6b1f7fa8bcf569ccda4ee5731c41472b35fa91b7582f8" Nov 03 13:24:01.617537 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:24:01.599587 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-ded926/payload-processing-external-model-e2e-extmod-a-ded926-7759ds2gl" Nov 03 13:24:01.780784 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:24:01.780742 5024 patch_prober.go:29] interesting pod/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-xrw9k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Nov 03 13:24:01.781052 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:24:01.781023 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-xrw9k" podUID="e218218f-5605-485f-a27c-eb68f01c01a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Nov 03 13:24:01.808332 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:24:01.795577 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bcff8324-c70b-49a7-80a0-8e4ebcf5b77e-kube-api-access-wdppn" pod "bcff8324-c70b-49a7-80a0-8e4ebcf5b77e" (UID: "bcff8324-c70b-49a7-80a0-8e4ebcf5b77e"). InnerVolumeSpecName "kube-api-access-wdppn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:24:01.860367 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:24:01.852715 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bcff8324-c70b-49a7-80a0-8e4ebcf5b77e-provider-credentials" pod "bcff8324-c70b-49a7-80a0-8e4ebcf5b77e" (UID: "bcff8324-c70b-49a7-80a0-8e4ebcf5b77e"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:24:01.894853 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:24:01.894814 5024 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/bcff8324-c70b-49a7-80a0-8e4ebcf5b77e-provider-credentials\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:24:01.895195 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:24:01.895169 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wdppn\" (UniqueName: \"kubernetes.io/projected/bcff8324-c70b-49a7-80a0-8e4ebcf5b77e-kube-api-access-wdppn\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:24:02.338534 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:24:02.338501 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-praxis-a-a2ccd8/praxis-route-a2ccd8-kserve-748544bd55-8rmdm" Nov 03 13:24:02.379531 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:24:02.379491 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-ded926/payload-processing-external-model-e2e-extmod-a-ded926-7759ds2gl"] Nov 03 13:24:02.393307 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:24:02.390182 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-ded926/payload-processing-external-model-e2e-extmod-a-ded926-7759ds2gl"] Nov 03 13:24:02.773782 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:24:02.773740 5024 patch_prober.go:29] interesting pod/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-xrw9k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Nov 03 13:24:02.774080 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:24:02.774046 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-xrw9k" podUID="e218218f-5605-485f-a27c-eb68f01c01a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Nov 03 13:24:03.769389 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:24:03.769347 5024 patch_prober.go:29] interesting pod/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-xrw9k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Nov 03 13:24:03.769778 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:24:03.769434 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-xrw9k" podUID="e218218f-5605-485f-a27c-eb68f01c01a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Nov 03 13:24:04.110107 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:24:04.110071 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bcff8324-c70b-49a7-80a0-8e4ebcf5b77e" path="/var/lib/kubelet/pods/bcff8324-c70b-49a7-80a0-8e4ebcf5b77e/volumes" Nov 03 13:24:04.773614 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:24:04.773577 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-xrw9k" Nov 03 13:24:04.774027 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:24:04.773983 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-xrw9k" Nov 03 13:24:04.774822 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:24:04.774800 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-xrw9k" Nov 03 13:24:05.409780 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:24:05.409747 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-d3ca48/premium-w3-d3ca48-kserve-75ffc959b-zvjbc"] Nov 03 13:24:05.410580 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:24:05.410556 5024 cpu_manager.go:414] "containerMap: removing container" podUID="bcff8324-c70b-49a7-80a0-8e4ebcf5b77e" containerName="payload-processing" Nov 03 13:24:05.410580 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:24:05.410576 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bcff8324-c70b-49a7-80a0-8e4ebcf5b77e" containerName="payload-processing" Nov 03 13:24:05.410772 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:24:05.410700 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="bcff8324-c70b-49a7-80a0-8e4ebcf5b77e" containerName="payload-processing" Nov 03 13:24:05.416191 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:24:05.416169 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-d3ca48/premium-w3-d3ca48-kserve-75ffc959b-zvjbc" Nov 03 13:24:05.432980 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:24:05.432452 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-d3ca48\"/\"openshift-service-ca.crt\"" Nov 03 13:24:05.432980 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:24:05.432542 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-d3ca48\"/\"kube-root-ca.crt\"" Nov 03 13:24:05.443250 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:24:05.443211 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-d3ca48/premium-w3-d3ca48-kserve-75ffc959b-zvjbc"] Nov 03 13:24:05.462219 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:24:05.462178 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-d3ca48\"/\"default-dockercfg-wf2rg\"" Nov 03 13:24:05.462461 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:24:05.462353 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-d3ca48\"/\"premium-w3-d3ca48-kserve-self-signed-certs\"" Nov 03 13:24:05.605695 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:24:05.605639 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/add1e8b9-e7fd-4e4d-8d10-f97d0e2612ba-model-cache\") pod \"premium-w3-d3ca48-kserve-75ffc959b-zvjbc\" (UID: \"add1e8b9-e7fd-4e4d-8d10-f97d0e2612ba\") " pod="e2e-models-e2e-worker-w3-d3ca48/premium-w3-d3ca48-kserve-75ffc959b-zvjbc" Nov 03 13:24:05.605911 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:24:05.605883 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/add1e8b9-e7fd-4e4d-8d10-f97d0e2612ba-tls-certs\") pod \"premium-w3-d3ca48-kserve-75ffc959b-zvjbc\" (UID: \"add1e8b9-e7fd-4e4d-8d10-f97d0e2612ba\") " pod="e2e-models-e2e-worker-w3-d3ca48/premium-w3-d3ca48-kserve-75ffc959b-zvjbc" Nov 03 13:24:05.606052 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:24:05.606032 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/add1e8b9-e7fd-4e4d-8d10-f97d0e2612ba-home\") pod \"premium-w3-d3ca48-kserve-75ffc959b-zvjbc\" (UID: \"add1e8b9-e7fd-4e4d-8d10-f97d0e2612ba\") " pod="e2e-models-e2e-worker-w3-d3ca48/premium-w3-d3ca48-kserve-75ffc959b-zvjbc" Nov 03 13:24:05.606120 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:24:05.606082 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/add1e8b9-e7fd-4e4d-8d10-f97d0e2612ba-dshm\") pod \"premium-w3-d3ca48-kserve-75ffc959b-zvjbc\" (UID: \"add1e8b9-e7fd-4e4d-8d10-f97d0e2612ba\") " pod="e2e-models-e2e-worker-w3-d3ca48/premium-w3-d3ca48-kserve-75ffc959b-zvjbc" Nov 03 13:24:05.606172 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:24:05.606150 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4v9bn\" (UniqueName: \"kubernetes.io/projected/add1e8b9-e7fd-4e4d-8d10-f97d0e2612ba-kube-api-access-4v9bn\") pod \"premium-w3-d3ca48-kserve-75ffc959b-zvjbc\" (UID: \"add1e8b9-e7fd-4e4d-8d10-f97d0e2612ba\") " pod="e2e-models-e2e-worker-w3-d3ca48/premium-w3-d3ca48-kserve-75ffc959b-zvjbc" Nov 03 13:24:05.606224 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:24:05.606209 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/add1e8b9-e7fd-4e4d-8d10-f97d0e2612ba-tmp-dir\") pod \"premium-w3-d3ca48-kserve-75ffc959b-zvjbc\" (UID: \"add1e8b9-e7fd-4e4d-8d10-f97d0e2612ba\") " pod="e2e-models-e2e-worker-w3-d3ca48/premium-w3-d3ca48-kserve-75ffc959b-zvjbc" Nov 03 13:24:05.709465 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:24:05.709135 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/add1e8b9-e7fd-4e4d-8d10-f97d0e2612ba-model-cache\") pod \"premium-w3-d3ca48-kserve-75ffc959b-zvjbc\" (UID: \"add1e8b9-e7fd-4e4d-8d10-f97d0e2612ba\") " pod="e2e-models-e2e-worker-w3-d3ca48/premium-w3-d3ca48-kserve-75ffc959b-zvjbc" Nov 03 13:24:05.709465 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:24:05.709337 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/add1e8b9-e7fd-4e4d-8d10-f97d0e2612ba-tls-certs\") pod \"premium-w3-d3ca48-kserve-75ffc959b-zvjbc\" (UID: \"add1e8b9-e7fd-4e4d-8d10-f97d0e2612ba\") " pod="e2e-models-e2e-worker-w3-d3ca48/premium-w3-d3ca48-kserve-75ffc959b-zvjbc" Nov 03 13:24:05.710035 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:24:05.709451 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/add1e8b9-e7fd-4e4d-8d10-f97d0e2612ba-home\") pod \"premium-w3-d3ca48-kserve-75ffc959b-zvjbc\" (UID: \"add1e8b9-e7fd-4e4d-8d10-f97d0e2612ba\") " pod="e2e-models-e2e-worker-w3-d3ca48/premium-w3-d3ca48-kserve-75ffc959b-zvjbc" Nov 03 13:24:05.710035 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:24:05.709516 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/add1e8b9-e7fd-4e4d-8d10-f97d0e2612ba-dshm\") pod \"premium-w3-d3ca48-kserve-75ffc959b-zvjbc\" (UID: \"add1e8b9-e7fd-4e4d-8d10-f97d0e2612ba\") " pod="e2e-models-e2e-worker-w3-d3ca48/premium-w3-d3ca48-kserve-75ffc959b-zvjbc" Nov 03 13:24:05.710035 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:24:05.709547 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4v9bn\" (UniqueName: \"kubernetes.io/projected/add1e8b9-e7fd-4e4d-8d10-f97d0e2612ba-kube-api-access-4v9bn\") pod \"premium-w3-d3ca48-kserve-75ffc959b-zvjbc\" (UID: \"add1e8b9-e7fd-4e4d-8d10-f97d0e2612ba\") " pod="e2e-models-e2e-worker-w3-d3ca48/premium-w3-d3ca48-kserve-75ffc959b-zvjbc" Nov 03 13:24:05.710035 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:24:05.709584 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/add1e8b9-e7fd-4e4d-8d10-f97d0e2612ba-tmp-dir\") pod \"premium-w3-d3ca48-kserve-75ffc959b-zvjbc\" (UID: \"add1e8b9-e7fd-4e4d-8d10-f97d0e2612ba\") " pod="e2e-models-e2e-worker-w3-d3ca48/premium-w3-d3ca48-kserve-75ffc959b-zvjbc" Nov 03 13:24:05.710240 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:24:05.710083 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/add1e8b9-e7fd-4e4d-8d10-f97d0e2612ba-tmp-dir\") pod \"premium-w3-d3ca48-kserve-75ffc959b-zvjbc\" (UID: \"add1e8b9-e7fd-4e4d-8d10-f97d0e2612ba\") " pod="e2e-models-e2e-worker-w3-d3ca48/premium-w3-d3ca48-kserve-75ffc959b-zvjbc" Nov 03 13:24:05.717307 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:24:05.710351 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/add1e8b9-e7fd-4e4d-8d10-f97d0e2612ba-model-cache\") pod \"premium-w3-d3ca48-kserve-75ffc959b-zvjbc\" (UID: \"add1e8b9-e7fd-4e4d-8d10-f97d0e2612ba\") " pod="e2e-models-e2e-worker-w3-d3ca48/premium-w3-d3ca48-kserve-75ffc959b-zvjbc" Nov 03 13:24:05.717307 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:24:05.711224 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/add1e8b9-e7fd-4e4d-8d10-f97d0e2612ba-home\") pod \"premium-w3-d3ca48-kserve-75ffc959b-zvjbc\" (UID: \"add1e8b9-e7fd-4e4d-8d10-f97d0e2612ba\") " pod="e2e-models-e2e-worker-w3-d3ca48/premium-w3-d3ca48-kserve-75ffc959b-zvjbc" Nov 03 13:24:05.735310 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:24:05.727557 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/add1e8b9-e7fd-4e4d-8d10-f97d0e2612ba-tls-certs\") pod \"premium-w3-d3ca48-kserve-75ffc959b-zvjbc\" (UID: \"add1e8b9-e7fd-4e4d-8d10-f97d0e2612ba\") " pod="e2e-models-e2e-worker-w3-d3ca48/premium-w3-d3ca48-kserve-75ffc959b-zvjbc" Nov 03 13:24:05.735310 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:24:05.734635 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/add1e8b9-e7fd-4e4d-8d10-f97d0e2612ba-dshm\") pod \"premium-w3-d3ca48-kserve-75ffc959b-zvjbc\" (UID: \"add1e8b9-e7fd-4e4d-8d10-f97d0e2612ba\") " pod="e2e-models-e2e-worker-w3-d3ca48/premium-w3-d3ca48-kserve-75ffc959b-zvjbc" Nov 03 13:24:05.736200 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:24:05.736169 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4v9bn\" (UniqueName: \"kubernetes.io/projected/add1e8b9-e7fd-4e4d-8d10-f97d0e2612ba-kube-api-access-4v9bn\") pod \"premium-w3-d3ca48-kserve-75ffc959b-zvjbc\" (UID: \"add1e8b9-e7fd-4e4d-8d10-f97d0e2612ba\") " pod="e2e-models-e2e-worker-w3-d3ca48/premium-w3-d3ca48-kserve-75ffc959b-zvjbc" Nov 03 13:24:05.747587 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:24:05.747552 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-d3ca48/premium-w3-d3ca48-kserve-75ffc959b-zvjbc" Nov 03 13:24:06.137908 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:24:06.137875 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-d3ca48/premium-w3-d3ca48-kserve-75ffc959b-zvjbc"] Nov 03 13:24:06.138713 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:24:06.138683 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podadd1e8b9_e7fd_4e4d_8d10_f97d0e2612ba.slice/crio-13487f6b814c734a9bb764f83259ef7f6816e8cfa6e7aa3b2a940e3922c94450 WatchSource:0}: Error finding container 13487f6b814c734a9bb764f83259ef7f6816e8cfa6e7aa3b2a940e3922c94450: Status 404 returned error can't find the container with id 13487f6b814c734a9bb764f83259ef7f6816e8cfa6e7aa3b2a940e3922c94450 Nov 03 13:24:06.825038 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:24:06.824366 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-d3ca48/premium-w3-d3ca48-kserve-75ffc959b-zvjbc" event={"ID":"add1e8b9-e7fd-4e4d-8d10-f97d0e2612ba","Type":"ContainerStarted","Data":"b4a70cd448c0750951bc94ce807bda6662f9f5085e553ef4d4b687941d8f8fb7"} Nov 03 13:24:06.825038 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:24:06.824419 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-d3ca48/premium-w3-d3ca48-kserve-75ffc959b-zvjbc" event={"ID":"add1e8b9-e7fd-4e4d-8d10-f97d0e2612ba","Type":"ContainerStarted","Data":"13487f6b814c734a9bb764f83259ef7f6816e8cfa6e7aa3b2a940e3922c94450"} Nov 03 13:24:06.844186 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:24:06.844132 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-d3ca48/premium-w3-d3ca48-kserve-75ffc959b-zvjbc" podStartSLOduration=1.844118074 podStartE2EDuration="1.844118074s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:24:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:24:06.844102196 +0000 UTC m=+926.852293474" watchObservedRunningTime="2026-11-03 13:24:06.844118074 +0000 UTC m=+926.852309350" Nov 03 13:24:07.060154 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:24:07.060122 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-8444x4nfn"] Nov 03 13:24:07.064387 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:24:07.064367 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-8444x4nfn" Nov 03 13:24:07.070353 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:24:07.070254 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-7c192b\"/\"agc-cred-384e44d183e3\"" Nov 03 13:24:07.070603 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:24:07.070574 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-7c192b\"/\"agc-cred-29dd648f7d07\"" Nov 03 13:24:07.075736 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:24:07.075671 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-8444x4nfn"] Nov 03 13:24:07.076429 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:24:07.076406 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-7c192b\"/\"agc-cred-06e7699b6ab0\"" Nov 03 13:24:07.076538 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:24:07.076452 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-7c192b\"/\"payload-processing-external-model-plugins-e2e-extmod-b-7c192b\"" Nov 03 13:24:07.076538 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:24:07.076494 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-7c192b\"/\"routing-overlay\"" Nov 03 13:24:07.076538 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:24:07.076536 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-7c192b\"/\"kube-root-ca.crt\"" Nov 03 13:24:07.076687 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:24:07.076571 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-7c192b\"/\"openshift-service-ca.crt\"" Nov 03 13:24:07.076687 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:24:07.076616 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-7c192b\"/\"agc-cred-f5ffe280c5d9\"" Nov 03 13:24:07.076788 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:24:07.076721 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-7c192b\"/\"payload-processing-external-model-e2e--cd1da41a-dockercfg-txc88\"" Nov 03 13:24:07.175985 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:24:07.175952 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b6932011-3c9e-4a12-a243-3692d4b09614-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-7c192b-8444x4nfn\" (UID: \"b6932011-3c9e-4a12-a243-3692d4b09614\") " pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-8444x4nfn" Nov 03 13:24:07.176487 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:24:07.176109 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6932011-3c9e-4a12-a243-3692d4b09614-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-7c192b-8444x4nfn\" (UID: \"b6932011-3c9e-4a12-a243-3692d4b09614\") " pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-8444x4nfn" Nov 03 13:24:07.176487 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:24:07.176218 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b6932011-3c9e-4a12-a243-3692d4b09614-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-7c192b-8444x4nfn\" (UID: \"b6932011-3c9e-4a12-a243-3692d4b09614\") " pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-8444x4nfn" Nov 03 13:24:07.176487 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:24:07.176253 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8tvlv\" (UniqueName: \"kubernetes.io/projected/b6932011-3c9e-4a12-a243-3692d4b09614-kube-api-access-8tvlv\") pod \"payload-processing-external-model-e2e-extmod-b-7c192b-8444x4nfn\" (UID: \"b6932011-3c9e-4a12-a243-3692d4b09614\") " pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-8444x4nfn" Nov 03 13:24:07.359363 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:24:07.357049 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b6932011-3c9e-4a12-a243-3692d4b09614-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-7c192b-8444x4nfn\" (UID: \"b6932011-3c9e-4a12-a243-3692d4b09614\") " pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-8444x4nfn" Nov 03 13:24:07.377307 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:24:07.375014 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6932011-3c9e-4a12-a243-3692d4b09614-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-7c192b-8444x4nfn\" (UID: \"b6932011-3c9e-4a12-a243-3692d4b09614\") " pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-8444x4nfn" Nov 03 13:24:07.377307 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:24:07.373367 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6932011-3c9e-4a12-a243-3692d4b09614-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-7c192b-8444x4nfn\" (UID: \"b6932011-3c9e-4a12-a243-3692d4b09614\") " pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-8444x4nfn" Nov 03 13:24:07.377307 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:24:07.375944 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b6932011-3c9e-4a12-a243-3692d4b09614-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-7c192b-8444x4nfn\" (UID: \"b6932011-3c9e-4a12-a243-3692d4b09614\") " pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-8444x4nfn" Nov 03 13:24:07.377307 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:24:07.376037 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b6932011-3c9e-4a12-a243-3692d4b09614-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-7c192b-8444x4nfn\" (UID: \"b6932011-3c9e-4a12-a243-3692d4b09614\") " pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-8444x4nfn" Nov 03 13:24:07.377307 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:24:07.376102 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8tvlv\" (UniqueName: \"kubernetes.io/projected/b6932011-3c9e-4a12-a243-3692d4b09614-kube-api-access-8tvlv\") pod \"payload-processing-external-model-e2e-extmod-b-7c192b-8444x4nfn\" (UID: \"b6932011-3c9e-4a12-a243-3692d4b09614\") " pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-8444x4nfn" Nov 03 13:24:07.420274 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:24:07.408124 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b6932011-3c9e-4a12-a243-3692d4b09614-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-7c192b-8444x4nfn\" (UID: \"b6932011-3c9e-4a12-a243-3692d4b09614\") " pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-8444x4nfn" Nov 03 13:24:07.441997 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:24:07.441914 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8tvlv\" (UniqueName: \"kubernetes.io/projected/b6932011-3c9e-4a12-a243-3692d4b09614-kube-api-access-8tvlv\") pod \"payload-processing-external-model-e2e-extmod-b-7c192b-8444x4nfn\" (UID: \"b6932011-3c9e-4a12-a243-3692d4b09614\") " pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-8444x4nfn" Nov 03 13:24:07.684437 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:24:07.684354 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-8444x4nfn" Nov 03 13:24:07.748400 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:24:07.748365 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-d3ca48/premium-w3-d3ca48-kserve-75ffc959b-zvjbc" Nov 03 13:24:08.024114 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:24:08.024080 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-8444x4nfn"] Nov 03 13:24:08.839827 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:24:08.839785 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-8444x4nfn" event={"ID":"b6932011-3c9e-4a12-a243-3692d4b09614","Type":"ContainerStarted","Data":"7b754bb4b3aa3dce84e00e399c58f3ad4a172e1f4a3a2262819f8acd37032738"} Nov 03 13:24:08.840363 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:24:08.839856 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-8444x4nfn" event={"ID":"b6932011-3c9e-4a12-a243-3692d4b09614","Type":"ContainerStarted","Data":"c9f098367c7ad043a7007e8a3f3e7eeb079776566d7230367ae873e048f20c50"} Nov 03 13:24:08.840363 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:24:08.839959 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-8444x4nfn" Nov 03 13:24:08.860355 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:24:08.860298 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-8444x4nfn" podStartSLOduration=1.860269963 podStartE2EDuration="1.860269963s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:24:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:24:08.858420929 +0000 UTC m=+928.866612207" watchObservedRunningTime="2026-11-03 13:24:08.860269963 +0000 UTC m=+928.868461261" Nov 03 13:24:09.509418 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:24:09.509374 5024 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d7mwz" containerName="extractor" Nov 03 13:24:12.294311 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:24:12.273549 5024 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-praxis-a-a2ccd8/praxis-route-a2ccd8-kserve-748544bd55-8rmdm" podUID="11621285-51db-4e67-903d-311f39837fc9" containerName="main" containerID="cri-o://0710966ec288a0c4283931918050baf6fc2e1f6800be5fe45fd4b4d0886378d5" gracePeriod=45 Nov 03 13:24:12.536305 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:24:12.535540 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-praxis-a-a2ccd8/praxis-route-a2ccd8-kserve-748544bd55-8rmdm" podUID="11621285-51db-4e67-903d-311f39837fc9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": EOF" Nov 03 13:24:13.049310 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:24:13.048966 5024 generic.go:356] "Generic (PLEG): container finished" podID="11621285-51db-4e67-903d-311f39837fc9" containerID="0710966ec288a0c4283931918050baf6fc2e1f6800be5fe45fd4b4d0886378d5" exitCode=0 Nov 03 13:24:13.049310 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:24:13.049159 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-a2ccd8/praxis-route-a2ccd8-kserve-748544bd55-8rmdm" event={"ID":"11621285-51db-4e67-903d-311f39837fc9","Type":"ContainerDied","Data":"0710966ec288a0c4283931918050baf6fc2e1f6800be5fe45fd4b4d0886378d5"} Nov 03 13:24:13.255818 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:24:13.255772 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-praxis-a-a2ccd8/praxis-route-a2ccd8-kserve-748544bd55-8rmdm" podUID="11621285-51db-4e67-903d-311f39837fc9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": dial tcp 10.132.0.78:8000: connect: connection refused" Nov 03 13:24:13.256176 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:24:13.256155 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-praxis-a-a2ccd8/praxis-route-a2ccd8-kserve-748544bd55-8rmdm" Nov 03 13:24:14.280642 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:24:14.280598 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-praxis-a-a2ccd8/praxis-route-a2ccd8-kserve-748544bd55-8rmdm" podUID="11621285-51db-4e67-903d-311f39837fc9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": dial tcp 10.132.0.78:8000: connect: connection refused" Nov 03 13:24:15.007190 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:24:15.007153 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-a2ccd8/praxis-route-a2ccd8-kserve-748544bd55-8rmdm" Nov 03 13:24:15.076304 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:24:15.070951 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11621285-51db-4e67-903d-311f39837fc9-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/11621285-51db-4e67-903d-311f39837fc9-tmp-dir\") pod \"11621285-51db-4e67-903d-311f39837fc9\" (UID: \"11621285-51db-4e67-903d-311f39837fc9\") " Nov 03 13:24:15.076304 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:24:15.071106 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11621285-51db-4e67-903d-311f39837fc9-kube-api-access-wxvsc\" (UniqueName: \"kubernetes.io/projected/11621285-51db-4e67-903d-311f39837fc9-kube-api-access-wxvsc\") pod \"11621285-51db-4e67-903d-311f39837fc9\" (UID: \"11621285-51db-4e67-903d-311f39837fc9\") " Nov 03 13:24:15.076304 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:24:15.071182 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11621285-51db-4e67-903d-311f39837fc9-home\" (UniqueName: \"kubernetes.io/empty-dir/11621285-51db-4e67-903d-311f39837fc9-home\") pod \"11621285-51db-4e67-903d-311f39837fc9\" (UID: \"11621285-51db-4e67-903d-311f39837fc9\") " Nov 03 13:24:15.076304 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:24:15.071324 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11621285-51db-4e67-903d-311f39837fc9-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/11621285-51db-4e67-903d-311f39837fc9-model-cache\") pod \"11621285-51db-4e67-903d-311f39837fc9\" (UID: \"11621285-51db-4e67-903d-311f39837fc9\") " Nov 03 13:24:15.076304 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:24:15.071381 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11621285-51db-4e67-903d-311f39837fc9-dshm\" (UniqueName: \"kubernetes.io/empty-dir/11621285-51db-4e67-903d-311f39837fc9-dshm\") pod \"11621285-51db-4e67-903d-311f39837fc9\" (UID: \"11621285-51db-4e67-903d-311f39837fc9\") " Nov 03 13:24:15.076304 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:24:15.071448 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/11621285-51db-4e67-903d-311f39837fc9-tls-certs\" (UniqueName: \"kubernetes.io/secret/11621285-51db-4e67-903d-311f39837fc9-tls-certs\") pod \"11621285-51db-4e67-903d-311f39837fc9\" (UID: \"11621285-51db-4e67-903d-311f39837fc9\") " Nov 03 13:24:15.076304 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:24:15.071562 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11621285-51db-4e67-903d-311f39837fc9-tmp-dir" pod "11621285-51db-4e67-903d-311f39837fc9" (UID: "11621285-51db-4e67-903d-311f39837fc9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:24:15.076304 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:24:15.071753 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11621285-51db-4e67-903d-311f39837fc9-home" pod "11621285-51db-4e67-903d-311f39837fc9" (UID: "11621285-51db-4e67-903d-311f39837fc9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:24:15.076304 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:24:15.072320 5024 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/11621285-51db-4e67-903d-311f39837fc9-tmp-dir\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:24:15.076304 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:24:15.072343 5024 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/11621285-51db-4e67-903d-311f39837fc9-home\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:24:15.084658 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:24:15.083426 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/11621285-51db-4e67-903d-311f39837fc9-tls-certs" pod "11621285-51db-4e67-903d-311f39837fc9" (UID: "11621285-51db-4e67-903d-311f39837fc9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 13:24:15.086928 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:24:15.084265 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11621285-51db-4e67-903d-311f39837fc9-dshm" pod "11621285-51db-4e67-903d-311f39837fc9" (UID: "11621285-51db-4e67-903d-311f39837fc9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:24:15.123086 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:24:15.122987 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11621285-51db-4e67-903d-311f39837fc9-kube-api-access-wxvsc" pod "11621285-51db-4e67-903d-311f39837fc9" (UID: "11621285-51db-4e67-903d-311f39837fc9"). InnerVolumeSpecName "kube-api-access-wxvsc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:24:15.129664 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:24:15.128419 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11621285-51db-4e67-903d-311f39837fc9-model-cache" pod "11621285-51db-4e67-903d-311f39837fc9" (UID: "11621285-51db-4e67-903d-311f39837fc9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:24:15.290938 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:24:15.290667 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wxvsc\" (UniqueName: \"kubernetes.io/projected/11621285-51db-4e67-903d-311f39837fc9-kube-api-access-wxvsc\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:24:15.291502 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:24:15.291481 5024 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/11621285-51db-4e67-903d-311f39837fc9-model-cache\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:24:15.291611 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:24:15.291597 5024 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/11621285-51db-4e67-903d-311f39837fc9-dshm\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:24:15.291702 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:24:15.291690 5024 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/11621285-51db-4e67-903d-311f39837fc9-tls-certs\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:24:15.340324 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:24:15.340273 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-a2ccd8/praxis-route-a2ccd8-kserve-748544bd55-8rmdm" event={"ID":"11621285-51db-4e67-903d-311f39837fc9","Type":"ContainerDied","Data":"6fbf154178ea0d77da4bbbe1cba5d93c70ec359a3c6939c3d2e8a49adbb61fd2"} Nov 03 13:24:15.340590 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:24:15.340570 5024 scope.go:122] "RemoveContainer" containerID="0710966ec288a0c4283931918050baf6fc2e1f6800be5fe45fd4b4d0886378d5" Nov 03 13:24:15.340843 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:24:15.340828 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-a2ccd8/praxis-route-a2ccd8-kserve-748544bd55-8rmdm" Nov 03 13:24:15.381505 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:24:15.381392 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-praxis-a-a2ccd8/praxis-route-a2ccd8-kserve-748544bd55-8rmdm"] Nov 03 13:24:15.386570 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:24:15.386536 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-praxis-a-a2ccd8/praxis-route-a2ccd8-kserve-748544bd55-8rmdm"] Nov 03 13:24:15.444740 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:24:15.444700 5024 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod11621285_51db_4e67_903d_311f39837fc9.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod11621285_51db_4e67_903d_311f39837fc9.slice/crio-6fbf154178ea0d77da4bbbe1cba5d93c70ec359a3c6939c3d2e8a49adbb61fd2\": RecentStats: unable to find data in memory cache]" Nov 03 13:24:15.749508 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:24:15.749412 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-d3ca48/premium-w3-d3ca48-kserve-75ffc959b-zvjbc" Nov 03 13:24:15.762213 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:24:15.762184 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-d3ca48/premium-w3-d3ca48-kserve-75ffc959b-zvjbc" Nov 03 13:24:16.109487 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:24:16.109457 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="11621285-51db-4e67-903d-311f39837fc9" path="/var/lib/kubelet/pods/11621285-51db-4e67-903d-311f39837fc9/volumes" Nov 03 13:24:16.361948 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:24:16.361833 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-d3ca48/premium-w3-d3ca48-kserve-75ffc959b-zvjbc" Nov 03 13:24:19.888940 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:24:19.888913 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-8444x4nfn" Nov 03 13:24:39.511929 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:24:39.511884 5024 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d7mwz" containerName="extractor" Nov 03 13:24:42.961451 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:24:42.961406 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-ef1693-86df94dcf8-nllf6"] Nov 03 13:24:42.975016 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:24:42.974975 5024 cpu_manager.go:414] "containerMap: removing container" podUID="11621285-51db-4e67-903d-311f39837fc9" containerName="main" Nov 03 13:24:42.975249 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:24:42.975228 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11621285-51db-4e67-903d-311f39837fc9" containerName="main" Nov 03 13:24:43.125968 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:24:43.125564 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="11621285-51db-4e67-903d-311f39837fc9" containerName="main" Nov 03 13:24:43.169372 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:24:43.169332 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-ef1693-86df94dcf8-nllf6"] Nov 03 13:24:43.170185 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:24:43.170159 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ef1693-86df94dcf8-nllf6" Nov 03 13:24:43.177718 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:24:43.177675 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-ef1693-serving-cert\"" Nov 03 13:24:43.294373 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:24:43.284350 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0104d1e3-0aa2-489b-85de-c95cf4c85524-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-ef1693-86df94dcf8-nllf6\" (UID: \"0104d1e3-0aa2-489b-85de-c95cf4c85524\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ef1693-86df94dcf8-nllf6" Nov 03 13:24:43.345044 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:24:43.345005 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0104d1e3-0aa2-489b-85de-c95cf4c85524-metrics-certs\") pod \"maas-api-e2e-extmod-a-ef1693-86df94dcf8-nllf6\" (UID: \"0104d1e3-0aa2-489b-85de-c95cf4c85524\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ef1693-86df94dcf8-nllf6" Nov 03 13:24:43.345446 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:24:43.345416 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0104d1e3-0aa2-489b-85de-c95cf4c85524-maas-api-tls\") pod \"maas-api-e2e-extmod-a-ef1693-86df94dcf8-nllf6\" (UID: \"0104d1e3-0aa2-489b-85de-c95cf4c85524\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ef1693-86df94dcf8-nllf6" Nov 03 13:24:43.345735 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:24:43.345713 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qp52t\" (UniqueName: \"kubernetes.io/projected/0104d1e3-0aa2-489b-85de-c95cf4c85524-kube-api-access-qp52t\") pod \"maas-api-e2e-extmod-a-ef1693-86df94dcf8-nllf6\" (UID: \"0104d1e3-0aa2-489b-85de-c95cf4c85524\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ef1693-86df94dcf8-nllf6" Nov 03 13:24:43.517566 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:24:43.517524 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qp52t\" (UniqueName: \"kubernetes.io/projected/0104d1e3-0aa2-489b-85de-c95cf4c85524-kube-api-access-qp52t\") pod \"maas-api-e2e-extmod-a-ef1693-86df94dcf8-nllf6\" (UID: \"0104d1e3-0aa2-489b-85de-c95cf4c85524\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ef1693-86df94dcf8-nllf6" Nov 03 13:24:43.521206 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:24:43.521172 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0104d1e3-0aa2-489b-85de-c95cf4c85524-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-ef1693-86df94dcf8-nllf6\" (UID: \"0104d1e3-0aa2-489b-85de-c95cf4c85524\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ef1693-86df94dcf8-nllf6" Nov 03 13:24:43.552366 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:24:43.545435 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0104d1e3-0aa2-489b-85de-c95cf4c85524-metrics-certs\") pod \"maas-api-e2e-extmod-a-ef1693-86df94dcf8-nllf6\" (UID: \"0104d1e3-0aa2-489b-85de-c95cf4c85524\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ef1693-86df94dcf8-nllf6" Nov 03 13:24:43.552366 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:24:43.545535 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0104d1e3-0aa2-489b-85de-c95cf4c85524-maas-api-tls\") pod \"maas-api-e2e-extmod-a-ef1693-86df94dcf8-nllf6\" (UID: \"0104d1e3-0aa2-489b-85de-c95cf4c85524\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ef1693-86df94dcf8-nllf6" Nov 03 13:24:43.552366 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:24:43.545790 5024 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-a-ef1693-serving-cert: secret "maas-api-e2e-extmod-a-ef1693-serving-cert" not found Nov 03 13:24:43.552366 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:24:43.545874 5024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0104d1e3-0aa2-489b-85de-c95cf4c85524-maas-api-tls podName:0104d1e3-0aa2-489b-85de-c95cf4c85524 nodeName:}" failed. No retries permitted until 2026-11-03 13:24:44.045850734 +0000 UTC m=+964.054042008 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0104d1e3-0aa2-489b-85de-c95cf4c85524-maas-api-tls") pod "maas-api-e2e-extmod-a-ef1693-86df94dcf8-nllf6" (UID: "0104d1e3-0aa2-489b-85de-c95cf4c85524") : secret "maas-api-e2e-extmod-a-ef1693-serving-cert" not found Nov 03 13:24:43.552366 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:24:43.522182 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0104d1e3-0aa2-489b-85de-c95cf4c85524-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-ef1693-86df94dcf8-nllf6\" (UID: \"0104d1e3-0aa2-489b-85de-c95cf4c85524\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ef1693-86df94dcf8-nllf6" Nov 03 13:24:43.590312 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:24:43.566994 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0104d1e3-0aa2-489b-85de-c95cf4c85524-metrics-certs\") pod \"maas-api-e2e-extmod-a-ef1693-86df94dcf8-nllf6\" (UID: \"0104d1e3-0aa2-489b-85de-c95cf4c85524\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ef1693-86df94dcf8-nllf6" Nov 03 13:24:43.688311 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:24:43.686493 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qp52t\" (UniqueName: \"kubernetes.io/projected/0104d1e3-0aa2-489b-85de-c95cf4c85524-kube-api-access-qp52t\") pod \"maas-api-e2e-extmod-a-ef1693-86df94dcf8-nllf6\" (UID: \"0104d1e3-0aa2-489b-85de-c95cf4c85524\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ef1693-86df94dcf8-nllf6" Nov 03 13:24:44.095768 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:24:44.088315 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0104d1e3-0aa2-489b-85de-c95cf4c85524-maas-api-tls\") pod \"maas-api-e2e-extmod-a-ef1693-86df94dcf8-nllf6\" (UID: \"0104d1e3-0aa2-489b-85de-c95cf4c85524\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ef1693-86df94dcf8-nllf6" Nov 03 13:24:44.126853 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:24:44.126808 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0104d1e3-0aa2-489b-85de-c95cf4c85524-maas-api-tls\") pod \"maas-api-e2e-extmod-a-ef1693-86df94dcf8-nllf6\" (UID: \"0104d1e3-0aa2-489b-85de-c95cf4c85524\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ef1693-86df94dcf8-nllf6" Nov 03 13:24:44.356367 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:24:44.353125 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d3ca48/premium-w3-d3ca48-kserve-75ffc959b-zvjbc" podUID="add1e8b9-e7fd-4e4d-8d10-f97d0e2612ba" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 13:24:44.394500 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:24:44.394447 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ef1693-86df94dcf8-nllf6" Nov 03 13:24:44.578190 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:24:44.561383 5024 patch_prober.go:29] interesting pod/e2e-worker-w4-c381a7-openshift-default-78bcd4695-7tc2j container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 13:24:44.578190 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:24:44.561471 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-7tc2j" podUID="b61db32a-7a3b-4aff-b331-f7888a1a4e0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 13:24:45.594181 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:24:45.593359 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-ef1693-86df94dcf8-nllf6"] Nov 03 13:24:45.599091 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:24:45.599055 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0104d1e3_0aa2_489b_85de_c95cf4c85524.slice/crio-10a4553ffa35daa5cbd87b83faea4c7ae4a0336df5575d883b0d3913e71cdc54 WatchSource:0}: Error finding container 10a4553ffa35daa5cbd87b83faea4c7ae4a0336df5575d883b0d3913e71cdc54: Status 404 returned error can't find the container with id 10a4553ffa35daa5cbd87b83faea4c7ae4a0336df5575d883b0d3913e71cdc54 Nov 03 13:24:45.979554 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:24:45.979513 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ef1693-86df94dcf8-nllf6" event={"ID":"0104d1e3-0aa2-489b-85de-c95cf4c85524","Type":"ContainerStarted","Data":"10a4553ffa35daa5cbd87b83faea4c7ae4a0336df5575d883b0d3913e71cdc54"} Nov 03 13:24:47.065551 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:24:47.058114 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ef1693-86df94dcf8-nllf6" event={"ID":"0104d1e3-0aa2-489b-85de-c95cf4c85524","Type":"ContainerStarted","Data":"01647972e148180edda0b8f3893a5fe354d8d5d4db6ebbe53aa9122e1d41c82f"} Nov 03 13:24:47.065551 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:24:47.059247 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ef1693-86df94dcf8-nllf6" Nov 03 13:24:47.095329 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:24:47.088781 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ef1693-86df94dcf8-nllf6" podStartSLOduration=4.659579469 podStartE2EDuration="5.088759215s" totalImagesPullingTime="429.179746ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:24:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:24:47.084946869 +0000 UTC m=+967.093138147" watchObservedRunningTime="2026-11-03 13:24:47.088759215 +0000 UTC m=+967.096950492" Nov 03 13:24:53.513930 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:24:53.513890 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-a2ccd8-6bfb4ddf8f-vqss2"] Nov 03 13:24:53.514808 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:24:53.514768 5024 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-a2ccd8-6bfb4ddf8f-vqss2" podUID="41f9ddad-1f63-4f31-86e4-831fd43b8f44" containerName="maas-api" containerID="cri-o://20fbeb4a902e176d63d2d0df1e0dac6336c8f5ae18aaaecc9c4f0489d5020cf9" gracePeriod=30 Nov 03 13:24:54.141174 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:24:54.141127 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ef1693-86df94dcf8-nllf6" Nov 03 13:24:54.144746 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:24:54.144710 5024 generic.go:356] "Generic (PLEG): container finished" podID="41f9ddad-1f63-4f31-86e4-831fd43b8f44" containerID="20fbeb4a902e176d63d2d0df1e0dac6336c8f5ae18aaaecc9c4f0489d5020cf9" exitCode=0 Nov 03 13:24:54.144931 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:24:54.144906 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-a2ccd8-6bfb4ddf8f-vqss2" event={"ID":"41f9ddad-1f63-4f31-86e4-831fd43b8f44","Type":"ContainerDied","Data":"20fbeb4a902e176d63d2d0df1e0dac6336c8f5ae18aaaecc9c4f0489d5020cf9"} Nov 03 13:24:54.185758 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:24:54.185722 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-a2ccd8-6bfb4ddf8f-vqss2" Nov 03 13:24:54.333641 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:24:54.333599 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/41f9ddad-1f63-4f31-86e4-831fd43b8f44-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/41f9ddad-1f63-4f31-86e4-831fd43b8f44-maas-api-tls\") pod \"41f9ddad-1f63-4f31-86e4-831fd43b8f44\" (UID: \"41f9ddad-1f63-4f31-86e4-831fd43b8f44\") " Nov 03 13:24:54.333641 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:24:54.333653 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/41f9ddad-1f63-4f31-86e4-831fd43b8f44-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/41f9ddad-1f63-4f31-86e4-831fd43b8f44-openshift-service-ca\") pod \"41f9ddad-1f63-4f31-86e4-831fd43b8f44\" (UID: \"41f9ddad-1f63-4f31-86e4-831fd43b8f44\") " Nov 03 13:24:54.333921 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:24:54.333676 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/41f9ddad-1f63-4f31-86e4-831fd43b8f44-metrics-certs\" (UniqueName: \"kubernetes.io/secret/41f9ddad-1f63-4f31-86e4-831fd43b8f44-metrics-certs\") pod \"41f9ddad-1f63-4f31-86e4-831fd43b8f44\" (UID: \"41f9ddad-1f63-4f31-86e4-831fd43b8f44\") " Nov 03 13:24:54.333921 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:24:54.333713 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41f9ddad-1f63-4f31-86e4-831fd43b8f44-kube-api-access-ggxr8\" (UniqueName: \"kubernetes.io/projected/41f9ddad-1f63-4f31-86e4-831fd43b8f44-kube-api-access-ggxr8\") pod \"41f9ddad-1f63-4f31-86e4-831fd43b8f44\" (UID: \"41f9ddad-1f63-4f31-86e4-831fd43b8f44\") " Nov 03 13:24:54.334190 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:24:54.334158 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41f9ddad-1f63-4f31-86e4-831fd43b8f44-openshift-service-ca" pod "41f9ddad-1f63-4f31-86e4-831fd43b8f44" (UID: "41f9ddad-1f63-4f31-86e4-831fd43b8f44"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:24:54.334388 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:24:54.334372 5024 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/41f9ddad-1f63-4f31-86e4-831fd43b8f44-openshift-service-ca\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:24:54.335939 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:24:54.335907 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/41f9ddad-1f63-4f31-86e4-831fd43b8f44-metrics-certs" pod "41f9ddad-1f63-4f31-86e4-831fd43b8f44" (UID: "41f9ddad-1f63-4f31-86e4-831fd43b8f44"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 13:24:54.335939 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:24:54.335922 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/41f9ddad-1f63-4f31-86e4-831fd43b8f44-maas-api-tls" pod "41f9ddad-1f63-4f31-86e4-831fd43b8f44" (UID: "41f9ddad-1f63-4f31-86e4-831fd43b8f44"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 13:24:54.335939 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:24:54.335914 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41f9ddad-1f63-4f31-86e4-831fd43b8f44-kube-api-access-ggxr8" pod "41f9ddad-1f63-4f31-86e4-831fd43b8f44" (UID: "41f9ddad-1f63-4f31-86e4-831fd43b8f44"). InnerVolumeSpecName "kube-api-access-ggxr8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:24:54.446113 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:24:54.440847 5024 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/41f9ddad-1f63-4f31-86e4-831fd43b8f44-maas-api-tls\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:24:54.446113 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:24:54.440887 5024 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/41f9ddad-1f63-4f31-86e4-831fd43b8f44-metrics-certs\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:24:54.446113 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:24:54.440903 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ggxr8\" (UniqueName: \"kubernetes.io/projected/41f9ddad-1f63-4f31-86e4-831fd43b8f44-kube-api-access-ggxr8\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:24:55.206307 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:24:55.205329 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-a2ccd8-6bfb4ddf8f-vqss2" event={"ID":"41f9ddad-1f63-4f31-86e4-831fd43b8f44","Type":"ContainerDied","Data":"40055e76b30d95d034de48082ba72a552b97e76f15c343eb66571614357f6346"} Nov 03 13:24:55.206307 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:24:55.205412 5024 scope.go:122] "RemoveContainer" containerID="20fbeb4a902e176d63d2d0df1e0dac6336c8f5ae18aaaecc9c4f0489d5020cf9" Nov 03 13:24:55.206307 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:24:55.205597 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-a2ccd8-6bfb4ddf8f-vqss2" Nov 03 13:24:55.267618 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:24:55.267133 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-a2ccd8-openshift-default-798b597bf7-5ljc2"] Nov 03 13:24:55.267618 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:24:55.267396 5024 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-a2ccd8-openshift-default-798b597bf7-5ljc2" podUID="c08c13a6-4853-40b2-9496-ac50f59b1bf8" containerName="istio-proxy" containerID="cri-o://f7f63080bf0fb5b2a59903eb4d3bf9d86a386f978c8bbc70fee460d73433df0b" gracePeriod=30 Nov 03 13:24:55.288471 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:24:55.288433 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-a2ccd8-6bfb4ddf8f-vqss2"] Nov 03 13:24:55.293207 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:24:55.293171 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-a2ccd8-6bfb4ddf8f-vqss2"] Nov 03 13:24:56.111073 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:24:56.111031 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="41f9ddad-1f63-4f31-86e4-831fd43b8f44" path="/var/lib/kubelet/pods/41f9ddad-1f63-4f31-86e4-831fd43b8f44/volumes" Nov 03 13:24:57.974236 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:24:57.974198 5024 patch_prober.go:29] interesting pod/e2e-praxis-a-a2ccd8-openshift-default-798b597bf7-5ljc2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 13:24:57.974663 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:24:57.974261 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-a-a2ccd8-openshift-default-798b597bf7-5ljc2" podUID="c08c13a6-4853-40b2-9496-ac50f59b1bf8" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 13:24:59.455332 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:24:59.446146 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-1a18c8-openshift-default-67d7b5995c-z4csj"] Nov 03 13:24:59.510871 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:24:59.510834 5024 cpu_manager.go:414] "containerMap: removing container" podUID="41f9ddad-1f63-4f31-86e4-831fd43b8f44" containerName="maas-api" Nov 03 13:24:59.511111 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:24:59.511090 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="41f9ddad-1f63-4f31-86e4-831fd43b8f44" containerName="maas-api" Nov 03 13:24:59.511467 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:24:59.511450 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="41f9ddad-1f63-4f31-86e4-831fd43b8f44" containerName="maas-api" Nov 03 13:24:59.544512 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:24:59.518807 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-1a18c8-openshift-default-67d7b5995c-z4csj" Nov 03 13:24:59.601305 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:24:59.532023 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-1a18c8-openshift-default-67d7b5995c-z4csj"] Nov 03 13:24:59.606326 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:24:59.606230 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-1a18c8-openshift-default-dockercfg-ff4vz\"" Nov 03 13:24:59.846306 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:24:59.846253 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/168564ee-9452-4aba-8391-d4f0bdaafa7b-istio-envoy\") pod \"e2e-extmod-b-1a18c8-openshift-default-67d7b5995c-z4csj\" (UID: \"168564ee-9452-4aba-8391-d4f0bdaafa7b\") " pod="openshift-ingress/e2e-extmod-b-1a18c8-openshift-default-67d7b5995c-z4csj" Nov 03 13:24:59.874556 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:24:59.873268 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/168564ee-9452-4aba-8391-d4f0bdaafa7b-istio-podinfo\") pod \"e2e-extmod-b-1a18c8-openshift-default-67d7b5995c-z4csj\" (UID: \"168564ee-9452-4aba-8391-d4f0bdaafa7b\") " pod="openshift-ingress/e2e-extmod-b-1a18c8-openshift-default-67d7b5995c-z4csj" Nov 03 13:24:59.876528 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:24:59.876486 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/168564ee-9452-4aba-8391-d4f0bdaafa7b-istio-data\") pod \"e2e-extmod-b-1a18c8-openshift-default-67d7b5995c-z4csj\" (UID: \"168564ee-9452-4aba-8391-d4f0bdaafa7b\") " pod="openshift-ingress/e2e-extmod-b-1a18c8-openshift-default-67d7b5995c-z4csj" Nov 03 13:24:59.876761 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:24:59.876740 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/168564ee-9452-4aba-8391-d4f0bdaafa7b-credential-socket\") pod \"e2e-extmod-b-1a18c8-openshift-default-67d7b5995c-z4csj\" (UID: \"168564ee-9452-4aba-8391-d4f0bdaafa7b\") " pod="openshift-ingress/e2e-extmod-b-1a18c8-openshift-default-67d7b5995c-z4csj" Nov 03 13:24:59.877164 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:24:59.877143 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/168564ee-9452-4aba-8391-d4f0bdaafa7b-workload-certs\") pod \"e2e-extmod-b-1a18c8-openshift-default-67d7b5995c-z4csj\" (UID: \"168564ee-9452-4aba-8391-d4f0bdaafa7b\") " pod="openshift-ingress/e2e-extmod-b-1a18c8-openshift-default-67d7b5995c-z4csj" Nov 03 13:24:59.877348 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:24:59.877331 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/168564ee-9452-4aba-8391-d4f0bdaafa7b-istio-token\") pod \"e2e-extmod-b-1a18c8-openshift-default-67d7b5995c-z4csj\" (UID: \"168564ee-9452-4aba-8391-d4f0bdaafa7b\") " pod="openshift-ingress/e2e-extmod-b-1a18c8-openshift-default-67d7b5995c-z4csj" Nov 03 13:24:59.877508 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:24:59.877494 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/168564ee-9452-4aba-8391-d4f0bdaafa7b-istiod-ca-cert\") pod \"e2e-extmod-b-1a18c8-openshift-default-67d7b5995c-z4csj\" (UID: \"168564ee-9452-4aba-8391-d4f0bdaafa7b\") " pod="openshift-ingress/e2e-extmod-b-1a18c8-openshift-default-67d7b5995c-z4csj" Nov 03 13:24:59.877605 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:24:59.877593 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/168564ee-9452-4aba-8391-d4f0bdaafa7b-workload-socket\") pod \"e2e-extmod-b-1a18c8-openshift-default-67d7b5995c-z4csj\" (UID: \"168564ee-9452-4aba-8391-d4f0bdaafa7b\") " pod="openshift-ingress/e2e-extmod-b-1a18c8-openshift-default-67d7b5995c-z4csj" Nov 03 13:24:59.877735 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:24:59.877721 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lrg9v\" (UniqueName: \"kubernetes.io/projected/168564ee-9452-4aba-8391-d4f0bdaafa7b-kube-api-access-lrg9v\") pod \"e2e-extmod-b-1a18c8-openshift-default-67d7b5995c-z4csj\" (UID: \"168564ee-9452-4aba-8391-d4f0bdaafa7b\") " pod="openshift-ingress/e2e-extmod-b-1a18c8-openshift-default-67d7b5995c-z4csj" Nov 03 13:24:59.985419 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:24:59.985355 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/168564ee-9452-4aba-8391-d4f0bdaafa7b-istio-podinfo\") pod \"e2e-extmod-b-1a18c8-openshift-default-67d7b5995c-z4csj\" (UID: \"168564ee-9452-4aba-8391-d4f0bdaafa7b\") " pod="openshift-ingress/e2e-extmod-b-1a18c8-openshift-default-67d7b5995c-z4csj" Nov 03 13:24:59.986204 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:24:59.986175 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/168564ee-9452-4aba-8391-d4f0bdaafa7b-istio-data\") pod \"e2e-extmod-b-1a18c8-openshift-default-67d7b5995c-z4csj\" (UID: \"168564ee-9452-4aba-8391-d4f0bdaafa7b\") " pod="openshift-ingress/e2e-extmod-b-1a18c8-openshift-default-67d7b5995c-z4csj" Nov 03 13:24:59.986415 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:24:59.986397 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/168564ee-9452-4aba-8391-d4f0bdaafa7b-credential-socket\") pod \"e2e-extmod-b-1a18c8-openshift-default-67d7b5995c-z4csj\" (UID: \"168564ee-9452-4aba-8391-d4f0bdaafa7b\") " pod="openshift-ingress/e2e-extmod-b-1a18c8-openshift-default-67d7b5995c-z4csj" Nov 03 13:24:59.986652 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:24:59.986633 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/168564ee-9452-4aba-8391-d4f0bdaafa7b-workload-certs\") pod \"e2e-extmod-b-1a18c8-openshift-default-67d7b5995c-z4csj\" (UID: \"168564ee-9452-4aba-8391-d4f0bdaafa7b\") " pod="openshift-ingress/e2e-extmod-b-1a18c8-openshift-default-67d7b5995c-z4csj" Nov 03 13:24:59.986802 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:24:59.986784 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/168564ee-9452-4aba-8391-d4f0bdaafa7b-istio-token\") pod \"e2e-extmod-b-1a18c8-openshift-default-67d7b5995c-z4csj\" (UID: \"168564ee-9452-4aba-8391-d4f0bdaafa7b\") " pod="openshift-ingress/e2e-extmod-b-1a18c8-openshift-default-67d7b5995c-z4csj" Nov 03 13:24:59.986951 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:24:59.986932 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/168564ee-9452-4aba-8391-d4f0bdaafa7b-istiod-ca-cert\") pod \"e2e-extmod-b-1a18c8-openshift-default-67d7b5995c-z4csj\" (UID: \"168564ee-9452-4aba-8391-d4f0bdaafa7b\") " pod="openshift-ingress/e2e-extmod-b-1a18c8-openshift-default-67d7b5995c-z4csj" Nov 03 13:24:59.987072 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:24:59.987057 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/168564ee-9452-4aba-8391-d4f0bdaafa7b-workload-socket\") pod \"e2e-extmod-b-1a18c8-openshift-default-67d7b5995c-z4csj\" (UID: \"168564ee-9452-4aba-8391-d4f0bdaafa7b\") " pod="openshift-ingress/e2e-extmod-b-1a18c8-openshift-default-67d7b5995c-z4csj" Nov 03 13:24:59.990414 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:24:59.990385 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/168564ee-9452-4aba-8391-d4f0bdaafa7b-workload-socket\") pod \"e2e-extmod-b-1a18c8-openshift-default-67d7b5995c-z4csj\" (UID: \"168564ee-9452-4aba-8391-d4f0bdaafa7b\") " pod="openshift-ingress/e2e-extmod-b-1a18c8-openshift-default-67d7b5995c-z4csj" Nov 03 13:24:59.990606 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:24:59.988970 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/168564ee-9452-4aba-8391-d4f0bdaafa7b-istio-data\") pod \"e2e-extmod-b-1a18c8-openshift-default-67d7b5995c-z4csj\" (UID: \"168564ee-9452-4aba-8391-d4f0bdaafa7b\") " pod="openshift-ingress/e2e-extmod-b-1a18c8-openshift-default-67d7b5995c-z4csj" Nov 03 13:24:59.990606 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:24:59.989181 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/168564ee-9452-4aba-8391-d4f0bdaafa7b-credential-socket\") pod \"e2e-extmod-b-1a18c8-openshift-default-67d7b5995c-z4csj\" (UID: \"168564ee-9452-4aba-8391-d4f0bdaafa7b\") " pod="openshift-ingress/e2e-extmod-b-1a18c8-openshift-default-67d7b5995c-z4csj" Nov 03 13:24:59.990606 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:24:59.990141 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/168564ee-9452-4aba-8391-d4f0bdaafa7b-istiod-ca-cert\") pod \"e2e-extmod-b-1a18c8-openshift-default-67d7b5995c-z4csj\" (UID: \"168564ee-9452-4aba-8391-d4f0bdaafa7b\") " pod="openshift-ingress/e2e-extmod-b-1a18c8-openshift-default-67d7b5995c-z4csj" Nov 03 13:24:59.990606 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:24:59.990387 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrg9v\" (UniqueName: \"kubernetes.io/projected/168564ee-9452-4aba-8391-d4f0bdaafa7b-kube-api-access-lrg9v\") pod \"e2e-extmod-b-1a18c8-openshift-default-67d7b5995c-z4csj\" (UID: \"168564ee-9452-4aba-8391-d4f0bdaafa7b\") " pod="openshift-ingress/e2e-extmod-b-1a18c8-openshift-default-67d7b5995c-z4csj" Nov 03 13:24:59.990859 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:24:59.990659 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/168564ee-9452-4aba-8391-d4f0bdaafa7b-istio-envoy\") pod \"e2e-extmod-b-1a18c8-openshift-default-67d7b5995c-z4csj\" (UID: \"168564ee-9452-4aba-8391-d4f0bdaafa7b\") " pod="openshift-ingress/e2e-extmod-b-1a18c8-openshift-default-67d7b5995c-z4csj" Nov 03 13:24:59.991243 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:24:59.988751 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/168564ee-9452-4aba-8391-d4f0bdaafa7b-workload-certs\") pod \"e2e-extmod-b-1a18c8-openshift-default-67d7b5995c-z4csj\" (UID: \"168564ee-9452-4aba-8391-d4f0bdaafa7b\") " pod="openshift-ingress/e2e-extmod-b-1a18c8-openshift-default-67d7b5995c-z4csj" Nov 03 13:24:59.998301 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:24:59.998188 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/168564ee-9452-4aba-8391-d4f0bdaafa7b-istio-podinfo\") pod \"e2e-extmod-b-1a18c8-openshift-default-67d7b5995c-z4csj\" (UID: \"168564ee-9452-4aba-8391-d4f0bdaafa7b\") " pod="openshift-ingress/e2e-extmod-b-1a18c8-openshift-default-67d7b5995c-z4csj" Nov 03 13:25:00.015890 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:00.014649 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/168564ee-9452-4aba-8391-d4f0bdaafa7b-istio-envoy\") pod \"e2e-extmod-b-1a18c8-openshift-default-67d7b5995c-z4csj\" (UID: \"168564ee-9452-4aba-8391-d4f0bdaafa7b\") " pod="openshift-ingress/e2e-extmod-b-1a18c8-openshift-default-67d7b5995c-z4csj" Nov 03 13:25:00.015890 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:00.015811 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/168564ee-9452-4aba-8391-d4f0bdaafa7b-istio-token\") pod \"e2e-extmod-b-1a18c8-openshift-default-67d7b5995c-z4csj\" (UID: \"168564ee-9452-4aba-8391-d4f0bdaafa7b\") " pod="openshift-ingress/e2e-extmod-b-1a18c8-openshift-default-67d7b5995c-z4csj" Nov 03 13:25:00.018996 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:00.018934 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lrg9v\" (UniqueName: \"kubernetes.io/projected/168564ee-9452-4aba-8391-d4f0bdaafa7b-kube-api-access-lrg9v\") pod \"e2e-extmod-b-1a18c8-openshift-default-67d7b5995c-z4csj\" (UID: \"168564ee-9452-4aba-8391-d4f0bdaafa7b\") " pod="openshift-ingress/e2e-extmod-b-1a18c8-openshift-default-67d7b5995c-z4csj" Nov 03 13:25:00.210159 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:00.210121 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-1a18c8-openshift-default-67d7b5995c-z4csj" Nov 03 13:25:00.586074 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:00.584928 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-8444x4nfn"] Nov 03 13:25:01.350844 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:01.350815 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-a2ccd8-openshift-default-798b597bf7-5ljc2" Nov 03 13:25:01.393836 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:01.393795 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-1a18c8-openshift-default-67d7b5995c-z4csj"] Nov 03 13:25:01.396869 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:01.396363 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:25:01.396869 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:01.396445 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:25:01.396869 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:01.396488 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:25:01.514847 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:01.514801 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c08c13a6-4853-40b2-9496-ac50f59b1bf8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c08c13a6-4853-40b2-9496-ac50f59b1bf8-istio-podinfo\") pod \"c08c13a6-4853-40b2-9496-ac50f59b1bf8\" (UID: \"c08c13a6-4853-40b2-9496-ac50f59b1bf8\") " Nov 03 13:25:01.515032 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:01.514868 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c08c13a6-4853-40b2-9496-ac50f59b1bf8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c08c13a6-4853-40b2-9496-ac50f59b1bf8-workload-socket\") pod \"c08c13a6-4853-40b2-9496-ac50f59b1bf8\" (UID: \"c08c13a6-4853-40b2-9496-ac50f59b1bf8\") " Nov 03 13:25:01.515032 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:01.514910 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c08c13a6-4853-40b2-9496-ac50f59b1bf8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c08c13a6-4853-40b2-9496-ac50f59b1bf8-istiod-ca-cert\") pod \"c08c13a6-4853-40b2-9496-ac50f59b1bf8\" (UID: \"c08c13a6-4853-40b2-9496-ac50f59b1bf8\") " Nov 03 13:25:01.515157 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:01.515054 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c08c13a6-4853-40b2-9496-ac50f59b1bf8-kube-api-access-l9q6t\" (UniqueName: \"kubernetes.io/projected/c08c13a6-4853-40b2-9496-ac50f59b1bf8-kube-api-access-l9q6t\") pod \"c08c13a6-4853-40b2-9496-ac50f59b1bf8\" (UID: \"c08c13a6-4853-40b2-9496-ac50f59b1bf8\") " Nov 03 13:25:01.515157 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:01.515103 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c08c13a6-4853-40b2-9496-ac50f59b1bf8-istio-token\" (UniqueName: \"kubernetes.io/projected/c08c13a6-4853-40b2-9496-ac50f59b1bf8-istio-token\") pod \"c08c13a6-4853-40b2-9496-ac50f59b1bf8\" (UID: \"c08c13a6-4853-40b2-9496-ac50f59b1bf8\") " Nov 03 13:25:01.515255 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:01.515226 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c08c13a6-4853-40b2-9496-ac50f59b1bf8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c08c13a6-4853-40b2-9496-ac50f59b1bf8-credential-socket\") pod \"c08c13a6-4853-40b2-9496-ac50f59b1bf8\" (UID: \"c08c13a6-4853-40b2-9496-ac50f59b1bf8\") " Nov 03 13:25:01.515324 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:01.515269 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c08c13a6-4853-40b2-9496-ac50f59b1bf8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c08c13a6-4853-40b2-9496-ac50f59b1bf8-workload-certs\") pod \"c08c13a6-4853-40b2-9496-ac50f59b1bf8\" (UID: \"c08c13a6-4853-40b2-9496-ac50f59b1bf8\") " Nov 03 13:25:01.515383 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:01.515329 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c08c13a6-4853-40b2-9496-ac50f59b1bf8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c08c13a6-4853-40b2-9496-ac50f59b1bf8-istio-data\") pod \"c08c13a6-4853-40b2-9496-ac50f59b1bf8\" (UID: \"c08c13a6-4853-40b2-9496-ac50f59b1bf8\") " Nov 03 13:25:01.515441 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:01.515397 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c08c13a6-4853-40b2-9496-ac50f59b1bf8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c08c13a6-4853-40b2-9496-ac50f59b1bf8-istio-envoy\") pod \"c08c13a6-4853-40b2-9496-ac50f59b1bf8\" (UID: \"c08c13a6-4853-40b2-9496-ac50f59b1bf8\") " Nov 03 13:25:01.519460 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:01.519424 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c08c13a6-4853-40b2-9496-ac50f59b1bf8-istiod-ca-cert" pod "c08c13a6-4853-40b2-9496-ac50f59b1bf8" (UID: "c08c13a6-4853-40b2-9496-ac50f59b1bf8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:25:01.521061 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:01.521016 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c08c13a6-4853-40b2-9496-ac50f59b1bf8-workload-socket" pod "c08c13a6-4853-40b2-9496-ac50f59b1bf8" (UID: "c08c13a6-4853-40b2-9496-ac50f59b1bf8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:25:01.521791 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:01.521769 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c08c13a6-4853-40b2-9496-ac50f59b1bf8-istio-data" pod "c08c13a6-4853-40b2-9496-ac50f59b1bf8" (UID: "c08c13a6-4853-40b2-9496-ac50f59b1bf8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:25:01.522027 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:01.521961 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c08c13a6-4853-40b2-9496-ac50f59b1bf8-workload-certs" pod "c08c13a6-4853-40b2-9496-ac50f59b1bf8" (UID: "c08c13a6-4853-40b2-9496-ac50f59b1bf8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:25:01.534483 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:01.522371 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c08c13a6-4853-40b2-9496-ac50f59b1bf8-credential-socket" pod "c08c13a6-4853-40b2-9496-ac50f59b1bf8" (UID: "c08c13a6-4853-40b2-9496-ac50f59b1bf8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:25:01.546060 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:01.546004 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c08c13a6-4853-40b2-9496-ac50f59b1bf8-istio-envoy" pod "c08c13a6-4853-40b2-9496-ac50f59b1bf8" (UID: "c08c13a6-4853-40b2-9496-ac50f59b1bf8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:25:01.546385 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:01.546359 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c08c13a6-4853-40b2-9496-ac50f59b1bf8-istio-podinfo" pod "c08c13a6-4853-40b2-9496-ac50f59b1bf8" (UID: "c08c13a6-4853-40b2-9496-ac50f59b1bf8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 13:25:01.546708 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:01.546689 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c08c13a6-4853-40b2-9496-ac50f59b1bf8-kube-api-access-l9q6t" pod "c08c13a6-4853-40b2-9496-ac50f59b1bf8" (UID: "c08c13a6-4853-40b2-9496-ac50f59b1bf8"). InnerVolumeSpecName "kube-api-access-l9q6t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:25:01.550063 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:01.550035 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c08c13a6-4853-40b2-9496-ac50f59b1bf8-istio-token" pod "c08c13a6-4853-40b2-9496-ac50f59b1bf8" (UID: "c08c13a6-4853-40b2-9496-ac50f59b1bf8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:25:01.601186 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:01.601147 5024 generic.go:356] "Generic (PLEG): container finished" podID="c08c13a6-4853-40b2-9496-ac50f59b1bf8" containerID="f7f63080bf0fb5b2a59903eb4d3bf9d86a386f978c8bbc70fee460d73433df0b" exitCode=0 Nov 03 13:25:01.601712 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:01.601249 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-a2ccd8-openshift-default-798b597bf7-5ljc2" event={"ID":"c08c13a6-4853-40b2-9496-ac50f59b1bf8","Type":"ContainerDied","Data":"f7f63080bf0fb5b2a59903eb4d3bf9d86a386f978c8bbc70fee460d73433df0b"} Nov 03 13:25:01.601712 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:01.601363 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-a2ccd8-openshift-default-798b597bf7-5ljc2" event={"ID":"c08c13a6-4853-40b2-9496-ac50f59b1bf8","Type":"ContainerDied","Data":"8fe4ccbbaee08675f98957a9fcf7cde2108c8e10ce0b4ec320b0dad2aae41786"} Nov 03 13:25:01.601712 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:01.601390 5024 scope.go:122] "RemoveContainer" containerID="f7f63080bf0fb5b2a59903eb4d3bf9d86a386f978c8bbc70fee460d73433df0b" Nov 03 13:25:01.601712 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:01.601527 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-a2ccd8-openshift-default-798b597bf7-5ljc2" Nov 03 13:25:01.605880 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:01.605848 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-1a18c8-openshift-default-67d7b5995c-z4csj" event={"ID":"168564ee-9452-4aba-8391-d4f0bdaafa7b","Type":"ContainerStarted","Data":"954b253b9e11610a4a8efa783f173c6d9a344093f76f73815f5b175465317dcf"} Nov 03 13:25:01.606030 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:01.605899 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-1a18c8-openshift-default-67d7b5995c-z4csj" event={"ID":"168564ee-9452-4aba-8391-d4f0bdaafa7b","Type":"ContainerStarted","Data":"4f72ab5f299049e65c8efd2cb529c1aecfabbb1d8980d03c4aad6ffbe9ed1679"} Nov 03 13:25:01.619016 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:01.618111 5024 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c08c13a6-4853-40b2-9496-ac50f59b1bf8-istio-envoy\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:25:01.619016 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:01.618147 5024 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c08c13a6-4853-40b2-9496-ac50f59b1bf8-istio-podinfo\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:25:01.619016 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:01.618176 5024 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c08c13a6-4853-40b2-9496-ac50f59b1bf8-workload-socket\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:25:01.619016 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:01.618192 5024 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c08c13a6-4853-40b2-9496-ac50f59b1bf8-istiod-ca-cert\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:25:01.619016 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:01.618209 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l9q6t\" (UniqueName: \"kubernetes.io/projected/c08c13a6-4853-40b2-9496-ac50f59b1bf8-kube-api-access-l9q6t\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:25:01.619016 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:01.618224 5024 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c08c13a6-4853-40b2-9496-ac50f59b1bf8-istio-token\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:25:01.619016 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:01.618238 5024 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c08c13a6-4853-40b2-9496-ac50f59b1bf8-credential-socket\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:25:01.619016 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:01.618263 5024 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c08c13a6-4853-40b2-9496-ac50f59b1bf8-workload-certs\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:25:01.619016 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:01.618277 5024 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c08c13a6-4853-40b2-9496-ac50f59b1bf8-istio-data\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:25:01.621449 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:01.621381 5024 scope.go:122] "RemoveContainer" containerID="13962ec05066f1acad8d5e943e5ed0f14c8b9d9fa9c29908e2223765a48932d4" Nov 03 13:25:01.634174 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:01.633147 5024 scope.go:122] "RemoveContainer" containerID="f7f63080bf0fb5b2a59903eb4d3bf9d86a386f978c8bbc70fee460d73433df0b" Nov 03 13:25:01.634174 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:25:01.633859 5024 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f7f63080bf0fb5b2a59903eb4d3bf9d86a386f978c8bbc70fee460d73433df0b\": container with ID starting with f7f63080bf0fb5b2a59903eb4d3bf9d86a386f978c8bbc70fee460d73433df0b not found: ID does not exist" containerID="f7f63080bf0fb5b2a59903eb4d3bf9d86a386f978c8bbc70fee460d73433df0b" Nov 03 13:25:01.634174 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:01.633886 5024 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7f63080bf0fb5b2a59903eb4d3bf9d86a386f978c8bbc70fee460d73433df0b"} err="failed to get container status \"f7f63080bf0fb5b2a59903eb4d3bf9d86a386f978c8bbc70fee460d73433df0b\": rpc error: code = NotFound desc = could not find container \"f7f63080bf0fb5b2a59903eb4d3bf9d86a386f978c8bbc70fee460d73433df0b\": container with ID starting with f7f63080bf0fb5b2a59903eb4d3bf9d86a386f978c8bbc70fee460d73433df0b not found: ID does not exist" Nov 03 13:25:01.634174 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:01.633903 5024 scope.go:122] "RemoveContainer" containerID="13962ec05066f1acad8d5e943e5ed0f14c8b9d9fa9c29908e2223765a48932d4" Nov 03 13:25:01.634853 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:25:01.634243 5024 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"13962ec05066f1acad8d5e943e5ed0f14c8b9d9fa9c29908e2223765a48932d4\": container with ID starting with 13962ec05066f1acad8d5e943e5ed0f14c8b9d9fa9c29908e2223765a48932d4 not found: ID does not exist" containerID="13962ec05066f1acad8d5e943e5ed0f14c8b9d9fa9c29908e2223765a48932d4" Nov 03 13:25:01.634853 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:01.634307 5024 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13962ec05066f1acad8d5e943e5ed0f14c8b9d9fa9c29908e2223765a48932d4"} err="failed to get container status \"13962ec05066f1acad8d5e943e5ed0f14c8b9d9fa9c29908e2223765a48932d4\": rpc error: code = NotFound desc = could not find container \"13962ec05066f1acad8d5e943e5ed0f14c8b9d9fa9c29908e2223765a48932d4\": container with ID starting with 13962ec05066f1acad8d5e943e5ed0f14c8b9d9fa9c29908e2223765a48932d4 not found: ID does not exist" Nov 03 13:25:01.642354 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:01.642275 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-1a18c8-openshift-default-67d7b5995c-z4csj" podStartSLOduration=2.642256354 podStartE2EDuration="2.642256354s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:24:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:25:01.639257041 +0000 UTC m=+981.647448318" watchObservedRunningTime="2026-11-03 13:25:01.642256354 +0000 UTC m=+981.650447632" Nov 03 13:25:01.662060 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:01.662023 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-a2ccd8-openshift-default-798b597bf7-5ljc2"] Nov 03 13:25:01.668014 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:01.667934 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-a2ccd8-openshift-default-798b597bf7-5ljc2"] Nov 03 13:25:02.147331 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:02.147277 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c08c13a6-4853-40b2-9496-ac50f59b1bf8" path="/var/lib/kubelet/pods/c08c13a6-4853-40b2-9496-ac50f59b1bf8/volumes" Nov 03 13:25:02.230852 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:02.230815 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-1a18c8-openshift-default-67d7b5995c-z4csj" Nov 03 13:25:02.237399 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:02.237371 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-1a18c8-openshift-default-67d7b5995c-z4csj" Nov 03 13:25:02.618134 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:02.618100 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-1a18c8-openshift-default-67d7b5995c-z4csj" Nov 03 13:25:02.619602 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:02.619580 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-1a18c8-openshift-default-67d7b5995c-z4csj" Nov 03 13:25:05.586409 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:05.586366 5024 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-8444x4nfn" podUID="b6932011-3c9e-4a12-a243-3692d4b09614" containerName="payload-processing" containerID="cri-o://7b754bb4b3aa3dce84e00e399c58f3ad4a172e1f4a3a2262819f8acd37032738" gracePeriod=25 Nov 03 13:25:06.180354 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:06.180323 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-8444x4nfn" Nov 03 13:25:06.241568 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:06.241522 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b6932011-3c9e-4a12-a243-3692d4b09614-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b6932011-3c9e-4a12-a243-3692d4b09614-routing-overlay-volume\") pod \"b6932011-3c9e-4a12-a243-3692d4b09614\" (UID: \"b6932011-3c9e-4a12-a243-3692d4b09614\") " Nov 03 13:25:06.241757 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:06.241675 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6932011-3c9e-4a12-a243-3692d4b09614-kube-api-access-8tvlv\" (UniqueName: \"kubernetes.io/projected/b6932011-3c9e-4a12-a243-3692d4b09614-kube-api-access-8tvlv\") pod \"b6932011-3c9e-4a12-a243-3692d4b09614\" (UID: \"b6932011-3c9e-4a12-a243-3692d4b09614\") " Nov 03 13:25:06.241839 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:06.241774 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b6932011-3c9e-4a12-a243-3692d4b09614-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6932011-3c9e-4a12-a243-3692d4b09614-plugins-config-volume\") pod \"b6932011-3c9e-4a12-a243-3692d4b09614\" (UID: \"b6932011-3c9e-4a12-a243-3692d4b09614\") " Nov 03 13:25:06.241839 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:06.241820 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6932011-3c9e-4a12-a243-3692d4b09614-provider-credentials\" (UniqueName: \"kubernetes.io/projected/b6932011-3c9e-4a12-a243-3692d4b09614-provider-credentials\") pod \"b6932011-3c9e-4a12-a243-3692d4b09614\" (UID: \"b6932011-3c9e-4a12-a243-3692d4b09614\") " Nov 03 13:25:06.242029 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:06.241997 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6932011-3c9e-4a12-a243-3692d4b09614-routing-overlay-volume" pod "b6932011-3c9e-4a12-a243-3692d4b09614" (UID: "b6932011-3c9e-4a12-a243-3692d4b09614"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:25:06.242423 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:06.242394 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6932011-3c9e-4a12-a243-3692d4b09614-plugins-config-volume" pod "b6932011-3c9e-4a12-a243-3692d4b09614" (UID: "b6932011-3c9e-4a12-a243-3692d4b09614"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:25:06.242555 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:06.242531 5024 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b6932011-3c9e-4a12-a243-3692d4b09614-routing-overlay-volume\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:25:06.242679 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:06.242556 5024 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6932011-3c9e-4a12-a243-3692d4b09614-plugins-config-volume\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:25:06.244802 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:06.244772 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6932011-3c9e-4a12-a243-3692d4b09614-provider-credentials" pod "b6932011-3c9e-4a12-a243-3692d4b09614" (UID: "b6932011-3c9e-4a12-a243-3692d4b09614"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:25:06.244897 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:06.244799 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6932011-3c9e-4a12-a243-3692d4b09614-kube-api-access-8tvlv" pod "b6932011-3c9e-4a12-a243-3692d4b09614" (UID: "b6932011-3c9e-4a12-a243-3692d4b09614"). InnerVolumeSpecName "kube-api-access-8tvlv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:25:06.343822 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:06.343783 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8tvlv\" (UniqueName: \"kubernetes.io/projected/b6932011-3c9e-4a12-a243-3692d4b09614-kube-api-access-8tvlv\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:25:06.343822 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:06.343816 5024 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b6932011-3c9e-4a12-a243-3692d4b09614-provider-credentials\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:25:06.662319 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:06.662249 5024 generic.go:356] "Generic (PLEG): container finished" podID="b6932011-3c9e-4a12-a243-3692d4b09614" containerID="7b754bb4b3aa3dce84e00e399c58f3ad4a172e1f4a3a2262819f8acd37032738" exitCode=0 Nov 03 13:25:06.662761 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:06.662345 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-8444x4nfn" event={"ID":"b6932011-3c9e-4a12-a243-3692d4b09614","Type":"ContainerDied","Data":"7b754bb4b3aa3dce84e00e399c58f3ad4a172e1f4a3a2262819f8acd37032738"} Nov 03 13:25:06.662761 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:06.662404 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-8444x4nfn" event={"ID":"b6932011-3c9e-4a12-a243-3692d4b09614","Type":"ContainerDied","Data":"c9f098367c7ad043a7007e8a3f3e7eeb079776566d7230367ae873e048f20c50"} Nov 03 13:25:06.662761 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:06.662427 5024 scope.go:122] "RemoveContainer" containerID="7b754bb4b3aa3dce84e00e399c58f3ad4a172e1f4a3a2262819f8acd37032738" Nov 03 13:25:06.662761 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:06.662356 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-8444x4nfn" Nov 03 13:25:06.675949 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:06.675926 5024 scope.go:122] "RemoveContainer" containerID="7b754bb4b3aa3dce84e00e399c58f3ad4a172e1f4a3a2262819f8acd37032738" Nov 03 13:25:06.676323 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:25:06.676294 5024 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b754bb4b3aa3dce84e00e399c58f3ad4a172e1f4a3a2262819f8acd37032738\": container with ID starting with 7b754bb4b3aa3dce84e00e399c58f3ad4a172e1f4a3a2262819f8acd37032738 not found: ID does not exist" containerID="7b754bb4b3aa3dce84e00e399c58f3ad4a172e1f4a3a2262819f8acd37032738" Nov 03 13:25:06.676391 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:06.676332 5024 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b754bb4b3aa3dce84e00e399c58f3ad4a172e1f4a3a2262819f8acd37032738"} err="failed to get container status \"7b754bb4b3aa3dce84e00e399c58f3ad4a172e1f4a3a2262819f8acd37032738\": rpc error: code = NotFound desc = could not find container \"7b754bb4b3aa3dce84e00e399c58f3ad4a172e1f4a3a2262819f8acd37032738\": container with ID starting with 7b754bb4b3aa3dce84e00e399c58f3ad4a172e1f4a3a2262819f8acd37032738 not found: ID does not exist" Nov 03 13:25:06.702344 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:06.697157 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-8444x4nfn"] Nov 03 13:25:06.702344 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:06.701944 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-8444x4nfn"] Nov 03 13:25:06.756609 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:06.756574 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-021a5a5c-58589f959-92jwh"] Nov 03 13:25:06.757196 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:06.757182 5024 cpu_manager.go:414] "containerMap: removing container" podUID="c08c13a6-4853-40b2-9496-ac50f59b1bf8" containerName="istio-proxy" Nov 03 13:25:06.757241 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:06.757197 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c08c13a6-4853-40b2-9496-ac50f59b1bf8" containerName="istio-proxy" Nov 03 13:25:06.757241 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:06.757210 5024 cpu_manager.go:414] "containerMap: removing container" podUID="c08c13a6-4853-40b2-9496-ac50f59b1bf8" containerName="istio-proxy" Nov 03 13:25:06.757241 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:06.757216 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c08c13a6-4853-40b2-9496-ac50f59b1bf8" containerName="istio-proxy" Nov 03 13:25:06.757350 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:06.757241 5024 cpu_manager.go:414] "containerMap: removing container" podUID="b6932011-3c9e-4a12-a243-3692d4b09614" containerName="payload-processing" Nov 03 13:25:06.757350 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:06.757247 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6932011-3c9e-4a12-a243-3692d4b09614" containerName="payload-processing" Nov 03 13:25:06.757412 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:06.757370 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6932011-3c9e-4a12-a243-3692d4b09614" containerName="payload-processing" Nov 03 13:25:06.757412 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:06.757386 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="c08c13a6-4853-40b2-9496-ac50f59b1bf8" containerName="istio-proxy" Nov 03 13:25:06.762115 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:06.762094 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-021a5a5c-58589f959-92jwh" Nov 03 13:25:06.767126 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:06.767098 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-021a5a5c\"" Nov 03 13:25:06.772587 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:06.772556 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-021a5a5c-58589f959-92jwh"] Nov 03 13:25:06.802318 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:06.802271 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-021a5a5c-859d56dfc9-6prfw"] Nov 03 13:25:06.803587 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:06.803563 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="c08c13a6-4853-40b2-9496-ac50f59b1bf8" containerName="istio-proxy" Nov 03 13:25:06.808738 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:06.808710 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-021a5a5c-859d56dfc9-6prfw" Nov 03 13:25:06.816955 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:06.816926 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-021a5a5c-859d56dfc9-6prfw"] Nov 03 13:25:06.884311 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:06.879084 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-021a5a5c-dockercfg-qlbmm\"" Nov 03 13:25:06.955376 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:06.954679 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab04f246-fadf-4901-b65b-523bfdaf7e0a-plugins-config-volume\") pod \"payload-processing-e2e-ait-021a5a5c-859d56dfc9-6prfw\" (UID: \"ab04f246-fadf-4901-b65b-523bfdaf7e0a\") " pod="openshift-ingress/payload-processing-e2e-ait-021a5a5c-859d56dfc9-6prfw" Nov 03 13:25:06.955376 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:06.954759 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4f6e2be-e468-4cf1-b7a3-ae8f9ee5e61b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-021a5a5c-58589f959-92jwh\" (UID: \"d4f6e2be-e468-4cf1-b7a3-ae8f9ee5e61b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-021a5a5c-58589f959-92jwh" Nov 03 13:25:06.955376 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:06.954813 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-czprz\" (UniqueName: \"kubernetes.io/projected/d4f6e2be-e468-4cf1-b7a3-ae8f9ee5e61b-kube-api-access-czprz\") pod \"payload-pre-processing-e2e-ait-021a5a5c-58589f959-92jwh\" (UID: \"d4f6e2be-e468-4cf1-b7a3-ae8f9ee5e61b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-021a5a5c-58589f959-92jwh" Nov 03 13:25:06.955376 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:06.954842 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rm5t9\" (UniqueName: \"kubernetes.io/projected/ab04f246-fadf-4901-b65b-523bfdaf7e0a-kube-api-access-rm5t9\") pod \"payload-processing-e2e-ait-021a5a5c-859d56dfc9-6prfw\" (UID: \"ab04f246-fadf-4901-b65b-523bfdaf7e0a\") " pod="openshift-ingress/payload-processing-e2e-ait-021a5a5c-859d56dfc9-6prfw" Nov 03 13:25:07.063330 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:07.057457 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab04f246-fadf-4901-b65b-523bfdaf7e0a-plugins-config-volume\") pod \"payload-processing-e2e-ait-021a5a5c-859d56dfc9-6prfw\" (UID: \"ab04f246-fadf-4901-b65b-523bfdaf7e0a\") " pod="openshift-ingress/payload-processing-e2e-ait-021a5a5c-859d56dfc9-6prfw" Nov 03 13:25:07.063330 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:07.057549 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4f6e2be-e468-4cf1-b7a3-ae8f9ee5e61b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-021a5a5c-58589f959-92jwh\" (UID: \"d4f6e2be-e468-4cf1-b7a3-ae8f9ee5e61b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-021a5a5c-58589f959-92jwh" Nov 03 13:25:07.063330 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:07.057613 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-czprz\" (UniqueName: \"kubernetes.io/projected/d4f6e2be-e468-4cf1-b7a3-ae8f9ee5e61b-kube-api-access-czprz\") pod \"payload-pre-processing-e2e-ait-021a5a5c-58589f959-92jwh\" (UID: \"d4f6e2be-e468-4cf1-b7a3-ae8f9ee5e61b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-021a5a5c-58589f959-92jwh" Nov 03 13:25:07.063330 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:07.057641 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rm5t9\" (UniqueName: \"kubernetes.io/projected/ab04f246-fadf-4901-b65b-523bfdaf7e0a-kube-api-access-rm5t9\") pod \"payload-processing-e2e-ait-021a5a5c-859d56dfc9-6prfw\" (UID: \"ab04f246-fadf-4901-b65b-523bfdaf7e0a\") " pod="openshift-ingress/payload-processing-e2e-ait-021a5a5c-859d56dfc9-6prfw" Nov 03 13:25:07.067500 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:07.065192 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4f6e2be-e468-4cf1-b7a3-ae8f9ee5e61b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-021a5a5c-58589f959-92jwh\" (UID: \"d4f6e2be-e468-4cf1-b7a3-ae8f9ee5e61b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-021a5a5c-58589f959-92jwh" Nov 03 13:25:07.072964 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:07.072799 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab04f246-fadf-4901-b65b-523bfdaf7e0a-plugins-config-volume\") pod \"payload-processing-e2e-ait-021a5a5c-859d56dfc9-6prfw\" (UID: \"ab04f246-fadf-4901-b65b-523bfdaf7e0a\") " pod="openshift-ingress/payload-processing-e2e-ait-021a5a5c-859d56dfc9-6prfw" Nov 03 13:25:07.076423 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:07.076392 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rm5t9\" (UniqueName: \"kubernetes.io/projected/ab04f246-fadf-4901-b65b-523bfdaf7e0a-kube-api-access-rm5t9\") pod \"payload-processing-e2e-ait-021a5a5c-859d56dfc9-6prfw\" (UID: \"ab04f246-fadf-4901-b65b-523bfdaf7e0a\") " pod="openshift-ingress/payload-processing-e2e-ait-021a5a5c-859d56dfc9-6prfw" Nov 03 13:25:07.119700 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:07.118424 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-czprz\" (UniqueName: \"kubernetes.io/projected/d4f6e2be-e468-4cf1-b7a3-ae8f9ee5e61b-kube-api-access-czprz\") pod \"payload-pre-processing-e2e-ait-021a5a5c-58589f959-92jwh\" (UID: \"d4f6e2be-e468-4cf1-b7a3-ae8f9ee5e61b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-021a5a5c-58589f959-92jwh" Nov 03 13:25:07.132509 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:07.131656 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-021a5a5c-859d56dfc9-6prfw" Nov 03 13:25:07.388311 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:07.386137 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-021a5a5c-58589f959-92jwh" Nov 03 13:25:07.445847 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:07.445787 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-021a5a5c-859d56dfc9-6prfw"] Nov 03 13:25:07.448449 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:25:07.448409 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab04f246_fadf_4901_b65b_523bfdaf7e0a.slice/crio-3ecc5f3e0d8d55d63025948592ed21882f5db349622f998b173f57ae124c0fab WatchSource:0}: Error finding container 3ecc5f3e0d8d55d63025948592ed21882f5db349622f998b173f57ae124c0fab: Status 404 returned error can't find the container with id 3ecc5f3e0d8d55d63025948592ed21882f5db349622f998b173f57ae124c0fab Nov 03 13:25:07.670895 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:07.670852 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-021a5a5c-859d56dfc9-6prfw" event={"ID":"ab04f246-fadf-4901-b65b-523bfdaf7e0a","Type":"ContainerStarted","Data":"b5617245bf97485772fed59b74bc26efca91f0cc2f85d1ff548a0ffd40f276a5"} Nov 03 13:25:07.671382 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:07.670933 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-021a5a5c-859d56dfc9-6prfw" Nov 03 13:25:07.671382 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:07.670959 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-021a5a5c-859d56dfc9-6prfw" event={"ID":"ab04f246-fadf-4901-b65b-523bfdaf7e0a","Type":"ContainerStarted","Data":"3ecc5f3e0d8d55d63025948592ed21882f5db349622f998b173f57ae124c0fab"} Nov 03 13:25:07.689151 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:07.689095 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-021a5a5c-859d56dfc9-6prfw" podStartSLOduration=1.689081777 podStartE2EDuration="1.689081777s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:25:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:25:07.687755106 +0000 UTC m=+987.695946384" watchObservedRunningTime="2026-11-03 13:25:07.689081777 +0000 UTC m=+987.697273054" Nov 03 13:25:07.726150 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:07.726112 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-021a5a5c-58589f959-92jwh"] Nov 03 13:25:07.726605 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:25:07.726576 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4f6e2be_e468_4cf1_b7a3_ae8f9ee5e61b.slice/crio-af093d25e7e9781f05025d56f93c0dbc4b481cbab45e8a2e1216a382f68b72d8 WatchSource:0}: Error finding container af093d25e7e9781f05025d56f93c0dbc4b481cbab45e8a2e1216a382f68b72d8: Status 404 returned error can't find the container with id af093d25e7e9781f05025d56f93c0dbc4b481cbab45e8a2e1216a382f68b72d8 Nov 03 13:25:08.124312 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:08.124063 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b6932011-3c9e-4a12-a243-3692d4b09614" path="/var/lib/kubelet/pods/b6932011-3c9e-4a12-a243-3692d4b09614/volumes" Nov 03 13:25:08.680919 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:08.680876 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-021a5a5c-58589f959-92jwh" event={"ID":"d4f6e2be-e468-4cf1-b7a3-ae8f9ee5e61b","Type":"ContainerStarted","Data":"910f5ef44662c553afee2c510a36c4ced1cdc8b171ad5c129c73de0ebc6269c6"} Nov 03 13:25:08.681432 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:08.680944 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-021a5a5c-58589f959-92jwh" event={"ID":"d4f6e2be-e468-4cf1-b7a3-ae8f9ee5e61b","Type":"ContainerStarted","Data":"af093d25e7e9781f05025d56f93c0dbc4b481cbab45e8a2e1216a382f68b72d8"} Nov 03 13:25:08.681432 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:08.681027 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-021a5a5c-58589f959-92jwh" Nov 03 13:25:08.703844 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:08.703788 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-021a5a5c-58589f959-92jwh" podStartSLOduration=2.70377557 podStartE2EDuration="2.70377557s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:25:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:25:08.699428335 +0000 UTC m=+988.707619612" watchObservedRunningTime="2026-11-03 13:25:08.70377557 +0000 UTC m=+988.711966847" Nov 03 13:25:08.730264 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:08.730153 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-b26c22-cc548f947-ddsxv"] Nov 03 13:25:09.511992 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:25:09.511943 5024 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d7mwz" containerName="extractor" Nov 03 13:25:11.024922 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:11.024888 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-1a18c8-d6c758bcc-pmrxw"] Nov 03 13:25:11.031950 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:11.031919 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-1a18c8-d6c758bcc-pmrxw" Nov 03 13:25:11.041818 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:11.041782 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-b-1a18c8-serving-cert\"" Nov 03 13:25:11.043053 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:11.043022 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-1a18c8-d6c758bcc-pmrxw"] Nov 03 13:25:11.054244 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:11.054206 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mqzb7\" (UniqueName: \"kubernetes.io/projected/937216cd-5ccd-43e1-8b64-a4d31af284d6-kube-api-access-mqzb7\") pod \"maas-api-e2e-extmod-b-1a18c8-d6c758bcc-pmrxw\" (UID: \"937216cd-5ccd-43e1-8b64-a4d31af284d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-1a18c8-d6c758bcc-pmrxw" Nov 03 13:25:11.054462 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:11.054406 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/937216cd-5ccd-43e1-8b64-a4d31af284d6-maas-api-tls\") pod \"maas-api-e2e-extmod-b-1a18c8-d6c758bcc-pmrxw\" (UID: \"937216cd-5ccd-43e1-8b64-a4d31af284d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-1a18c8-d6c758bcc-pmrxw" Nov 03 13:25:11.054596 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:11.054571 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/937216cd-5ccd-43e1-8b64-a4d31af284d6-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-1a18c8-d6c758bcc-pmrxw\" (UID: \"937216cd-5ccd-43e1-8b64-a4d31af284d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-1a18c8-d6c758bcc-pmrxw" Nov 03 13:25:11.054658 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:11.054611 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/937216cd-5ccd-43e1-8b64-a4d31af284d6-metrics-certs\") pod \"maas-api-e2e-extmod-b-1a18c8-d6c758bcc-pmrxw\" (UID: \"937216cd-5ccd-43e1-8b64-a4d31af284d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-1a18c8-d6c758bcc-pmrxw" Nov 03 13:25:11.219308 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:11.217511 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/937216cd-5ccd-43e1-8b64-a4d31af284d6-metrics-certs\") pod \"maas-api-e2e-extmod-b-1a18c8-d6c758bcc-pmrxw\" (UID: \"937216cd-5ccd-43e1-8b64-a4d31af284d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-1a18c8-d6c758bcc-pmrxw" Nov 03 13:25:11.219308 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:11.217692 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mqzb7\" (UniqueName: \"kubernetes.io/projected/937216cd-5ccd-43e1-8b64-a4d31af284d6-kube-api-access-mqzb7\") pod \"maas-api-e2e-extmod-b-1a18c8-d6c758bcc-pmrxw\" (UID: \"937216cd-5ccd-43e1-8b64-a4d31af284d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-1a18c8-d6c758bcc-pmrxw" Nov 03 13:25:11.219308 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:11.217793 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/937216cd-5ccd-43e1-8b64-a4d31af284d6-maas-api-tls\") pod \"maas-api-e2e-extmod-b-1a18c8-d6c758bcc-pmrxw\" (UID: \"937216cd-5ccd-43e1-8b64-a4d31af284d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-1a18c8-d6c758bcc-pmrxw" Nov 03 13:25:11.219308 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:11.217971 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/937216cd-5ccd-43e1-8b64-a4d31af284d6-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-1a18c8-d6c758bcc-pmrxw\" (UID: \"937216cd-5ccd-43e1-8b64-a4d31af284d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-1a18c8-d6c758bcc-pmrxw" Nov 03 13:25:11.249307 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:11.238234 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/937216cd-5ccd-43e1-8b64-a4d31af284d6-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-1a18c8-d6c758bcc-pmrxw\" (UID: \"937216cd-5ccd-43e1-8b64-a4d31af284d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-1a18c8-d6c758bcc-pmrxw" Nov 03 13:25:11.263308 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:11.258190 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mqzb7\" (UniqueName: \"kubernetes.io/projected/937216cd-5ccd-43e1-8b64-a4d31af284d6-kube-api-access-mqzb7\") pod \"maas-api-e2e-extmod-b-1a18c8-d6c758bcc-pmrxw\" (UID: \"937216cd-5ccd-43e1-8b64-a4d31af284d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-1a18c8-d6c758bcc-pmrxw" Nov 03 13:25:11.295085 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:11.294899 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/937216cd-5ccd-43e1-8b64-a4d31af284d6-maas-api-tls\") pod \"maas-api-e2e-extmod-b-1a18c8-d6c758bcc-pmrxw\" (UID: \"937216cd-5ccd-43e1-8b64-a4d31af284d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-1a18c8-d6c758bcc-pmrxw" Nov 03 13:25:11.316323 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:11.308027 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/937216cd-5ccd-43e1-8b64-a4d31af284d6-metrics-certs\") pod \"maas-api-e2e-extmod-b-1a18c8-d6c758bcc-pmrxw\" (UID: \"937216cd-5ccd-43e1-8b64-a4d31af284d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-1a18c8-d6c758bcc-pmrxw" Nov 03 13:25:11.360892 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:11.360541 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-1a18c8-d6c758bcc-pmrxw" Nov 03 13:25:11.772198 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:11.769021 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-1a18c8-d6c758bcc-pmrxw"] Nov 03 13:25:12.849943 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:12.849901 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-1a18c8-d6c758bcc-pmrxw" event={"ID":"937216cd-5ccd-43e1-8b64-a4d31af284d6","Type":"ContainerStarted","Data":"f1b5973ec3f4f63798b5167ecce777ae1841dd246b521f25db2ab7e1db0f7538"} Nov 03 13:25:12.850655 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:12.850628 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-1a18c8-d6c758bcc-pmrxw" event={"ID":"937216cd-5ccd-43e1-8b64-a4d31af284d6","Type":"ContainerStarted","Data":"413a30f11b8aba299dd4b9111ce51845820ce43cea1527af60e7c82f521dbb0a"} Nov 03 13:25:12.850846 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:12.850830 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-1a18c8-d6c758bcc-pmrxw" Nov 03 13:25:12.879275 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:12.879216 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-1a18c8-d6c758bcc-pmrxw" podStartSLOduration=2.488419808 podStartE2EDuration="2.879202274s" totalImagesPullingTime="390.782466ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:25:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:25:12.877202392 +0000 UTC m=+992.885393701" watchObservedRunningTime="2026-11-03 13:25:12.879202274 +0000 UTC m=+992.887393551" Nov 03 13:25:13.731461 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:13.731390 5024 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-b26c22-cc548f947-ddsxv" podUID="1c05ee6f-6019-4020-8d8e-a8fb87a42565" containerName="payload-pre-processing" containerID="cri-o://a06338f012595ad3e22ac5ef1424f2485da6402cc32aef7393ac6a5741fcf380" gracePeriod=25 Nov 03 13:25:13.858629 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:13.858580 5024 generic.go:356] "Generic (PLEG): container finished" podID="1c05ee6f-6019-4020-8d8e-a8fb87a42565" containerID="a06338f012595ad3e22ac5ef1424f2485da6402cc32aef7393ac6a5741fcf380" exitCode=0 Nov 03 13:25:13.859026 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:13.858657 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-b26c22-cc548f947-ddsxv" event={"ID":"1c05ee6f-6019-4020-8d8e-a8fb87a42565","Type":"ContainerDied","Data":"a06338f012595ad3e22ac5ef1424f2485da6402cc32aef7393ac6a5741fcf380"} Nov 03 13:25:14.033475 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:14.033448 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-b26c22-cc548f947-ddsxv" Nov 03 13:25:14.078412 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:14.076514 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c05ee6f-6019-4020-8d8e-a8fb87a42565-kube-api-access-cnvkp\" (UniqueName: \"kubernetes.io/projected/1c05ee6f-6019-4020-8d8e-a8fb87a42565-kube-api-access-cnvkp\") pod \"1c05ee6f-6019-4020-8d8e-a8fb87a42565\" (UID: \"1c05ee6f-6019-4020-8d8e-a8fb87a42565\") " Nov 03 13:25:14.078412 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:14.076615 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c05ee6f-6019-4020-8d8e-a8fb87a42565-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c05ee6f-6019-4020-8d8e-a8fb87a42565-plugins-config-volume\") pod \"1c05ee6f-6019-4020-8d8e-a8fb87a42565\" (UID: \"1c05ee6f-6019-4020-8d8e-a8fb87a42565\") " Nov 03 13:25:14.078412 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:14.077197 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c05ee6f-6019-4020-8d8e-a8fb87a42565-plugins-config-volume" pod "1c05ee6f-6019-4020-8d8e-a8fb87a42565" (UID: "1c05ee6f-6019-4020-8d8e-a8fb87a42565"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:25:14.078412 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:14.077969 5024 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c05ee6f-6019-4020-8d8e-a8fb87a42565-plugins-config-volume\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:25:14.079722 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:14.079354 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c05ee6f-6019-4020-8d8e-a8fb87a42565-kube-api-access-cnvkp" pod "1c05ee6f-6019-4020-8d8e-a8fb87a42565" (UID: "1c05ee6f-6019-4020-8d8e-a8fb87a42565"). InnerVolumeSpecName "kube-api-access-cnvkp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:25:14.192307 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:14.191409 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cnvkp\" (UniqueName: \"kubernetes.io/projected/1c05ee6f-6019-4020-8d8e-a8fb87a42565-kube-api-access-cnvkp\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:25:14.596309 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:14.588223 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-b26c22-77d4fd9fb-wtf7j"] Nov 03 13:25:14.596309 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:14.588564 5024 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b26c22-77d4fd9fb-wtf7j" podUID="24b840c8-af1d-41da-8b95-b70f13c93125" containerName="maas-api" containerID="cri-o://00924c985904874527910172f45cf969c1681be901fdf724d8c36ab39065d551" gracePeriod=30 Nov 03 13:25:15.013309 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:15.004715 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-b26c22-cc548f947-ddsxv" event={"ID":"1c05ee6f-6019-4020-8d8e-a8fb87a42565","Type":"ContainerDied","Data":"3541b29ccdf91aab6698b43d6f05bada689c6a209c4cc54352797c62851c052e"} Nov 03 13:25:15.013309 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:15.004794 5024 scope.go:122] "RemoveContainer" containerID="a06338f012595ad3e22ac5ef1424f2485da6402cc32aef7393ac6a5741fcf380" Nov 03 13:25:15.013309 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:15.004986 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-b26c22-cc548f947-ddsxv" Nov 03 13:25:15.360552 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:15.360480 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-b26c22-cc548f947-ddsxv"] Nov 03 13:25:15.365015 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:15.364471 5024 generic.go:356] "Generic (PLEG): container finished" podID="24b840c8-af1d-41da-8b95-b70f13c93125" containerID="00924c985904874527910172f45cf969c1681be901fdf724d8c36ab39065d551" exitCode=0 Nov 03 13:25:15.365015 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:15.364705 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b26c22-77d4fd9fb-wtf7j" event={"ID":"24b840c8-af1d-41da-8b95-b70f13c93125","Type":"ContainerDied","Data":"00924c985904874527910172f45cf969c1681be901fdf724d8c36ab39065d551"} Nov 03 13:25:15.366434 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:15.366405 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-b26c22-cc548f947-ddsxv"] Nov 03 13:25:15.406856 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:15.406827 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b26c22-77d4fd9fb-wtf7j" Nov 03 13:25:15.482653 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:15.481225 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24b840c8-af1d-41da-8b95-b70f13c93125-kube-api-access-ntp67\" (UniqueName: \"kubernetes.io/projected/24b840c8-af1d-41da-8b95-b70f13c93125-kube-api-access-ntp67\") pod \"24b840c8-af1d-41da-8b95-b70f13c93125\" (UID: \"24b840c8-af1d-41da-8b95-b70f13c93125\") " Nov 03 13:25:15.482653 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:15.481332 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/24b840c8-af1d-41da-8b95-b70f13c93125-metrics-certs\" (UniqueName: \"kubernetes.io/secret/24b840c8-af1d-41da-8b95-b70f13c93125-metrics-certs\") pod \"24b840c8-af1d-41da-8b95-b70f13c93125\" (UID: \"24b840c8-af1d-41da-8b95-b70f13c93125\") " Nov 03 13:25:15.482653 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:15.481401 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/24b840c8-af1d-41da-8b95-b70f13c93125-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/24b840c8-af1d-41da-8b95-b70f13c93125-maas-api-tls\") pod \"24b840c8-af1d-41da-8b95-b70f13c93125\" (UID: \"24b840c8-af1d-41da-8b95-b70f13c93125\") " Nov 03 13:25:15.482653 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:15.481445 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/24b840c8-af1d-41da-8b95-b70f13c93125-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/24b840c8-af1d-41da-8b95-b70f13c93125-openshift-service-ca\") pod \"24b840c8-af1d-41da-8b95-b70f13c93125\" (UID: \"24b840c8-af1d-41da-8b95-b70f13c93125\") " Nov 03 13:25:15.494312 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:15.494201 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24b840c8-af1d-41da-8b95-b70f13c93125-openshift-service-ca" pod "24b840c8-af1d-41da-8b95-b70f13c93125" (UID: "24b840c8-af1d-41da-8b95-b70f13c93125"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:25:15.541914 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:15.541724 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24b840c8-af1d-41da-8b95-b70f13c93125-kube-api-access-ntp67" pod "24b840c8-af1d-41da-8b95-b70f13c93125" (UID: "24b840c8-af1d-41da-8b95-b70f13c93125"). InnerVolumeSpecName "kube-api-access-ntp67". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:25:15.541914 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:15.541877 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/24b840c8-af1d-41da-8b95-b70f13c93125-metrics-certs" pod "24b840c8-af1d-41da-8b95-b70f13c93125" (UID: "24b840c8-af1d-41da-8b95-b70f13c93125"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 13:25:15.544692 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:15.544648 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/24b840c8-af1d-41da-8b95-b70f13c93125-maas-api-tls" pod "24b840c8-af1d-41da-8b95-b70f13c93125" (UID: "24b840c8-af1d-41da-8b95-b70f13c93125"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 13:25:15.596324 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:15.583781 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ntp67\" (UniqueName: \"kubernetes.io/projected/24b840c8-af1d-41da-8b95-b70f13c93125-kube-api-access-ntp67\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:25:15.596324 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:15.583828 5024 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/24b840c8-af1d-41da-8b95-b70f13c93125-metrics-certs\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:25:15.596324 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:15.583845 5024 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/24b840c8-af1d-41da-8b95-b70f13c93125-maas-api-tls\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:25:15.596324 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:15.583896 5024 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/24b840c8-af1d-41da-8b95-b70f13c93125-openshift-service-ca\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:25:16.167315 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:16.166648 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c05ee6f-6019-4020-8d8e-a8fb87a42565" path="/var/lib/kubelet/pods/1c05ee6f-6019-4020-8d8e-a8fb87a42565/volumes" Nov 03 13:25:16.259482 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:25:16.259424 5024 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24b840c8_af1d_41da_8b95_b70f13c93125.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24b840c8_af1d_41da_8b95_b70f13c93125.slice/crio-2e33395710ecb85eba7dc1e71e3372696162361c777a91f161fa9a2020d3c3ca\": RecentStats: unable to find data in memory cache]" Nov 03 13:25:16.375754 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:16.375723 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b26c22-77d4fd9fb-wtf7j" Nov 03 13:25:16.375754 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:16.375734 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b26c22-77d4fd9fb-wtf7j" event={"ID":"24b840c8-af1d-41da-8b95-b70f13c93125","Type":"ContainerDied","Data":"2e33395710ecb85eba7dc1e71e3372696162361c777a91f161fa9a2020d3c3ca"} Nov 03 13:25:16.375987 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:16.375807 5024 scope.go:122] "RemoveContainer" containerID="00924c985904874527910172f45cf969c1681be901fdf724d8c36ab39065d551" Nov 03 13:25:16.407695 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:16.407658 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-b26c22-77d4fd9fb-wtf7j"] Nov 03 13:25:16.411052 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:16.411013 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-b26c22-77d4fd9fb-wtf7j"] Nov 03 13:25:17.760197 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:17.760156 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-csthf"] Nov 03 13:25:17.760985 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:17.760961 5024 cpu_manager.go:414] "containerMap: removing container" podUID="1c05ee6f-6019-4020-8d8e-a8fb87a42565" containerName="payload-pre-processing" Nov 03 13:25:17.760985 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:17.760984 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c05ee6f-6019-4020-8d8e-a8fb87a42565" containerName="payload-pre-processing" Nov 03 13:25:17.761178 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:17.761036 5024 cpu_manager.go:414] "containerMap: removing container" podUID="24b840c8-af1d-41da-8b95-b70f13c93125" containerName="maas-api" Nov 03 13:25:17.761178 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:17.761043 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24b840c8-af1d-41da-8b95-b70f13c93125" containerName="maas-api" Nov 03 13:25:17.761178 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:17.761148 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="24b840c8-af1d-41da-8b95-b70f13c93125" containerName="maas-api" Nov 03 13:25:17.761178 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:17.761158 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c05ee6f-6019-4020-8d8e-a8fb87a42565" containerName="payload-pre-processing" Nov 03 13:25:17.766175 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:17.766144 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-csthf" Nov 03 13:25:17.781519 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:17.781485 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-csthf"] Nov 03 13:25:17.792754 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:17.792723 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w0-8f8aa8-openshift-default-dockercfg-cvr4j\"" Nov 03 13:25:17.868007 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:17.867963 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c69ea47-20cb-40de-94fb-add95bd74c10-workload-certs\") pod \"e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-csthf\" (UID: \"3c69ea47-20cb-40de-94fb-add95bd74c10\") " pod="openshift-ingress/e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-csthf" Nov 03 13:25:17.868517 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:17.868480 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c69ea47-20cb-40de-94fb-add95bd74c10-istio-token\") pod \"e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-csthf\" (UID: \"3c69ea47-20cb-40de-94fb-add95bd74c10\") " pod="openshift-ingress/e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-csthf" Nov 03 13:25:17.868648 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:17.868609 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c69ea47-20cb-40de-94fb-add95bd74c10-credential-socket\") pod \"e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-csthf\" (UID: \"3c69ea47-20cb-40de-94fb-add95bd74c10\") " pod="openshift-ingress/e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-csthf" Nov 03 13:25:17.868730 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:17.868668 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c69ea47-20cb-40de-94fb-add95bd74c10-istio-data\") pod \"e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-csthf\" (UID: \"3c69ea47-20cb-40de-94fb-add95bd74c10\") " pod="openshift-ingress/e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-csthf" Nov 03 13:25:17.868730 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:17.868692 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c69ea47-20cb-40de-94fb-add95bd74c10-istio-podinfo\") pod \"e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-csthf\" (UID: \"3c69ea47-20cb-40de-94fb-add95bd74c10\") " pod="openshift-ingress/e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-csthf" Nov 03 13:25:17.868843 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:17.868748 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c69ea47-20cb-40de-94fb-add95bd74c10-istio-envoy\") pod \"e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-csthf\" (UID: \"3c69ea47-20cb-40de-94fb-add95bd74c10\") " pod="openshift-ingress/e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-csthf" Nov 03 13:25:17.868843 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:17.868810 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qmfz7\" (UniqueName: \"kubernetes.io/projected/3c69ea47-20cb-40de-94fb-add95bd74c10-kube-api-access-qmfz7\") pod \"e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-csthf\" (UID: \"3c69ea47-20cb-40de-94fb-add95bd74c10\") " pod="openshift-ingress/e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-csthf" Nov 03 13:25:17.868978 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:17.868912 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c69ea47-20cb-40de-94fb-add95bd74c10-istiod-ca-cert\") pod \"e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-csthf\" (UID: \"3c69ea47-20cb-40de-94fb-add95bd74c10\") " pod="openshift-ingress/e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-csthf" Nov 03 13:25:17.868978 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:17.868947 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c69ea47-20cb-40de-94fb-add95bd74c10-workload-socket\") pod \"e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-csthf\" (UID: \"3c69ea47-20cb-40de-94fb-add95bd74c10\") " pod="openshift-ingress/e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-csthf" Nov 03 13:25:17.985311 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:17.982379 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c69ea47-20cb-40de-94fb-add95bd74c10-istiod-ca-cert\") pod \"e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-csthf\" (UID: \"3c69ea47-20cb-40de-94fb-add95bd74c10\") " pod="openshift-ingress/e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-csthf" Nov 03 13:25:17.985311 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:17.982444 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c69ea47-20cb-40de-94fb-add95bd74c10-workload-socket\") pod \"e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-csthf\" (UID: \"3c69ea47-20cb-40de-94fb-add95bd74c10\") " pod="openshift-ingress/e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-csthf" Nov 03 13:25:17.985311 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:17.982499 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c69ea47-20cb-40de-94fb-add95bd74c10-workload-certs\") pod \"e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-csthf\" (UID: \"3c69ea47-20cb-40de-94fb-add95bd74c10\") " pod="openshift-ingress/e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-csthf" Nov 03 13:25:17.985311 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:17.982594 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c69ea47-20cb-40de-94fb-add95bd74c10-istio-token\") pod \"e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-csthf\" (UID: \"3c69ea47-20cb-40de-94fb-add95bd74c10\") " pod="openshift-ingress/e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-csthf" Nov 03 13:25:17.985311 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:17.982709 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c69ea47-20cb-40de-94fb-add95bd74c10-credential-socket\") pod \"e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-csthf\" (UID: \"3c69ea47-20cb-40de-94fb-add95bd74c10\") " pod="openshift-ingress/e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-csthf" Nov 03 13:25:17.985311 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:17.982765 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c69ea47-20cb-40de-94fb-add95bd74c10-istio-data\") pod \"e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-csthf\" (UID: \"3c69ea47-20cb-40de-94fb-add95bd74c10\") " pod="openshift-ingress/e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-csthf" Nov 03 13:25:17.985311 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:17.982789 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c69ea47-20cb-40de-94fb-add95bd74c10-istio-podinfo\") pod \"e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-csthf\" (UID: \"3c69ea47-20cb-40de-94fb-add95bd74c10\") " pod="openshift-ingress/e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-csthf" Nov 03 13:25:17.985311 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:17.982845 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c69ea47-20cb-40de-94fb-add95bd74c10-istio-envoy\") pod \"e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-csthf\" (UID: \"3c69ea47-20cb-40de-94fb-add95bd74c10\") " pod="openshift-ingress/e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-csthf" Nov 03 13:25:17.985311 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:17.982910 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmfz7\" (UniqueName: \"kubernetes.io/projected/3c69ea47-20cb-40de-94fb-add95bd74c10-kube-api-access-qmfz7\") pod \"e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-csthf\" (UID: \"3c69ea47-20cb-40de-94fb-add95bd74c10\") " pod="openshift-ingress/e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-csthf" Nov 03 13:25:17.985311 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:17.984138 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c69ea47-20cb-40de-94fb-add95bd74c10-istiod-ca-cert\") pod \"e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-csthf\" (UID: \"3c69ea47-20cb-40de-94fb-add95bd74c10\") " pod="openshift-ingress/e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-csthf" Nov 03 13:25:17.985311 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:17.984451 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c69ea47-20cb-40de-94fb-add95bd74c10-workload-socket\") pod \"e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-csthf\" (UID: \"3c69ea47-20cb-40de-94fb-add95bd74c10\") " pod="openshift-ingress/e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-csthf" Nov 03 13:25:17.985311 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:17.984685 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c69ea47-20cb-40de-94fb-add95bd74c10-workload-certs\") pod \"e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-csthf\" (UID: \"3c69ea47-20cb-40de-94fb-add95bd74c10\") " pod="openshift-ingress/e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-csthf" Nov 03 13:25:17.985311 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:17.985177 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c69ea47-20cb-40de-94fb-add95bd74c10-credential-socket\") pod \"e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-csthf\" (UID: \"3c69ea47-20cb-40de-94fb-add95bd74c10\") " pod="openshift-ingress/e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-csthf" Nov 03 13:25:18.030356 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:18.003624 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmfz7\" (UniqueName: \"kubernetes.io/projected/3c69ea47-20cb-40de-94fb-add95bd74c10-kube-api-access-qmfz7\") pod \"e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-csthf\" (UID: \"3c69ea47-20cb-40de-94fb-add95bd74c10\") " pod="openshift-ingress/e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-csthf" Nov 03 13:25:18.063430 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:18.006221 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c69ea47-20cb-40de-94fb-add95bd74c10-istio-data\") pod \"e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-csthf\" (UID: \"3c69ea47-20cb-40de-94fb-add95bd74c10\") " pod="openshift-ingress/e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-csthf" Nov 03 13:25:18.063661 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:18.033945 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c69ea47-20cb-40de-94fb-add95bd74c10-istio-token\") pod \"e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-csthf\" (UID: \"3c69ea47-20cb-40de-94fb-add95bd74c10\") " pod="openshift-ingress/e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-csthf" Nov 03 13:25:18.063821 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:18.036863 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c69ea47-20cb-40de-94fb-add95bd74c10-istio-podinfo\") pod \"e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-csthf\" (UID: \"3c69ea47-20cb-40de-94fb-add95bd74c10\") " pod="openshift-ingress/e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-csthf" Nov 03 13:25:18.063923 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:18.050215 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c69ea47-20cb-40de-94fb-add95bd74c10-istio-envoy\") pod \"e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-csthf\" (UID: \"3c69ea47-20cb-40de-94fb-add95bd74c10\") " pod="openshift-ingress/e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-csthf" Nov 03 13:25:18.092311 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:18.091561 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-csthf" Nov 03 13:25:18.141309 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:18.128142 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="24b840c8-af1d-41da-8b95-b70f13c93125" path="/var/lib/kubelet/pods/24b840c8-af1d-41da-8b95-b70f13c93125/volumes" Nov 03 13:25:18.501096 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:18.500143 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-csthf"] Nov 03 13:25:18.505343 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:18.504353 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:25:18.505343 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:18.504443 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:25:18.505343 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:18.504488 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:25:18.684927 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:18.684897 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-021a5a5c-859d56dfc9-6prfw" Nov 03 13:25:18.864905 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:18.864877 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-1a18c8-d6c758bcc-pmrxw" Nov 03 13:25:19.400584 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:19.400547 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-csthf" event={"ID":"3c69ea47-20cb-40de-94fb-add95bd74c10","Type":"ContainerStarted","Data":"fb1b7bcd2ac51511eb46ebfa4e730d348cddc59d0a252d65cd58e6b9191402d5"} Nov 03 13:25:19.400784 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:19.400607 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-csthf" event={"ID":"3c69ea47-20cb-40de-94fb-add95bd74c10","Type":"ContainerStarted","Data":"6445b6239a63794829fb307c11feccbb7e2fba95503bd1e8e5f39dd65bfdc11c"} Nov 03 13:25:19.431783 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:19.431725 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-csthf" podStartSLOduration=2.431709309 podStartE2EDuration="2.431709309s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:25:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:25:19.428684433 +0000 UTC m=+999.436875711" watchObservedRunningTime="2026-11-03 13:25:19.431709309 +0000 UTC m=+999.439900586" Nov 03 13:25:19.816676 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:19.816576 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-021a5a5c-58589f959-92jwh" Nov 03 13:25:20.094645 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:20.093629 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-csthf" Nov 03 13:25:20.100547 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:20.100519 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-csthf" Nov 03 13:25:20.409327 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:20.409262 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-csthf" Nov 03 13:25:20.410227 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:20.410205 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-csthf" Nov 03 13:25:22.110826 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:22.110798 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-8444wtvfx"] Nov 03 13:25:22.114982 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:22.114959 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-8444wtvfx" Nov 03 13:25:22.121251 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:22.121223 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-8444wtvfx"] Nov 03 13:25:22.122845 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:22.122813 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-7c192b\"/\"routing-overlay\"" Nov 03 13:25:22.122845 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:22.122827 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-7c192b\"/\"payload-processing-external-model-plugins-e2e-extmod-b-7c192b\"" Nov 03 13:25:22.123028 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:22.122869 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-7c192b\"/\"openshift-service-ca.crt\"" Nov 03 13:25:22.123100 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:22.123081 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-7c192b\"/\"kube-root-ca.crt\"" Nov 03 13:25:22.136171 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:22.136136 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-7c192b\"/\"agc-cred-06e7699b6ab0\"" Nov 03 13:25:22.136609 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:22.136587 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-7c192b\"/\"payload-processing-external-model-e2e--cd1da41a-dockercfg-txc88\"" Nov 03 13:25:22.136714 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:22.136627 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-7c192b\"/\"agc-cred-384e44d183e3\"" Nov 03 13:25:22.136714 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:22.136650 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-7c192b\"/\"agc-cred-f5ffe280c5d9\"" Nov 03 13:25:22.136714 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:22.136672 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-7c192b\"/\"agc-cred-29dd648f7d07\"" Nov 03 13:25:22.199104 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:22.199061 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-thgrc\" (UniqueName: \"kubernetes.io/projected/ea6d1245-8491-4503-9a7d-c80b6b04d811-kube-api-access-thgrc\") pod \"payload-processing-external-model-e2e-extmod-b-7c192b-8444wtvfx\" (UID: \"ea6d1245-8491-4503-9a7d-c80b6b04d811\") " pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-8444wtvfx" Nov 03 13:25:22.199358 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:22.199126 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ea6d1245-8491-4503-9a7d-c80b6b04d811-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-7c192b-8444wtvfx\" (UID: \"ea6d1245-8491-4503-9a7d-c80b6b04d811\") " pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-8444wtvfx" Nov 03 13:25:22.199358 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:22.199214 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea6d1245-8491-4503-9a7d-c80b6b04d811-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-7c192b-8444wtvfx\" (UID: \"ea6d1245-8491-4503-9a7d-c80b6b04d811\") " pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-8444wtvfx" Nov 03 13:25:22.199358 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:22.199244 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ea6d1245-8491-4503-9a7d-c80b6b04d811-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-7c192b-8444wtvfx\" (UID: \"ea6d1245-8491-4503-9a7d-c80b6b04d811\") " pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-8444wtvfx" Nov 03 13:25:22.300885 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:22.300849 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-thgrc\" (UniqueName: \"kubernetes.io/projected/ea6d1245-8491-4503-9a7d-c80b6b04d811-kube-api-access-thgrc\") pod \"payload-processing-external-model-e2e-extmod-b-7c192b-8444wtvfx\" (UID: \"ea6d1245-8491-4503-9a7d-c80b6b04d811\") " pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-8444wtvfx" Nov 03 13:25:22.301082 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:22.300892 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ea6d1245-8491-4503-9a7d-c80b6b04d811-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-7c192b-8444wtvfx\" (UID: \"ea6d1245-8491-4503-9a7d-c80b6b04d811\") " pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-8444wtvfx" Nov 03 13:25:22.301082 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:22.300956 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea6d1245-8491-4503-9a7d-c80b6b04d811-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-7c192b-8444wtvfx\" (UID: \"ea6d1245-8491-4503-9a7d-c80b6b04d811\") " pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-8444wtvfx" Nov 03 13:25:22.301082 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:22.300973 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ea6d1245-8491-4503-9a7d-c80b6b04d811-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-7c192b-8444wtvfx\" (UID: \"ea6d1245-8491-4503-9a7d-c80b6b04d811\") " pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-8444wtvfx" Nov 03 13:25:22.301891 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:22.301858 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ea6d1245-8491-4503-9a7d-c80b6b04d811-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-7c192b-8444wtvfx\" (UID: \"ea6d1245-8491-4503-9a7d-c80b6b04d811\") " pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-8444wtvfx" Nov 03 13:25:22.302002 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:22.301934 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea6d1245-8491-4503-9a7d-c80b6b04d811-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-7c192b-8444wtvfx\" (UID: \"ea6d1245-8491-4503-9a7d-c80b6b04d811\") " pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-8444wtvfx" Nov 03 13:25:22.303918 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:22.303886 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ea6d1245-8491-4503-9a7d-c80b6b04d811-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-7c192b-8444wtvfx\" (UID: \"ea6d1245-8491-4503-9a7d-c80b6b04d811\") " pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-8444wtvfx" Nov 03 13:25:22.309946 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:22.309921 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-thgrc\" (UniqueName: \"kubernetes.io/projected/ea6d1245-8491-4503-9a7d-c80b6b04d811-kube-api-access-thgrc\") pod \"payload-processing-external-model-e2e-extmod-b-7c192b-8444wtvfx\" (UID: \"ea6d1245-8491-4503-9a7d-c80b6b04d811\") " pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-8444wtvfx" Nov 03 13:25:22.431959 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:22.431867 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-8444wtvfx" Nov 03 13:25:22.732142 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:22.732110 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-8444wtvfx"] Nov 03 13:25:22.734627 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:25:22.734554 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea6d1245_8491_4503_9a7d_c80b6b04d811.slice/crio-d2cdde7d85c6bcb167c8bbb1bcaea8dc7d78036e1b7e43f6b340902bda3b94b5 WatchSource:0}: Error finding container d2cdde7d85c6bcb167c8bbb1bcaea8dc7d78036e1b7e43f6b340902bda3b94b5: Status 404 returned error can't find the container with id d2cdde7d85c6bcb167c8bbb1bcaea8dc7d78036e1b7e43f6b340902bda3b94b5 Nov 03 13:25:23.552072 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:23.552028 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-8444wtvfx" event={"ID":"ea6d1245-8491-4503-9a7d-c80b6b04d811","Type":"ContainerStarted","Data":"1c94023325759c305773eaa6eda3b4d7483194c8b118b3253efc058b2666b548"} Nov 03 13:25:23.552677 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:23.552649 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-8444wtvfx" event={"ID":"ea6d1245-8491-4503-9a7d-c80b6b04d811","Type":"ContainerStarted","Data":"d2cdde7d85c6bcb167c8bbb1bcaea8dc7d78036e1b7e43f6b340902bda3b94b5"} Nov 03 13:25:23.553967 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:23.553872 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-8444wtvfx" Nov 03 13:25:27.869007 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:27.868929 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-8444wtvfx" podStartSLOduration=5.868909007 podStartE2EDuration="5.868909007s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:25:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:25:23.596492426 +0000 UTC m=+1003.604683703" watchObservedRunningTime="2026-11-03 13:25:27.868909007 +0000 UTC m=+1007.877100287" Nov 03 13:25:27.870938 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:27.870904 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-1a18c8/payload-processing-external-model-e2e-extmod-b-1a18c8-8545rjtw7"] Nov 03 13:25:27.887931 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:27.887891 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-1a18c8/payload-processing-external-model-e2e-extmod-b-1a18c8-8545rjtw7" Nov 03 13:25:27.900119 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:27.896711 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-1a18c8\"/\"openshift-service-ca.crt\"" Nov 03 13:25:27.900119 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:27.899121 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-1a18c8\"/\"payload-processing-external-model-plugins-e2e-extmod-b-1a18c8\"" Nov 03 13:25:27.900119 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:27.899555 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-1a18c8\"/\"routing-overlay\"" Nov 03 13:25:27.900119 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:27.899668 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-1a18c8\"/\"kube-root-ca.crt\"" Nov 03 13:25:27.900119 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:27.899741 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-1a18c8/payload-processing-external-model-e2e-extmod-b-1a18c8-8545rjtw7"] Nov 03 13:25:27.959983 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:27.959944 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-1a18c8\"/\"payload-processing-external-model-e2e--d832c86e-dockercfg-sq45c\"" Nov 03 13:25:27.961612 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:27.960868 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-1a18c8\"/\"agc-cred-de83083e156c\"" Nov 03 13:25:28.022307 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:28.001444 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6hqd\" (UniqueName: \"kubernetes.io/projected/b9a50676-42a1-411a-a8a4-ed1ae870c843-kube-api-access-h6hqd\") pod \"payload-processing-external-model-e2e-extmod-b-1a18c8-8545rjtw7\" (UID: \"b9a50676-42a1-411a-a8a4-ed1ae870c843\") " pod="ai-tenant-e2e-extmod-b-1a18c8/payload-processing-external-model-e2e-extmod-b-1a18c8-8545rjtw7" Nov 03 13:25:28.055509 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:28.055463 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b9a50676-42a1-411a-a8a4-ed1ae870c843-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-1a18c8-8545rjtw7\" (UID: \"b9a50676-42a1-411a-a8a4-ed1ae870c843\") " pod="ai-tenant-e2e-extmod-b-1a18c8/payload-processing-external-model-e2e-extmod-b-1a18c8-8545rjtw7" Nov 03 13:25:28.055821 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:28.055795 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9a50676-42a1-411a-a8a4-ed1ae870c843-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-1a18c8-8545rjtw7\" (UID: \"b9a50676-42a1-411a-a8a4-ed1ae870c843\") " pod="ai-tenant-e2e-extmod-b-1a18c8/payload-processing-external-model-e2e-extmod-b-1a18c8-8545rjtw7" Nov 03 13:25:28.068344 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:28.056077 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b9a50676-42a1-411a-a8a4-ed1ae870c843-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-1a18c8-8545rjtw7\" (UID: \"b9a50676-42a1-411a-a8a4-ed1ae870c843\") " pod="ai-tenant-e2e-extmod-b-1a18c8/payload-processing-external-model-e2e-extmod-b-1a18c8-8545rjtw7" Nov 03 13:25:28.188388 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:28.179260 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6hqd\" (UniqueName: \"kubernetes.io/projected/b9a50676-42a1-411a-a8a4-ed1ae870c843-kube-api-access-h6hqd\") pod \"payload-processing-external-model-e2e-extmod-b-1a18c8-8545rjtw7\" (UID: \"b9a50676-42a1-411a-a8a4-ed1ae870c843\") " pod="ai-tenant-e2e-extmod-b-1a18c8/payload-processing-external-model-e2e-extmod-b-1a18c8-8545rjtw7" Nov 03 13:25:28.188388 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:28.179451 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b9a50676-42a1-411a-a8a4-ed1ae870c843-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-1a18c8-8545rjtw7\" (UID: \"b9a50676-42a1-411a-a8a4-ed1ae870c843\") " pod="ai-tenant-e2e-extmod-b-1a18c8/payload-processing-external-model-e2e-extmod-b-1a18c8-8545rjtw7" Nov 03 13:25:28.188388 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:28.179520 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9a50676-42a1-411a-a8a4-ed1ae870c843-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-1a18c8-8545rjtw7\" (UID: \"b9a50676-42a1-411a-a8a4-ed1ae870c843\") " pod="ai-tenant-e2e-extmod-b-1a18c8/payload-processing-external-model-e2e-extmod-b-1a18c8-8545rjtw7" Nov 03 13:25:28.188388 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:28.179648 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b9a50676-42a1-411a-a8a4-ed1ae870c843-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-1a18c8-8545rjtw7\" (UID: \"b9a50676-42a1-411a-a8a4-ed1ae870c843\") " pod="ai-tenant-e2e-extmod-b-1a18c8/payload-processing-external-model-e2e-extmod-b-1a18c8-8545rjtw7" Nov 03 13:25:28.188388 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:28.180553 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b9a50676-42a1-411a-a8a4-ed1ae870c843-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-1a18c8-8545rjtw7\" (UID: \"b9a50676-42a1-411a-a8a4-ed1ae870c843\") " pod="ai-tenant-e2e-extmod-b-1a18c8/payload-processing-external-model-e2e-extmod-b-1a18c8-8545rjtw7" Nov 03 13:25:28.244310 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:28.241943 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9a50676-42a1-411a-a8a4-ed1ae870c843-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-1a18c8-8545rjtw7\" (UID: \"b9a50676-42a1-411a-a8a4-ed1ae870c843\") " pod="ai-tenant-e2e-extmod-b-1a18c8/payload-processing-external-model-e2e-extmod-b-1a18c8-8545rjtw7" Nov 03 13:25:28.306362 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:28.293401 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b9a50676-42a1-411a-a8a4-ed1ae870c843-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-1a18c8-8545rjtw7\" (UID: \"b9a50676-42a1-411a-a8a4-ed1ae870c843\") " pod="ai-tenant-e2e-extmod-b-1a18c8/payload-processing-external-model-e2e-extmod-b-1a18c8-8545rjtw7" Nov 03 13:25:28.321055 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:28.321022 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6hqd\" (UniqueName: \"kubernetes.io/projected/b9a50676-42a1-411a-a8a4-ed1ae870c843-kube-api-access-h6hqd\") pod \"payload-processing-external-model-e2e-extmod-b-1a18c8-8545rjtw7\" (UID: \"b9a50676-42a1-411a-a8a4-ed1ae870c843\") " pod="ai-tenant-e2e-extmod-b-1a18c8/payload-processing-external-model-e2e-extmod-b-1a18c8-8545rjtw7" Nov 03 13:25:28.382764 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:28.382728 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-1a18c8/payload-processing-external-model-e2e-extmod-b-1a18c8-8545rjtw7" Nov 03 13:25:29.417253 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:29.417208 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-1a18c8/payload-processing-external-model-e2e-extmod-b-1a18c8-8545rjtw7"] Nov 03 13:25:29.707638 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:29.707553 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-1a18c8/payload-processing-external-model-e2e-extmod-b-1a18c8-8545rjtw7" event={"ID":"b9a50676-42a1-411a-a8a4-ed1ae870c843","Type":"ContainerStarted","Data":"f53482dd3d87d1be46535b54162ef64aff8d8f6fd1b9dcd6ee2458d91ffe4ed7"} Nov 03 13:25:30.791385 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:30.781137 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-1a18c8/payload-processing-external-model-e2e-extmod-b-1a18c8-8545rjtw7" event={"ID":"b9a50676-42a1-411a-a8a4-ed1ae870c843","Type":"ContainerStarted","Data":"f7476c27e84e714bb39c4d4391a6d71317af1164fd26074aeaeb96559c480b22"} Nov 03 13:25:30.810344 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:30.809808 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-1a18c8/payload-processing-external-model-e2e-extmod-b-1a18c8-8545rjtw7" Nov 03 13:25:35.569608 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:35.569575 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-8444wtvfx" Nov 03 13:25:35.593945 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:35.593887 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-1a18c8/payload-processing-external-model-e2e-extmod-b-1a18c8-8545rjtw7" podStartSLOduration=8.59387171 podStartE2EDuration="8.59387171s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:25:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:25:30.854954109 +0000 UTC m=+1010.863145389" watchObservedRunningTime="2026-11-03 13:25:35.59387171 +0000 UTC m=+1015.602063042" Nov 03 13:25:36.156110 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:36.156066 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-021a5a5c-859d56dfc9-6prfw"] Nov 03 13:25:36.170761 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:36.170724 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-021a5a5c-58589f959-92jwh"] Nov 03 13:25:39.010697 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:39.010600 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-2dbf739c-gw-openshift-default-64b7d84c5c-kmkgg"] Nov 03 13:25:39.017411 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:39.017382 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-2dbf739c-gw-openshift-default-64b7d84c5c-kmkgg" Nov 03 13:25:39.029142 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:39.029109 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-2dbf739c-gw-openshift-default-64b7d84c5c-kmkgg"] Nov 03 13:25:39.030378 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:39.030351 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-2dbf739c-gw-openshift-default-dockercfg-r6859\"" Nov 03 13:25:39.082957 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:39.082914 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jzjgh\" (UniqueName: \"kubernetes.io/projected/84095934-31b5-4b42-9495-5ac5286722c2-kube-api-access-jzjgh\") pod \"e2e-migrate-2dbf739c-gw-openshift-default-64b7d84c5c-kmkgg\" (UID: \"84095934-31b5-4b42-9495-5ac5286722c2\") " pod="openshift-ingress/e2e-migrate-2dbf739c-gw-openshift-default-64b7d84c5c-kmkgg" Nov 03 13:25:39.083172 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:39.082971 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84095934-31b5-4b42-9495-5ac5286722c2-istio-envoy\") pod \"e2e-migrate-2dbf739c-gw-openshift-default-64b7d84c5c-kmkgg\" (UID: \"84095934-31b5-4b42-9495-5ac5286722c2\") " pod="openshift-ingress/e2e-migrate-2dbf739c-gw-openshift-default-64b7d84c5c-kmkgg" Nov 03 13:25:39.083172 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:39.083084 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84095934-31b5-4b42-9495-5ac5286722c2-istiod-ca-cert\") pod \"e2e-migrate-2dbf739c-gw-openshift-default-64b7d84c5c-kmkgg\" (UID: \"84095934-31b5-4b42-9495-5ac5286722c2\") " pod="openshift-ingress/e2e-migrate-2dbf739c-gw-openshift-default-64b7d84c5c-kmkgg" Nov 03 13:25:39.083315 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:39.083270 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84095934-31b5-4b42-9495-5ac5286722c2-workload-certs\") pod \"e2e-migrate-2dbf739c-gw-openshift-default-64b7d84c5c-kmkgg\" (UID: \"84095934-31b5-4b42-9495-5ac5286722c2\") " pod="openshift-ingress/e2e-migrate-2dbf739c-gw-openshift-default-64b7d84c5c-kmkgg" Nov 03 13:25:39.083460 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:39.083432 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84095934-31b5-4b42-9495-5ac5286722c2-credential-socket\") pod \"e2e-migrate-2dbf739c-gw-openshift-default-64b7d84c5c-kmkgg\" (UID: \"84095934-31b5-4b42-9495-5ac5286722c2\") " pod="openshift-ingress/e2e-migrate-2dbf739c-gw-openshift-default-64b7d84c5c-kmkgg" Nov 03 13:25:39.083505 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:39.083494 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84095934-31b5-4b42-9495-5ac5286722c2-istio-podinfo\") pod \"e2e-migrate-2dbf739c-gw-openshift-default-64b7d84c5c-kmkgg\" (UID: \"84095934-31b5-4b42-9495-5ac5286722c2\") " pod="openshift-ingress/e2e-migrate-2dbf739c-gw-openshift-default-64b7d84c5c-kmkgg" Nov 03 13:25:39.083583 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:39.083566 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84095934-31b5-4b42-9495-5ac5286722c2-istio-data\") pod \"e2e-migrate-2dbf739c-gw-openshift-default-64b7d84c5c-kmkgg\" (UID: \"84095934-31b5-4b42-9495-5ac5286722c2\") " pod="openshift-ingress/e2e-migrate-2dbf739c-gw-openshift-default-64b7d84c5c-kmkgg" Nov 03 13:25:39.083626 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:39.083613 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84095934-31b5-4b42-9495-5ac5286722c2-istio-token\") pod \"e2e-migrate-2dbf739c-gw-openshift-default-64b7d84c5c-kmkgg\" (UID: \"84095934-31b5-4b42-9495-5ac5286722c2\") " pod="openshift-ingress/e2e-migrate-2dbf739c-gw-openshift-default-64b7d84c5c-kmkgg" Nov 03 13:25:39.083662 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:39.083644 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84095934-31b5-4b42-9495-5ac5286722c2-workload-socket\") pod \"e2e-migrate-2dbf739c-gw-openshift-default-64b7d84c5c-kmkgg\" (UID: \"84095934-31b5-4b42-9495-5ac5286722c2\") " pod="openshift-ingress/e2e-migrate-2dbf739c-gw-openshift-default-64b7d84c5c-kmkgg" Nov 03 13:25:39.203916 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:39.196388 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84095934-31b5-4b42-9495-5ac5286722c2-workload-certs\") pod \"e2e-migrate-2dbf739c-gw-openshift-default-64b7d84c5c-kmkgg\" (UID: \"84095934-31b5-4b42-9495-5ac5286722c2\") " pod="openshift-ingress/e2e-migrate-2dbf739c-gw-openshift-default-64b7d84c5c-kmkgg" Nov 03 13:25:39.203916 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:39.196510 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84095934-31b5-4b42-9495-5ac5286722c2-credential-socket\") pod \"e2e-migrate-2dbf739c-gw-openshift-default-64b7d84c5c-kmkgg\" (UID: \"84095934-31b5-4b42-9495-5ac5286722c2\") " pod="openshift-ingress/e2e-migrate-2dbf739c-gw-openshift-default-64b7d84c5c-kmkgg" Nov 03 13:25:39.203916 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:39.196559 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84095934-31b5-4b42-9495-5ac5286722c2-istio-podinfo\") pod \"e2e-migrate-2dbf739c-gw-openshift-default-64b7d84c5c-kmkgg\" (UID: \"84095934-31b5-4b42-9495-5ac5286722c2\") " pod="openshift-ingress/e2e-migrate-2dbf739c-gw-openshift-default-64b7d84c5c-kmkgg" Nov 03 13:25:39.203916 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:39.196635 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84095934-31b5-4b42-9495-5ac5286722c2-istio-data\") pod \"e2e-migrate-2dbf739c-gw-openshift-default-64b7d84c5c-kmkgg\" (UID: \"84095934-31b5-4b42-9495-5ac5286722c2\") " pod="openshift-ingress/e2e-migrate-2dbf739c-gw-openshift-default-64b7d84c5c-kmkgg" Nov 03 13:25:39.203916 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:39.196677 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84095934-31b5-4b42-9495-5ac5286722c2-istio-token\") pod \"e2e-migrate-2dbf739c-gw-openshift-default-64b7d84c5c-kmkgg\" (UID: \"84095934-31b5-4b42-9495-5ac5286722c2\") " pod="openshift-ingress/e2e-migrate-2dbf739c-gw-openshift-default-64b7d84c5c-kmkgg" Nov 03 13:25:39.203916 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:39.196707 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84095934-31b5-4b42-9495-5ac5286722c2-workload-socket\") pod \"e2e-migrate-2dbf739c-gw-openshift-default-64b7d84c5c-kmkgg\" (UID: \"84095934-31b5-4b42-9495-5ac5286722c2\") " pod="openshift-ingress/e2e-migrate-2dbf739c-gw-openshift-default-64b7d84c5c-kmkgg" Nov 03 13:25:39.203916 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:39.196761 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzjgh\" (UniqueName: \"kubernetes.io/projected/84095934-31b5-4b42-9495-5ac5286722c2-kube-api-access-jzjgh\") pod \"e2e-migrate-2dbf739c-gw-openshift-default-64b7d84c5c-kmkgg\" (UID: \"84095934-31b5-4b42-9495-5ac5286722c2\") " pod="openshift-ingress/e2e-migrate-2dbf739c-gw-openshift-default-64b7d84c5c-kmkgg" Nov 03 13:25:39.203916 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:39.196794 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84095934-31b5-4b42-9495-5ac5286722c2-istio-envoy\") pod \"e2e-migrate-2dbf739c-gw-openshift-default-64b7d84c5c-kmkgg\" (UID: \"84095934-31b5-4b42-9495-5ac5286722c2\") " pod="openshift-ingress/e2e-migrate-2dbf739c-gw-openshift-default-64b7d84c5c-kmkgg" Nov 03 13:25:39.203916 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:39.196879 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84095934-31b5-4b42-9495-5ac5286722c2-istiod-ca-cert\") pod \"e2e-migrate-2dbf739c-gw-openshift-default-64b7d84c5c-kmkgg\" (UID: \"84095934-31b5-4b42-9495-5ac5286722c2\") " pod="openshift-ingress/e2e-migrate-2dbf739c-gw-openshift-default-64b7d84c5c-kmkgg" Nov 03 13:25:39.203916 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:39.197913 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84095934-31b5-4b42-9495-5ac5286722c2-istiod-ca-cert\") pod \"e2e-migrate-2dbf739c-gw-openshift-default-64b7d84c5c-kmkgg\" (UID: \"84095934-31b5-4b42-9495-5ac5286722c2\") " pod="openshift-ingress/e2e-migrate-2dbf739c-gw-openshift-default-64b7d84c5c-kmkgg" Nov 03 13:25:39.203916 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:39.198271 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84095934-31b5-4b42-9495-5ac5286722c2-workload-certs\") pod \"e2e-migrate-2dbf739c-gw-openshift-default-64b7d84c5c-kmkgg\" (UID: \"84095934-31b5-4b42-9495-5ac5286722c2\") " pod="openshift-ingress/e2e-migrate-2dbf739c-gw-openshift-default-64b7d84c5c-kmkgg" Nov 03 13:25:39.203916 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:39.198564 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84095934-31b5-4b42-9495-5ac5286722c2-credential-socket\") pod \"e2e-migrate-2dbf739c-gw-openshift-default-64b7d84c5c-kmkgg\" (UID: \"84095934-31b5-4b42-9495-5ac5286722c2\") " pod="openshift-ingress/e2e-migrate-2dbf739c-gw-openshift-default-64b7d84c5c-kmkgg" Nov 03 13:25:39.233332 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:39.217910 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84095934-31b5-4b42-9495-5ac5286722c2-istio-podinfo\") pod \"e2e-migrate-2dbf739c-gw-openshift-default-64b7d84c5c-kmkgg\" (UID: \"84095934-31b5-4b42-9495-5ac5286722c2\") " pod="openshift-ingress/e2e-migrate-2dbf739c-gw-openshift-default-64b7d84c5c-kmkgg" Nov 03 13:25:39.233332 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:39.218232 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84095934-31b5-4b42-9495-5ac5286722c2-istio-data\") pod \"e2e-migrate-2dbf739c-gw-openshift-default-64b7d84c5c-kmkgg\" (UID: \"84095934-31b5-4b42-9495-5ac5286722c2\") " pod="openshift-ingress/e2e-migrate-2dbf739c-gw-openshift-default-64b7d84c5c-kmkgg" Nov 03 13:25:39.233332 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:39.226139 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84095934-31b5-4b42-9495-5ac5286722c2-workload-socket\") pod \"e2e-migrate-2dbf739c-gw-openshift-default-64b7d84c5c-kmkgg\" (UID: \"84095934-31b5-4b42-9495-5ac5286722c2\") " pod="openshift-ingress/e2e-migrate-2dbf739c-gw-openshift-default-64b7d84c5c-kmkgg" Nov 03 13:25:39.360507 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:39.350125 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84095934-31b5-4b42-9495-5ac5286722c2-istio-token\") pod \"e2e-migrate-2dbf739c-gw-openshift-default-64b7d84c5c-kmkgg\" (UID: \"84095934-31b5-4b42-9495-5ac5286722c2\") " pod="openshift-ingress/e2e-migrate-2dbf739c-gw-openshift-default-64b7d84c5c-kmkgg" Nov 03 13:25:39.405309 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:39.376807 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84095934-31b5-4b42-9495-5ac5286722c2-istio-envoy\") pod \"e2e-migrate-2dbf739c-gw-openshift-default-64b7d84c5c-kmkgg\" (UID: \"84095934-31b5-4b42-9495-5ac5286722c2\") " pod="openshift-ingress/e2e-migrate-2dbf739c-gw-openshift-default-64b7d84c5c-kmkgg" Nov 03 13:25:39.405309 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:39.385146 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jzjgh\" (UniqueName: \"kubernetes.io/projected/84095934-31b5-4b42-9495-5ac5286722c2-kube-api-access-jzjgh\") pod \"e2e-migrate-2dbf739c-gw-openshift-default-64b7d84c5c-kmkgg\" (UID: \"84095934-31b5-4b42-9495-5ac5286722c2\") " pod="openshift-ingress/e2e-migrate-2dbf739c-gw-openshift-default-64b7d84c5c-kmkgg" Nov 03 13:25:39.516743 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:25:39.516703 5024 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d7mwz" containerName="extractor" Nov 03 13:25:39.638663 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:39.638571 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-2dbf739c-gw-openshift-default-64b7d84c5c-kmkgg" Nov 03 13:25:39.934048 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:25:39.934010 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84095934_31b5_4b42_9495_5ac5286722c2.slice/crio-b08f55129f58bfb11be7eacbf0cbb45498626a928a825655663ab41b625c7961 WatchSource:0}: Error finding container b08f55129f58bfb11be7eacbf0cbb45498626a928a825655663ab41b625c7961: Status 404 returned error can't find the container with id b08f55129f58bfb11be7eacbf0cbb45498626a928a825655663ab41b625c7961 Nov 03 13:25:39.934201 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:39.934089 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-2dbf739c-gw-openshift-default-64b7d84c5c-kmkgg"] Nov 03 13:25:39.936908 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:39.936874 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:25:39.937021 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:39.936953 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:25:39.937021 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:39.936999 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:25:39.976922 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:39.976884 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-2dbf739c-gw-openshift-default-64b7d84c5c-kmkgg" event={"ID":"84095934-31b5-4b42-9495-5ac5286722c2","Type":"ContainerStarted","Data":"b08f55129f58bfb11be7eacbf0cbb45498626a928a825655663ab41b625c7961"} Nov 03 13:25:40.795642 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:40.795607 5024 scope.go:122] "RemoveContainer" containerID="6ab9d2c6cbd7023205c3c125b210bde795226750662dd659a173d9d75c5fdc6f" Nov 03 13:25:40.986766 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:40.986732 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-2dbf739c-gw-openshift-default-64b7d84c5c-kmkgg" event={"ID":"84095934-31b5-4b42-9495-5ac5286722c2","Type":"ContainerStarted","Data":"ce2914a031260070d7ab30eb79a765584ed94b6e54403d18756d3fc2cf53f2b0"} Nov 03 13:25:41.162062 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:41.157371 5024 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-021a5a5c-859d56dfc9-6prfw" podUID="ab04f246-fadf-4901-b65b-523bfdaf7e0a" containerName="payload-processing" containerID="cri-o://b5617245bf97485772fed59b74bc26efca91f0cc2f85d1ff548a0ffd40f276a5" gracePeriod=25 Nov 03 13:25:41.178433 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:41.178376 5024 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-021a5a5c-58589f959-92jwh" podUID="d4f6e2be-e468-4cf1-b7a3-ae8f9ee5e61b" containerName="payload-pre-processing" containerID="cri-o://910f5ef44662c553afee2c510a36c4ced1cdc8b171ad5c129c73de0ebc6269c6" gracePeriod=25 Nov 03 13:25:41.584627 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:41.584594 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-021a5a5c-859d56dfc9-6prfw" Nov 03 13:25:41.615758 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:41.615060 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-2dbf739c-gw-openshift-default-64b7d84c5c-kmkgg" podStartSLOduration=3.6150406029999997 podStartE2EDuration="3.615040603s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:25:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:25:41.01380088 +0000 UTC m=+1021.021992158" watchObservedRunningTime="2026-11-03 13:25:41.615040603 +0000 UTC m=+1021.623231881" Nov 03 13:25:41.638718 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:41.637300 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ab04f246-fadf-4901-b65b-523bfdaf7e0a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab04f246-fadf-4901-b65b-523bfdaf7e0a-plugins-config-volume\") pod \"ab04f246-fadf-4901-b65b-523bfdaf7e0a\" (UID: \"ab04f246-fadf-4901-b65b-523bfdaf7e0a\") " Nov 03 13:25:41.638718 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:41.637370 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab04f246-fadf-4901-b65b-523bfdaf7e0a-kube-api-access-rm5t9\" (UniqueName: \"kubernetes.io/projected/ab04f246-fadf-4901-b65b-523bfdaf7e0a-kube-api-access-rm5t9\") pod \"ab04f246-fadf-4901-b65b-523bfdaf7e0a\" (UID: \"ab04f246-fadf-4901-b65b-523bfdaf7e0a\") " Nov 03 13:25:41.638718 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:41.638239 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab04f246-fadf-4901-b65b-523bfdaf7e0a-plugins-config-volume" pod "ab04f246-fadf-4901-b65b-523bfdaf7e0a" (UID: "ab04f246-fadf-4901-b65b-523bfdaf7e0a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:25:41.640593 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:41.640558 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-2dbf739c-gw-openshift-default-64b7d84c5c-kmkgg" Nov 03 13:25:41.643072 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:41.642899 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab04f246-fadf-4901-b65b-523bfdaf7e0a-kube-api-access-rm5t9" pod "ab04f246-fadf-4901-b65b-523bfdaf7e0a" (UID: "ab04f246-fadf-4901-b65b-523bfdaf7e0a"). InnerVolumeSpecName "kube-api-access-rm5t9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:25:41.647877 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:41.647853 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-2dbf739c-gw-openshift-default-64b7d84c5c-kmkgg" Nov 03 13:25:41.658234 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:41.658201 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-021a5a5c-58589f959-92jwh" Nov 03 13:25:41.739087 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:41.739045 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d4f6e2be-e468-4cf1-b7a3-ae8f9ee5e61b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4f6e2be-e468-4cf1-b7a3-ae8f9ee5e61b-plugins-config-volume\") pod \"d4f6e2be-e468-4cf1-b7a3-ae8f9ee5e61b\" (UID: \"d4f6e2be-e468-4cf1-b7a3-ae8f9ee5e61b\") " Nov 03 13:25:41.739349 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:41.739325 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4f6e2be-e468-4cf1-b7a3-ae8f9ee5e61b-kube-api-access-czprz\" (UniqueName: \"kubernetes.io/projected/d4f6e2be-e468-4cf1-b7a3-ae8f9ee5e61b-kube-api-access-czprz\") pod \"d4f6e2be-e468-4cf1-b7a3-ae8f9ee5e61b\" (UID: \"d4f6e2be-e468-4cf1-b7a3-ae8f9ee5e61b\") " Nov 03 13:25:41.739485 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:41.739457 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4f6e2be-e468-4cf1-b7a3-ae8f9ee5e61b-plugins-config-volume" pod "d4f6e2be-e468-4cf1-b7a3-ae8f9ee5e61b" (UID: "d4f6e2be-e468-4cf1-b7a3-ae8f9ee5e61b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:25:41.739982 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:41.739956 5024 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab04f246-fadf-4901-b65b-523bfdaf7e0a-plugins-config-volume\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:25:41.740088 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:41.739986 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rm5t9\" (UniqueName: \"kubernetes.io/projected/ab04f246-fadf-4901-b65b-523bfdaf7e0a-kube-api-access-rm5t9\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:25:41.740088 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:41.740001 5024 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4f6e2be-e468-4cf1-b7a3-ae8f9ee5e61b-plugins-config-volume\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:25:41.741454 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:41.741432 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4f6e2be-e468-4cf1-b7a3-ae8f9ee5e61b-kube-api-access-czprz" pod "d4f6e2be-e468-4cf1-b7a3-ae8f9ee5e61b" (UID: "d4f6e2be-e468-4cf1-b7a3-ae8f9ee5e61b"). InnerVolumeSpecName "kube-api-access-czprz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:25:41.841887 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:41.841792 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-czprz\" (UniqueName: \"kubernetes.io/projected/d4f6e2be-e468-4cf1-b7a3-ae8f9ee5e61b-kube-api-access-czprz\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:25:41.995942 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:41.995907 5024 generic.go:356] "Generic (PLEG): container finished" podID="168564ee-9452-4aba-8391-d4f0bdaafa7b" containerID="954b253b9e11610a4a8efa783f173c6d9a344093f76f73815f5b175465317dcf" exitCode=0 Nov 03 13:25:41.996143 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:41.995978 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-1a18c8-openshift-default-67d7b5995c-z4csj" event={"ID":"168564ee-9452-4aba-8391-d4f0bdaafa7b","Type":"ContainerDied","Data":"954b253b9e11610a4a8efa783f173c6d9a344093f76f73815f5b175465317dcf"} Nov 03 13:25:41.996655 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:41.996542 5024 scope.go:122] "RemoveContainer" containerID="954b253b9e11610a4a8efa783f173c6d9a344093f76f73815f5b175465317dcf" Nov 03 13:25:41.997546 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:41.997458 5024 generic.go:356] "Generic (PLEG): container finished" podID="ab04f246-fadf-4901-b65b-523bfdaf7e0a" containerID="b5617245bf97485772fed59b74bc26efca91f0cc2f85d1ff548a0ffd40f276a5" exitCode=0 Nov 03 13:25:41.997546 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:41.997485 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-021a5a5c-859d56dfc9-6prfw" event={"ID":"ab04f246-fadf-4901-b65b-523bfdaf7e0a","Type":"ContainerDied","Data":"b5617245bf97485772fed59b74bc26efca91f0cc2f85d1ff548a0ffd40f276a5"} Nov 03 13:25:41.997546 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:41.997507 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-021a5a5c-859d56dfc9-6prfw" Nov 03 13:25:41.997758 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:41.997552 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-021a5a5c-859d56dfc9-6prfw" event={"ID":"ab04f246-fadf-4901-b65b-523bfdaf7e0a","Type":"ContainerDied","Data":"3ecc5f3e0d8d55d63025948592ed21882f5db349622f998b173f57ae124c0fab"} Nov 03 13:25:41.997758 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:41.997582 5024 scope.go:122] "RemoveContainer" containerID="b5617245bf97485772fed59b74bc26efca91f0cc2f85d1ff548a0ffd40f276a5" Nov 03 13:25:41.998438 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:41.998407 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:25:41.998525 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:41.998504 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:25:41.998578 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:41.998560 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:25:41.999119 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:41.999070 5024 generic.go:356] "Generic (PLEG): container finished" podID="d4f6e2be-e468-4cf1-b7a3-ae8f9ee5e61b" containerID="910f5ef44662c553afee2c510a36c4ced1cdc8b171ad5c129c73de0ebc6269c6" exitCode=0 Nov 03 13:25:41.999211 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:41.999190 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-021a5a5c-58589f959-92jwh" event={"ID":"d4f6e2be-e468-4cf1-b7a3-ae8f9ee5e61b","Type":"ContainerDied","Data":"910f5ef44662c553afee2c510a36c4ced1cdc8b171ad5c129c73de0ebc6269c6"} Nov 03 13:25:41.999274 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:41.999236 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-021a5a5c-58589f959-92jwh" event={"ID":"d4f6e2be-e468-4cf1-b7a3-ae8f9ee5e61b","Type":"ContainerDied","Data":"af093d25e7e9781f05025d56f93c0dbc4b481cbab45e8a2e1216a382f68b72d8"} Nov 03 13:25:41.999274 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:41.999260 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-021a5a5c-58589f959-92jwh" Nov 03 13:25:42.001497 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:42.001476 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-2dbf739c-gw-openshift-default-64b7d84c5c-kmkgg" Nov 03 13:25:42.002801 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:42.002780 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-2dbf739c-gw-openshift-default-64b7d84c5c-kmkgg" Nov 03 13:25:42.009561 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:42.009542 5024 scope.go:122] "RemoveContainer" containerID="b5617245bf97485772fed59b74bc26efca91f0cc2f85d1ff548a0ffd40f276a5" Nov 03 13:25:42.009894 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:25:42.009868 5024 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5617245bf97485772fed59b74bc26efca91f0cc2f85d1ff548a0ffd40f276a5\": container with ID starting with b5617245bf97485772fed59b74bc26efca91f0cc2f85d1ff548a0ffd40f276a5 not found: ID does not exist" containerID="b5617245bf97485772fed59b74bc26efca91f0cc2f85d1ff548a0ffd40f276a5" Nov 03 13:25:42.009984 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:42.009907 5024 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5617245bf97485772fed59b74bc26efca91f0cc2f85d1ff548a0ffd40f276a5"} err="failed to get container status \"b5617245bf97485772fed59b74bc26efca91f0cc2f85d1ff548a0ffd40f276a5\": rpc error: code = NotFound desc = could not find container \"b5617245bf97485772fed59b74bc26efca91f0cc2f85d1ff548a0ffd40f276a5\": container with ID starting with b5617245bf97485772fed59b74bc26efca91f0cc2f85d1ff548a0ffd40f276a5 not found: ID does not exist" Nov 03 13:25:42.009984 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:42.009934 5024 scope.go:122] "RemoveContainer" containerID="910f5ef44662c553afee2c510a36c4ced1cdc8b171ad5c129c73de0ebc6269c6" Nov 03 13:25:42.023630 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:42.023505 5024 scope.go:122] "RemoveContainer" containerID="910f5ef44662c553afee2c510a36c4ced1cdc8b171ad5c129c73de0ebc6269c6" Nov 03 13:25:42.024204 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:25:42.024128 5024 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"910f5ef44662c553afee2c510a36c4ced1cdc8b171ad5c129c73de0ebc6269c6\": container with ID starting with 910f5ef44662c553afee2c510a36c4ced1cdc8b171ad5c129c73de0ebc6269c6 not found: ID does not exist" containerID="910f5ef44662c553afee2c510a36c4ced1cdc8b171ad5c129c73de0ebc6269c6" Nov 03 13:25:42.024204 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:42.024177 5024 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"910f5ef44662c553afee2c510a36c4ced1cdc8b171ad5c129c73de0ebc6269c6"} err="failed to get container status \"910f5ef44662c553afee2c510a36c4ced1cdc8b171ad5c129c73de0ebc6269c6\": rpc error: code = NotFound desc = could not find container \"910f5ef44662c553afee2c510a36c4ced1cdc8b171ad5c129c73de0ebc6269c6\": container with ID starting with 910f5ef44662c553afee2c510a36c4ced1cdc8b171ad5c129c73de0ebc6269c6 not found: ID does not exist" Nov 03 13:25:42.047721 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:42.046527 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-021a5a5c-859d56dfc9-6prfw"] Nov 03 13:25:42.054790 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:42.054706 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-021a5a5c-859d56dfc9-6prfw"] Nov 03 13:25:42.124083 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:42.123953 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab04f246-fadf-4901-b65b-523bfdaf7e0a" path="/var/lib/kubelet/pods/ab04f246-fadf-4901-b65b-523bfdaf7e0a/volumes" Nov 03 13:25:42.124795 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:42.124752 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-021a5a5c-58589f959-92jwh"] Nov 03 13:25:42.128532 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:42.128497 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-021a5a5c-58589f959-92jwh"] Nov 03 13:25:42.832124 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:42.832092 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-1a18c8/payload-processing-external-model-e2e-extmod-b-1a18c8-8545rjtw7" Nov 03 13:25:43.178379 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:43.154340 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-6z9tv"] Nov 03 13:25:43.178379 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:43.155341 5024 cpu_manager.go:414] "containerMap: removing container" podUID="ab04f246-fadf-4901-b65b-523bfdaf7e0a" containerName="payload-processing" Nov 03 13:25:43.178379 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:43.155364 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab04f246-fadf-4901-b65b-523bfdaf7e0a" containerName="payload-processing" Nov 03 13:25:43.178379 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:43.155419 5024 cpu_manager.go:414] "containerMap: removing container" podUID="d4f6e2be-e468-4cf1-b7a3-ae8f9ee5e61b" containerName="payload-pre-processing" Nov 03 13:25:43.178379 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:43.155429 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4f6e2be-e468-4cf1-b7a3-ae8f9ee5e61b" containerName="payload-pre-processing" Nov 03 13:25:43.178379 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:43.155629 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4f6e2be-e468-4cf1-b7a3-ae8f9ee5e61b" containerName="payload-pre-processing" Nov 03 13:25:43.178379 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:43.155648 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab04f246-fadf-4901-b65b-523bfdaf7e0a" containerName="payload-processing" Nov 03 13:25:43.252051 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:43.251979 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-6z9tv" Nov 03 13:25:43.268351 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:43.265978 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w0-9add72-openshift-default-dockercfg-c6nrt\"" Nov 03 13:25:43.337405 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:43.284520 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d77cd17-8ce7-4fe7-9b61-7e4c7cdc720b-istio-data\") pod \"e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-6z9tv\" (UID: \"7d77cd17-8ce7-4fe7-9b61-7e4c7cdc720b\") " pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-6z9tv" Nov 03 13:25:43.337405 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:43.284588 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d77cd17-8ce7-4fe7-9b61-7e4c7cdc720b-istio-envoy\") pod \"e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-6z9tv\" (UID: \"7d77cd17-8ce7-4fe7-9b61-7e4c7cdc720b\") " pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-6z9tv" Nov 03 13:25:43.337405 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:43.284640 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d77cd17-8ce7-4fe7-9b61-7e4c7cdc720b-istiod-ca-cert\") pod \"e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-6z9tv\" (UID: \"7d77cd17-8ce7-4fe7-9b61-7e4c7cdc720b\") " pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-6z9tv" Nov 03 13:25:43.337405 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:43.284678 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d77cd17-8ce7-4fe7-9b61-7e4c7cdc720b-workload-certs\") pod \"e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-6z9tv\" (UID: \"7d77cd17-8ce7-4fe7-9b61-7e4c7cdc720b\") " pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-6z9tv" Nov 03 13:25:43.337405 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:43.284712 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d77cd17-8ce7-4fe7-9b61-7e4c7cdc720b-credential-socket\") pod \"e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-6z9tv\" (UID: \"7d77cd17-8ce7-4fe7-9b61-7e4c7cdc720b\") " pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-6z9tv" Nov 03 13:25:43.337405 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:43.284781 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d77cd17-8ce7-4fe7-9b61-7e4c7cdc720b-workload-socket\") pod \"e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-6z9tv\" (UID: \"7d77cd17-8ce7-4fe7-9b61-7e4c7cdc720b\") " pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-6z9tv" Nov 03 13:25:43.337405 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:43.284866 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s9vt9\" (UniqueName: \"kubernetes.io/projected/7d77cd17-8ce7-4fe7-9b61-7e4c7cdc720b-kube-api-access-s9vt9\") pod \"e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-6z9tv\" (UID: \"7d77cd17-8ce7-4fe7-9b61-7e4c7cdc720b\") " pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-6z9tv" Nov 03 13:25:43.337405 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:43.285000 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d77cd17-8ce7-4fe7-9b61-7e4c7cdc720b-istio-token\") pod \"e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-6z9tv\" (UID: \"7d77cd17-8ce7-4fe7-9b61-7e4c7cdc720b\") " pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-6z9tv" Nov 03 13:25:43.337405 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:43.314355 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d77cd17-8ce7-4fe7-9b61-7e4c7cdc720b-istio-podinfo\") pod \"e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-6z9tv\" (UID: \"7d77cd17-8ce7-4fe7-9b61-7e4c7cdc720b\") " pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-6z9tv" Nov 03 13:25:43.356524 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:43.355470 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-6z9tv"] Nov 03 13:25:43.427307 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:43.423962 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d77cd17-8ce7-4fe7-9b61-7e4c7cdc720b-istio-data\") pod \"e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-6z9tv\" (UID: \"7d77cd17-8ce7-4fe7-9b61-7e4c7cdc720b\") " pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-6z9tv" Nov 03 13:25:43.427307 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:43.424016 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d77cd17-8ce7-4fe7-9b61-7e4c7cdc720b-istio-envoy\") pod \"e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-6z9tv\" (UID: \"7d77cd17-8ce7-4fe7-9b61-7e4c7cdc720b\") " pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-6z9tv" Nov 03 13:25:43.427307 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:43.424061 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d77cd17-8ce7-4fe7-9b61-7e4c7cdc720b-istiod-ca-cert\") pod \"e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-6z9tv\" (UID: \"7d77cd17-8ce7-4fe7-9b61-7e4c7cdc720b\") " pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-6z9tv" Nov 03 13:25:43.427307 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:43.424091 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d77cd17-8ce7-4fe7-9b61-7e4c7cdc720b-workload-certs\") pod \"e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-6z9tv\" (UID: \"7d77cd17-8ce7-4fe7-9b61-7e4c7cdc720b\") " pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-6z9tv" Nov 03 13:25:43.427307 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:43.424124 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d77cd17-8ce7-4fe7-9b61-7e4c7cdc720b-credential-socket\") pod \"e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-6z9tv\" (UID: \"7d77cd17-8ce7-4fe7-9b61-7e4c7cdc720b\") " pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-6z9tv" Nov 03 13:25:43.427307 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:43.424163 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d77cd17-8ce7-4fe7-9b61-7e4c7cdc720b-workload-socket\") pod \"e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-6z9tv\" (UID: \"7d77cd17-8ce7-4fe7-9b61-7e4c7cdc720b\") " pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-6z9tv" Nov 03 13:25:43.427307 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:43.424227 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9vt9\" (UniqueName: \"kubernetes.io/projected/7d77cd17-8ce7-4fe7-9b61-7e4c7cdc720b-kube-api-access-s9vt9\") pod \"e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-6z9tv\" (UID: \"7d77cd17-8ce7-4fe7-9b61-7e4c7cdc720b\") " pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-6z9tv" Nov 03 13:25:43.460420 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:43.432086 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d77cd17-8ce7-4fe7-9b61-7e4c7cdc720b-istiod-ca-cert\") pod \"e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-6z9tv\" (UID: \"7d77cd17-8ce7-4fe7-9b61-7e4c7cdc720b\") " pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-6z9tv" Nov 03 13:25:43.460420 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:43.445746 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d77cd17-8ce7-4fe7-9b61-7e4c7cdc720b-istio-token\") pod \"e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-6z9tv\" (UID: \"7d77cd17-8ce7-4fe7-9b61-7e4c7cdc720b\") " pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-6z9tv" Nov 03 13:25:43.460420 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:43.445798 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d77cd17-8ce7-4fe7-9b61-7e4c7cdc720b-istio-podinfo\") pod \"e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-6z9tv\" (UID: \"7d77cd17-8ce7-4fe7-9b61-7e4c7cdc720b\") " pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-6z9tv" Nov 03 13:25:43.460707 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:43.460630 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d77cd17-8ce7-4fe7-9b61-7e4c7cdc720b-workload-certs\") pod \"e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-6z9tv\" (UID: \"7d77cd17-8ce7-4fe7-9b61-7e4c7cdc720b\") " pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-6z9tv" Nov 03 13:25:43.513777 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:43.490083 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-1a18c8-openshift-default-67d7b5995c-z4csj" event={"ID":"168564ee-9452-4aba-8391-d4f0bdaafa7b","Type":"ContainerStarted","Data":"538c794a1ce0474466bbbc53c119dac920c1451c9b9ab21f523c5f9e02e4a4fa"} Nov 03 13:25:43.522600 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:43.508587 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d77cd17-8ce7-4fe7-9b61-7e4c7cdc720b-credential-socket\") pod \"e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-6z9tv\" (UID: \"7d77cd17-8ce7-4fe7-9b61-7e4c7cdc720b\") " pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-6z9tv" Nov 03 13:25:43.529501 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:43.524008 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d77cd17-8ce7-4fe7-9b61-7e4c7cdc720b-workload-socket\") pod \"e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-6z9tv\" (UID: \"7d77cd17-8ce7-4fe7-9b61-7e4c7cdc720b\") " pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-6z9tv" Nov 03 13:25:43.529501 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:43.524250 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d77cd17-8ce7-4fe7-9b61-7e4c7cdc720b-istio-data\") pod \"e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-6z9tv\" (UID: \"7d77cd17-8ce7-4fe7-9b61-7e4c7cdc720b\") " pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-6z9tv" Nov 03 13:25:43.560238 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:43.544931 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d77cd17-8ce7-4fe7-9b61-7e4c7cdc720b-istio-token\") pod \"e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-6z9tv\" (UID: \"7d77cd17-8ce7-4fe7-9b61-7e4c7cdc720b\") " pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-6z9tv" Nov 03 13:25:43.560238 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:43.559034 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d77cd17-8ce7-4fe7-9b61-7e4c7cdc720b-istio-podinfo\") pod \"e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-6z9tv\" (UID: \"7d77cd17-8ce7-4fe7-9b61-7e4c7cdc720b\") " pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-6z9tv" Nov 03 13:25:43.560238 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:43.560177 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s9vt9\" (UniqueName: \"kubernetes.io/projected/7d77cd17-8ce7-4fe7-9b61-7e4c7cdc720b-kube-api-access-s9vt9\") pod \"e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-6z9tv\" (UID: \"7d77cd17-8ce7-4fe7-9b61-7e4c7cdc720b\") " pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-6z9tv" Nov 03 13:25:43.561980 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:43.561883 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d77cd17-8ce7-4fe7-9b61-7e4c7cdc720b-istio-envoy\") pod \"e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-6z9tv\" (UID: \"7d77cd17-8ce7-4fe7-9b61-7e4c7cdc720b\") " pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-6z9tv" Nov 03 13:25:43.709991 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:43.707560 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-6z9tv" Nov 03 13:25:44.119270 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:44.119227 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d4f6e2be-e468-4cf1-b7a3-ae8f9ee5e61b" path="/var/lib/kubelet/pods/d4f6e2be-e468-4cf1-b7a3-ae8f9ee5e61b/volumes" Nov 03 13:25:44.119673 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:44.119652 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-2dbf739c-75f684fcd9-zw8cw"] Nov 03 13:25:44.125679 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:44.125647 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-2dbf739c-75f684fcd9-zw8cw" Nov 03 13:25:44.126706 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:44.126649 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-2dbf739c-75f684fcd9-zw8cw"] Nov 03 13:25:44.133992 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:44.133595 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-2dbf739c-dockercfg-jg9xr\"" Nov 03 13:25:44.136506 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:44.135524 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-2dbf739c\"" Nov 03 13:25:44.210722 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:44.210692 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-6z9tv"] Nov 03 13:25:44.212956 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:25:44.212921 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d77cd17_8ce7_4fe7_9b61_7e4c7cdc720b.slice/crio-d4ee62a1b39081f06a523f86ba39c8fe9bce3b81b579f7ff80e4feb80c9d07e5 WatchSource:0}: Error finding container d4ee62a1b39081f06a523f86ba39c8fe9bce3b81b579f7ff80e4feb80c9d07e5: Status 404 returned error can't find the container with id d4ee62a1b39081f06a523f86ba39c8fe9bce3b81b579f7ff80e4feb80c9d07e5 Nov 03 13:25:44.216860 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:44.216825 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:25:44.216963 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:44.216895 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:25:44.216963 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:44.216924 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:25:44.230537 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:44.230488 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-1a18c8-openshift-default-67d7b5995c-z4csj" Nov 03 13:25:44.231500 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:44.231477 5024 patch_prober.go:29] interesting pod/e2e-extmod-b-1a18c8-openshift-default-67d7b5995c-z4csj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Nov 03 13:25:44.231606 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:44.231523 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-1a18c8-openshift-default-67d7b5995c-z4csj" podUID="168564ee-9452-4aba-8391-d4f0bdaafa7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Nov 03 13:25:44.324942 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:44.324902 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4l586\" (UniqueName: \"kubernetes.io/projected/7dae3b52-27f3-42bb-98f1-46770039ab96-kube-api-access-4l586\") pod \"payload-processing-e2e-migrate-2dbf739c-75f684fcd9-zw8cw\" (UID: \"7dae3b52-27f3-42bb-98f1-46770039ab96\") " pod="openshift-ingress/payload-processing-e2e-migrate-2dbf739c-75f684fcd9-zw8cw" Nov 03 13:25:44.326132 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:44.326105 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7dae3b52-27f3-42bb-98f1-46770039ab96-plugins-config-volume\") pod \"payload-processing-e2e-migrate-2dbf739c-75f684fcd9-zw8cw\" (UID: \"7dae3b52-27f3-42bb-98f1-46770039ab96\") " pod="openshift-ingress/payload-processing-e2e-migrate-2dbf739c-75f684fcd9-zw8cw" Nov 03 13:25:44.430244 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:44.430202 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7dae3b52-27f3-42bb-98f1-46770039ab96-plugins-config-volume\") pod \"payload-processing-e2e-migrate-2dbf739c-75f684fcd9-zw8cw\" (UID: \"7dae3b52-27f3-42bb-98f1-46770039ab96\") " pod="openshift-ingress/payload-processing-e2e-migrate-2dbf739c-75f684fcd9-zw8cw" Nov 03 13:25:44.432459 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:44.432429 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4l586\" (UniqueName: \"kubernetes.io/projected/7dae3b52-27f3-42bb-98f1-46770039ab96-kube-api-access-4l586\") pod \"payload-processing-e2e-migrate-2dbf739c-75f684fcd9-zw8cw\" (UID: \"7dae3b52-27f3-42bb-98f1-46770039ab96\") " pod="openshift-ingress/payload-processing-e2e-migrate-2dbf739c-75f684fcd9-zw8cw" Nov 03 13:25:44.450300 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:44.432765 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7dae3b52-27f3-42bb-98f1-46770039ab96-plugins-config-volume\") pod \"payload-processing-e2e-migrate-2dbf739c-75f684fcd9-zw8cw\" (UID: \"7dae3b52-27f3-42bb-98f1-46770039ab96\") " pod="openshift-ingress/payload-processing-e2e-migrate-2dbf739c-75f684fcd9-zw8cw" Nov 03 13:25:44.501306 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:44.499107 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4l586\" (UniqueName: \"kubernetes.io/projected/7dae3b52-27f3-42bb-98f1-46770039ab96-kube-api-access-4l586\") pod \"payload-processing-e2e-migrate-2dbf739c-75f684fcd9-zw8cw\" (UID: \"7dae3b52-27f3-42bb-98f1-46770039ab96\") " pod="openshift-ingress/payload-processing-e2e-migrate-2dbf739c-75f684fcd9-zw8cw" Nov 03 13:25:44.668503 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:44.661642 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-6z9tv" event={"ID":"7d77cd17-8ce7-4fe7-9b61-7e4c7cdc720b","Type":"ContainerStarted","Data":"d7b5b985f79627cac6c18273d4df80d60d9c163fd8078d539b655cd26becda5a"} Nov 03 13:25:44.668503 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:44.661706 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-6z9tv" event={"ID":"7d77cd17-8ce7-4fe7-9b61-7e4c7cdc720b","Type":"ContainerStarted","Data":"d4ee62a1b39081f06a523f86ba39c8fe9bce3b81b579f7ff80e4feb80c9d07e5"} Nov 03 13:25:44.722364 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:44.712552 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-6z9tv" Nov 03 13:25:44.722364 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:44.718187 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-6z9tv" podStartSLOduration=2.718164516 podStartE2EDuration="2.718164516s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:25:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:25:44.712395285 +0000 UTC m=+1024.720586558" watchObservedRunningTime="2026-11-03 13:25:44.718164516 +0000 UTC m=+1024.726355793" Nov 03 13:25:44.752313 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:44.751576 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-2dbf739c-75f684fcd9-zw8cw" Nov 03 13:25:45.230665 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:45.230629 5024 patch_prober.go:29] interesting pod/e2e-extmod-b-1a18c8-openshift-default-67d7b5995c-z4csj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Nov 03 13:25:45.231114 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:45.231080 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-1a18c8-openshift-default-67d7b5995c-z4csj" podUID="168564ee-9452-4aba-8391-d4f0bdaafa7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Nov 03 13:25:45.410173 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:45.408691 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-2dbf739c-75f684fcd9-zw8cw"] Nov 03 13:25:45.721481 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:45.720685 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-6z9tv" Nov 03 13:25:45.729225 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:45.729192 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-2dbf739c-75f684fcd9-zw8cw" event={"ID":"7dae3b52-27f3-42bb-98f1-46770039ab96","Type":"ContainerStarted","Data":"0c662cd5ac1cf8ac9c414e0cd7481dada401316ddf91fc0e958cf11e5dee396d"} Nov 03 13:25:45.729578 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:45.729562 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-6z9tv" Nov 03 13:25:45.730725 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:45.730702 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-6z9tv" Nov 03 13:25:46.230326 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:46.230274 5024 patch_prober.go:29] interesting pod/e2e-extmod-b-1a18c8-openshift-default-67d7b5995c-z4csj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Nov 03 13:25:46.230513 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:46.230342 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-1a18c8-openshift-default-67d7b5995c-z4csj" podUID="168564ee-9452-4aba-8391-d4f0bdaafa7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Nov 03 13:25:46.861425 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:46.861393 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-2dbf739c-75f684fcd9-zw8cw" event={"ID":"7dae3b52-27f3-42bb-98f1-46770039ab96","Type":"ContainerStarted","Data":"4893976f07c7a369c8a84d120eaf90b9f5c7f49d88d8c65ac5c8e4883cc24df4"} Nov 03 13:25:46.861936 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:46.861913 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-2dbf739c-75f684fcd9-zw8cw" Nov 03 13:25:46.885149 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:46.885081 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-2dbf739c-75f684fcd9-zw8cw" podStartSLOduration=2.885060213 podStartE2EDuration="2.885060213s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:25:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:25:46.882097363 +0000 UTC m=+1026.890288651" watchObservedRunningTime="2026-11-03 13:25:46.885060213 +0000 UTC m=+1026.893251489" Nov 03 13:25:47.239415 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:47.238165 5024 patch_prober.go:29] interesting pod/e2e-extmod-b-1a18c8-openshift-default-67d7b5995c-z4csj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Nov 03 13:25:47.239415 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:47.238228 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-1a18c8-openshift-default-67d7b5995c-z4csj" podUID="168564ee-9452-4aba-8391-d4f0bdaafa7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Nov 03 13:25:47.618588 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:47.618544 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-1a18c8-openshift-default-67d7b5995c-z4csj" Nov 03 13:25:48.230533 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:48.230354 5024 patch_prober.go:29] interesting pod/e2e-extmod-b-1a18c8-openshift-default-67d7b5995c-z4csj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Nov 03 13:25:48.230533 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:48.230424 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-1a18c8-openshift-default-67d7b5995c-z4csj" podUID="168564ee-9452-4aba-8391-d4f0bdaafa7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Nov 03 13:25:48.547464 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:48.547380 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-9add72-6b8fc4988f-kxlrl"] Nov 03 13:25:48.553813 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:48.553789 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-9add72-6b8fc4988f-kxlrl" Nov 03 13:25:48.561540 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:48.561504 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-9add72-6b8fc4988f-kxlrl"] Nov 03 13:25:48.733329 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:48.700880 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w0-9add72\"" Nov 03 13:25:48.733329 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:48.700961 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w0-9add72-dockercfg-wdhth\"" Nov 03 13:25:48.798415 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:48.764211 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4f6951d-6317-4b83-b744-723ed7e576a4-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-9add72-6b8fc4988f-kxlrl\" (UID: \"a4f6951d-6317-4b83-b744-723ed7e576a4\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-9add72-6b8fc4988f-kxlrl" Nov 03 13:25:48.799023 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:48.798975 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jrf25\" (UniqueName: \"kubernetes.io/projected/a4f6951d-6317-4b83-b744-723ed7e576a4-kube-api-access-jrf25\") pod \"payload-processing-e2e-shared-b-w0-9add72-6b8fc4988f-kxlrl\" (UID: \"a4f6951d-6317-4b83-b744-723ed7e576a4\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-9add72-6b8fc4988f-kxlrl" Nov 03 13:25:48.904307 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:48.901277 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4f6951d-6317-4b83-b744-723ed7e576a4-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-9add72-6b8fc4988f-kxlrl\" (UID: \"a4f6951d-6317-4b83-b744-723ed7e576a4\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-9add72-6b8fc4988f-kxlrl" Nov 03 13:25:48.904307 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:48.901572 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrf25\" (UniqueName: \"kubernetes.io/projected/a4f6951d-6317-4b83-b744-723ed7e576a4-kube-api-access-jrf25\") pod \"payload-processing-e2e-shared-b-w0-9add72-6b8fc4988f-kxlrl\" (UID: \"a4f6951d-6317-4b83-b744-723ed7e576a4\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-9add72-6b8fc4988f-kxlrl" Nov 03 13:25:48.904307 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:48.902919 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4f6951d-6317-4b83-b744-723ed7e576a4-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-9add72-6b8fc4988f-kxlrl\" (UID: \"a4f6951d-6317-4b83-b744-723ed7e576a4\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-9add72-6b8fc4988f-kxlrl" Nov 03 13:25:48.952949 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:48.952920 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrf25\" (UniqueName: \"kubernetes.io/projected/a4f6951d-6317-4b83-b744-723ed7e576a4-kube-api-access-jrf25\") pod \"payload-processing-e2e-shared-b-w0-9add72-6b8fc4988f-kxlrl\" (UID: \"a4f6951d-6317-4b83-b744-723ed7e576a4\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-9add72-6b8fc4988f-kxlrl" Nov 03 13:25:49.199306 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:49.194600 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-9add72-6b8fc4988f-kxlrl" Nov 03 13:25:49.246639 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:49.246598 5024 patch_prober.go:29] interesting pod/e2e-extmod-b-1a18c8-openshift-default-67d7b5995c-z4csj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Nov 03 13:25:49.247477 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:49.247342 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-1a18c8-openshift-default-67d7b5995c-z4csj" podUID="168564ee-9452-4aba-8391-d4f0bdaafa7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Nov 03 13:25:49.534585 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:49.534341 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-9add72-6b8fc4988f-kxlrl"] Nov 03 13:25:49.535528 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:25:49.535500 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4f6951d_6317_4b83_b744_723ed7e576a4.slice/crio-340a0ae6096a46948e378716e173176790aff81d3a68f364023e2daea1f65e77 WatchSource:0}: Error finding container 340a0ae6096a46948e378716e173176790aff81d3a68f364023e2daea1f65e77: Status 404 returned error can't find the container with id 340a0ae6096a46948e378716e173176790aff81d3a68f364023e2daea1f65e77 Nov 03 13:25:49.972675 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:49.972628 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-9add72-6b8fc4988f-kxlrl" event={"ID":"a4f6951d-6317-4b83-b744-723ed7e576a4","Type":"ContainerStarted","Data":"d05fa2e6f498a28aed9b557a946245a32b83b18384d5d8202984791b478ff736"} Nov 03 13:25:49.972988 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:49.972959 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-9add72-6b8fc4988f-kxlrl" event={"ID":"a4f6951d-6317-4b83-b744-723ed7e576a4","Type":"ContainerStarted","Data":"340a0ae6096a46948e378716e173176790aff81d3a68f364023e2daea1f65e77"} Nov 03 13:25:49.980399 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:49.980369 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-9add72-6b8fc4988f-kxlrl" Nov 03 13:25:50.033910 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:50.033848 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-9add72-6b8fc4988f-kxlrl" podStartSLOduration=2.033814853 podStartE2EDuration="2.033814853s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:25:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:25:50.031858846 +0000 UTC m=+1030.040050132" watchObservedRunningTime="2026-11-03 13:25:50.033814853 +0000 UTC m=+1030.042006132" Nov 03 13:25:50.247400 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:50.247311 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-1a18c8-openshift-default-67d7b5995c-z4csj" Nov 03 13:25:50.258049 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:50.258019 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-1a18c8-openshift-default-67d7b5995c-z4csj" Nov 03 13:25:56.555203 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:56.555159 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-7b5bt8chm"] Nov 03 13:25:56.561544 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:56.561517 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-7b5bt8chm" Nov 03 13:25:56.566793 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:56.566760 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aad3011d-09ea-4a06-8979-f234cde784bf-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-7c192b-7b5bt8chm\" (UID: \"aad3011d-09ea-4a06-8979-f234cde784bf\") " pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-7b5bt8chm" Nov 03 13:25:56.566953 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:56.566915 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-phswp\" (UniqueName: \"kubernetes.io/projected/aad3011d-09ea-4a06-8979-f234cde784bf-kube-api-access-phswp\") pod \"payload-processing-external-model-e2e-extmod-b-7c192b-7b5bt8chm\" (UID: \"aad3011d-09ea-4a06-8979-f234cde784bf\") " pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-7b5bt8chm" Nov 03 13:25:56.567065 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:56.567038 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/aad3011d-09ea-4a06-8979-f234cde784bf-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-7c192b-7b5bt8chm\" (UID: \"aad3011d-09ea-4a06-8979-f234cde784bf\") " pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-7b5bt8chm" Nov 03 13:25:56.567321 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:56.567279 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/aad3011d-09ea-4a06-8979-f234cde784bf-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-7c192b-7b5bt8chm\" (UID: \"aad3011d-09ea-4a06-8979-f234cde784bf\") " pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-7b5bt8chm" Nov 03 13:25:56.569016 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:56.568993 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-7b5bt8chm"] Nov 03 13:25:56.569646 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:56.569625 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-7c192b\"/\"agc-cred-a79db4da35b5\"" Nov 03 13:25:56.668503 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:56.668464 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/aad3011d-09ea-4a06-8979-f234cde784bf-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-7c192b-7b5bt8chm\" (UID: \"aad3011d-09ea-4a06-8979-f234cde784bf\") " pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-7b5bt8chm" Nov 03 13:25:56.671253 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:56.671210 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aad3011d-09ea-4a06-8979-f234cde784bf-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-7c192b-7b5bt8chm\" (UID: \"aad3011d-09ea-4a06-8979-f234cde784bf\") " pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-7b5bt8chm" Nov 03 13:25:56.673177 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:56.672566 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phswp\" (UniqueName: \"kubernetes.io/projected/aad3011d-09ea-4a06-8979-f234cde784bf-kube-api-access-phswp\") pod \"payload-processing-external-model-e2e-extmod-b-7c192b-7b5bt8chm\" (UID: \"aad3011d-09ea-4a06-8979-f234cde784bf\") " pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-7b5bt8chm" Nov 03 13:25:56.674965 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:56.674892 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/aad3011d-09ea-4a06-8979-f234cde784bf-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-7c192b-7b5bt8chm\" (UID: \"aad3011d-09ea-4a06-8979-f234cde784bf\") " pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-7b5bt8chm" Nov 03 13:25:56.675324 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:56.675279 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/aad3011d-09ea-4a06-8979-f234cde784bf-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-7c192b-7b5bt8chm\" (UID: \"aad3011d-09ea-4a06-8979-f234cde784bf\") " pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-7b5bt8chm" Nov 03 13:25:56.676269 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:56.675977 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aad3011d-09ea-4a06-8979-f234cde784bf-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-7c192b-7b5bt8chm\" (UID: \"aad3011d-09ea-4a06-8979-f234cde784bf\") " pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-7b5bt8chm" Nov 03 13:25:56.682789 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:56.682759 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/aad3011d-09ea-4a06-8979-f234cde784bf-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-7c192b-7b5bt8chm\" (UID: \"aad3011d-09ea-4a06-8979-f234cde784bf\") " pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-7b5bt8chm" Nov 03 13:25:56.687086 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:56.687054 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-phswp\" (UniqueName: \"kubernetes.io/projected/aad3011d-09ea-4a06-8979-f234cde784bf-kube-api-access-phswp\") pod \"payload-processing-external-model-e2e-extmod-b-7c192b-7b5bt8chm\" (UID: \"aad3011d-09ea-4a06-8979-f234cde784bf\") " pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-7b5bt8chm" Nov 03 13:25:56.810485 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:56.810393 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-9add72-f74fc96d-xgkrk"] Nov 03 13:25:56.816406 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:56.816367 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-9add72-f74fc96d-xgkrk" Nov 03 13:25:56.828136 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:56.828107 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-9add72-f74fc96d-xgkrk"] Nov 03 13:25:56.857849 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:56.857819 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w0-9add72-serving-cert\"" Nov 03 13:25:56.879549 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:56.879494 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fa15c0e4-6d08-4a12-9320-687d6f47ed4e-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-9add72-f74fc96d-xgkrk\" (UID: \"fa15c0e4-6d08-4a12-9320-687d6f47ed4e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-9add72-f74fc96d-xgkrk" Nov 03 13:25:56.879836 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:56.879639 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa15c0e4-6d08-4a12-9320-687d6f47ed4e-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-9add72-f74fc96d-xgkrk\" (UID: \"fa15c0e4-6d08-4a12-9320-687d6f47ed4e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-9add72-f74fc96d-xgkrk" Nov 03 13:25:56.879836 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:56.879705 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-phvzn\" (UniqueName: \"kubernetes.io/projected/fa15c0e4-6d08-4a12-9320-687d6f47ed4e-kube-api-access-phvzn\") pod \"maas-api-e2e-shared-b-w0-9add72-f74fc96d-xgkrk\" (UID: \"fa15c0e4-6d08-4a12-9320-687d6f47ed4e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-9add72-f74fc96d-xgkrk" Nov 03 13:25:56.879836 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:56.879734 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fa15c0e4-6d08-4a12-9320-687d6f47ed4e-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-9add72-f74fc96d-xgkrk\" (UID: \"fa15c0e4-6d08-4a12-9320-687d6f47ed4e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-9add72-f74fc96d-xgkrk" Nov 03 13:25:56.881711 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:56.881684 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-7b5bt8chm" Nov 03 13:25:57.085449 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:57.074700 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa15c0e4-6d08-4a12-9320-687d6f47ed4e-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-9add72-f74fc96d-xgkrk\" (UID: \"fa15c0e4-6d08-4a12-9320-687d6f47ed4e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-9add72-f74fc96d-xgkrk" Nov 03 13:25:57.085449 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:57.074804 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phvzn\" (UniqueName: \"kubernetes.io/projected/fa15c0e4-6d08-4a12-9320-687d6f47ed4e-kube-api-access-phvzn\") pod \"maas-api-e2e-shared-b-w0-9add72-f74fc96d-xgkrk\" (UID: \"fa15c0e4-6d08-4a12-9320-687d6f47ed4e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-9add72-f74fc96d-xgkrk" Nov 03 13:25:57.085449 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:57.074862 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fa15c0e4-6d08-4a12-9320-687d6f47ed4e-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-9add72-f74fc96d-xgkrk\" (UID: \"fa15c0e4-6d08-4a12-9320-687d6f47ed4e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-9add72-f74fc96d-xgkrk" Nov 03 13:25:57.085449 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:57.075294 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fa15c0e4-6d08-4a12-9320-687d6f47ed4e-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-9add72-f74fc96d-xgkrk\" (UID: \"fa15c0e4-6d08-4a12-9320-687d6f47ed4e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-9add72-f74fc96d-xgkrk" Nov 03 13:25:57.085449 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:57.083523 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa15c0e4-6d08-4a12-9320-687d6f47ed4e-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-9add72-f74fc96d-xgkrk\" (UID: \"fa15c0e4-6d08-4a12-9320-687d6f47ed4e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-9add72-f74fc96d-xgkrk" Nov 03 13:25:57.087348 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:57.087147 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fa15c0e4-6d08-4a12-9320-687d6f47ed4e-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-9add72-f74fc96d-xgkrk\" (UID: \"fa15c0e4-6d08-4a12-9320-687d6f47ed4e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-9add72-f74fc96d-xgkrk" Nov 03 13:25:57.136086 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:57.132070 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fa15c0e4-6d08-4a12-9320-687d6f47ed4e-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-9add72-f74fc96d-xgkrk\" (UID: \"fa15c0e4-6d08-4a12-9320-687d6f47ed4e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-9add72-f74fc96d-xgkrk" Nov 03 13:25:57.136086 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:57.132858 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-phvzn\" (UniqueName: \"kubernetes.io/projected/fa15c0e4-6d08-4a12-9320-687d6f47ed4e-kube-api-access-phvzn\") pod \"maas-api-e2e-shared-b-w0-9add72-f74fc96d-xgkrk\" (UID: \"fa15c0e4-6d08-4a12-9320-687d6f47ed4e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-9add72-f74fc96d-xgkrk" Nov 03 13:25:57.146329 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:57.144986 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-9add72-f74fc96d-xgkrk" Nov 03 13:25:57.538275 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:57.538242 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-7b5bt8chm"] Nov 03 13:25:57.552313 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:57.545900 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-9add72-f74fc96d-xgkrk"] Nov 03 13:25:57.560148 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:25:57.560111 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaad3011d_09ea_4a06_8979_f234cde784bf.slice/crio-743d0472ba8ee72e8c2e704528114806e22aa079b97f3952b7da785e789931f5 WatchSource:0}: Error finding container 743d0472ba8ee72e8c2e704528114806e22aa079b97f3952b7da785e789931f5: Status 404 returned error can't find the container with id 743d0472ba8ee72e8c2e704528114806e22aa079b97f3952b7da785e789931f5 Nov 03 13:25:57.566312 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:57.566250 5024 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 13:25:58.195187 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:58.195137 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-9add72-f74fc96d-xgkrk" event={"ID":"fa15c0e4-6d08-4a12-9320-687d6f47ed4e","Type":"ContainerStarted","Data":"e661833ee83ad6f141e043af449aa4d1477fde8be130663c70aa02cbed593871"} Nov 03 13:25:58.195393 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:58.195211 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-9add72-f74fc96d-xgkrk" event={"ID":"fa15c0e4-6d08-4a12-9320-687d6f47ed4e","Type":"ContainerStarted","Data":"12d20e8ea37e025eeda488523b7f296ac845f25d12006da2be3f313b5dc1f545"} Nov 03 13:25:58.195472 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:58.195396 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-9add72-f74fc96d-xgkrk" Nov 03 13:25:58.197041 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:58.197014 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-7b5bt8chm" event={"ID":"aad3011d-09ea-4a06-8979-f234cde784bf","Type":"ContainerStarted","Data":"1481840671a84baf66ea96d1a55d37385142951e5025cd18eb927e8473c6a140"} Nov 03 13:25:58.197180 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:58.197061 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-7b5bt8chm" event={"ID":"aad3011d-09ea-4a06-8979-f234cde784bf","Type":"ContainerStarted","Data":"743d0472ba8ee72e8c2e704528114806e22aa079b97f3952b7da785e789931f5"} Nov 03 13:25:58.197180 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:58.197092 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-7b5bt8chm" Nov 03 13:25:58.219180 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:58.218856 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-9add72-f74fc96d-xgkrk" podStartSLOduration=1.722655939 podStartE2EDuration="2.218837172s" totalImagesPullingTime="496.181233ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:25:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:25:58.216638043 +0000 UTC m=+1038.224829320" watchObservedRunningTime="2026-11-03 13:25:58.218837172 +0000 UTC m=+1038.227028448" Nov 03 13:25:58.249413 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:58.249358 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-7b5bt8chm" podStartSLOduration=2.249342093 podStartE2EDuration="2.249342093s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:25:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:25:58.246629795 +0000 UTC m=+1038.254821072" watchObservedRunningTime="2026-11-03 13:25:58.249342093 +0000 UTC m=+1038.257533369" Nov 03 13:25:58.959399 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:25:58.959370 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-2dbf739c-75f684fcd9-zw8cw" Nov 03 13:26:02.069194 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:02.069161 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-9add72-6b8fc4988f-kxlrl" Nov 03 13:26:04.211264 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:04.211229 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-9add72-f74fc96d-xgkrk" Nov 03 13:26:09.209065 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:09.209031 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-7b5bt8chm" Nov 03 13:26:09.271274 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:09.271239 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-8444wtvfx"] Nov 03 13:26:09.536204 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:26:09.536106 5024 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d7mwz" containerName="extractor" Nov 03 13:26:09.786970 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:09.786881 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-8lgm7"] Nov 03 13:26:09.794551 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:09.794463 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-8lgm7" Nov 03 13:26:09.843502 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:09.843457 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-8lgm7"] Nov 03 13:26:09.860466 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:09.860428 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8657989-e2c5-4dfc-b32a-3491ad189395-credential-socket\") pod \"e2e-extmod-b-7c192b-openshift-default-5c496b978f-8lgm7\" (UID: \"c8657989-e2c5-4dfc-b32a-3491ad189395\") " pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-8lgm7" Nov 03 13:26:09.860662 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:09.860561 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8657989-e2c5-4dfc-b32a-3491ad189395-workload-socket\") pod \"e2e-extmod-b-7c192b-openshift-default-5c496b978f-8lgm7\" (UID: \"c8657989-e2c5-4dfc-b32a-3491ad189395\") " pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-8lgm7" Nov 03 13:26:09.860662 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:09.860604 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8657989-e2c5-4dfc-b32a-3491ad189395-istio-podinfo\") pod \"e2e-extmod-b-7c192b-openshift-default-5c496b978f-8lgm7\" (UID: \"c8657989-e2c5-4dfc-b32a-3491ad189395\") " pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-8lgm7" Nov 03 13:26:09.860662 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:09.860648 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8657989-e2c5-4dfc-b32a-3491ad189395-istio-envoy\") pod \"e2e-extmod-b-7c192b-openshift-default-5c496b978f-8lgm7\" (UID: \"c8657989-e2c5-4dfc-b32a-3491ad189395\") " pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-8lgm7" Nov 03 13:26:09.860844 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:09.860742 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8657989-e2c5-4dfc-b32a-3491ad189395-istio-token\") pod \"e2e-extmod-b-7c192b-openshift-default-5c496b978f-8lgm7\" (UID: \"c8657989-e2c5-4dfc-b32a-3491ad189395\") " pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-8lgm7" Nov 03 13:26:09.860844 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:09.860789 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8657989-e2c5-4dfc-b32a-3491ad189395-istio-data\") pod \"e2e-extmod-b-7c192b-openshift-default-5c496b978f-8lgm7\" (UID: \"c8657989-e2c5-4dfc-b32a-3491ad189395\") " pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-8lgm7" Nov 03 13:26:09.860952 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:09.860864 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8657989-e2c5-4dfc-b32a-3491ad189395-workload-certs\") pod \"e2e-extmod-b-7c192b-openshift-default-5c496b978f-8lgm7\" (UID: \"c8657989-e2c5-4dfc-b32a-3491ad189395\") " pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-8lgm7" Nov 03 13:26:09.860952 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:09.860945 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8657989-e2c5-4dfc-b32a-3491ad189395-istiod-ca-cert\") pod \"e2e-extmod-b-7c192b-openshift-default-5c496b978f-8lgm7\" (UID: \"c8657989-e2c5-4dfc-b32a-3491ad189395\") " pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-8lgm7" Nov 03 13:26:09.861052 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:09.861000 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jk96l\" (UniqueName: \"kubernetes.io/projected/c8657989-e2c5-4dfc-b32a-3491ad189395-kube-api-access-jk96l\") pod \"e2e-extmod-b-7c192b-openshift-default-5c496b978f-8lgm7\" (UID: \"c8657989-e2c5-4dfc-b32a-3491ad189395\") " pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-8lgm7" Nov 03 13:26:09.963059 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:09.963014 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8657989-e2c5-4dfc-b32a-3491ad189395-istio-token\") pod \"e2e-extmod-b-7c192b-openshift-default-5c496b978f-8lgm7\" (UID: \"c8657989-e2c5-4dfc-b32a-3491ad189395\") " pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-8lgm7" Nov 03 13:26:09.963263 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:09.963072 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8657989-e2c5-4dfc-b32a-3491ad189395-istio-data\") pod \"e2e-extmod-b-7c192b-openshift-default-5c496b978f-8lgm7\" (UID: \"c8657989-e2c5-4dfc-b32a-3491ad189395\") " pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-8lgm7" Nov 03 13:26:09.963263 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:09.963120 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8657989-e2c5-4dfc-b32a-3491ad189395-workload-certs\") pod \"e2e-extmod-b-7c192b-openshift-default-5c496b978f-8lgm7\" (UID: \"c8657989-e2c5-4dfc-b32a-3491ad189395\") " pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-8lgm7" Nov 03 13:26:09.963263 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:09.963168 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8657989-e2c5-4dfc-b32a-3491ad189395-istiod-ca-cert\") pod \"e2e-extmod-b-7c192b-openshift-default-5c496b978f-8lgm7\" (UID: \"c8657989-e2c5-4dfc-b32a-3491ad189395\") " pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-8lgm7" Nov 03 13:26:09.963263 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:09.963201 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jk96l\" (UniqueName: \"kubernetes.io/projected/c8657989-e2c5-4dfc-b32a-3491ad189395-kube-api-access-jk96l\") pod \"e2e-extmod-b-7c192b-openshift-default-5c496b978f-8lgm7\" (UID: \"c8657989-e2c5-4dfc-b32a-3491ad189395\") " pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-8lgm7" Nov 03 13:26:09.963263 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:09.963244 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8657989-e2c5-4dfc-b32a-3491ad189395-credential-socket\") pod \"e2e-extmod-b-7c192b-openshift-default-5c496b978f-8lgm7\" (UID: \"c8657989-e2c5-4dfc-b32a-3491ad189395\") " pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-8lgm7" Nov 03 13:26:09.963571 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:09.963347 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8657989-e2c5-4dfc-b32a-3491ad189395-workload-socket\") pod \"e2e-extmod-b-7c192b-openshift-default-5c496b978f-8lgm7\" (UID: \"c8657989-e2c5-4dfc-b32a-3491ad189395\") " pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-8lgm7" Nov 03 13:26:09.963571 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:09.963384 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8657989-e2c5-4dfc-b32a-3491ad189395-istio-podinfo\") pod \"e2e-extmod-b-7c192b-openshift-default-5c496b978f-8lgm7\" (UID: \"c8657989-e2c5-4dfc-b32a-3491ad189395\") " pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-8lgm7" Nov 03 13:26:09.963571 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:09.963430 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8657989-e2c5-4dfc-b32a-3491ad189395-istio-envoy\") pod \"e2e-extmod-b-7c192b-openshift-default-5c496b978f-8lgm7\" (UID: \"c8657989-e2c5-4dfc-b32a-3491ad189395\") " pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-8lgm7" Nov 03 13:26:09.963737 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:09.963675 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8657989-e2c5-4dfc-b32a-3491ad189395-istio-data\") pod \"e2e-extmod-b-7c192b-openshift-default-5c496b978f-8lgm7\" (UID: \"c8657989-e2c5-4dfc-b32a-3491ad189395\") " pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-8lgm7" Nov 03 13:26:09.963800 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:09.963751 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8657989-e2c5-4dfc-b32a-3491ad189395-credential-socket\") pod \"e2e-extmod-b-7c192b-openshift-default-5c496b978f-8lgm7\" (UID: \"c8657989-e2c5-4dfc-b32a-3491ad189395\") " pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-8lgm7" Nov 03 13:26:09.963855 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:09.963841 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8657989-e2c5-4dfc-b32a-3491ad189395-workload-certs\") pod \"e2e-extmod-b-7c192b-openshift-default-5c496b978f-8lgm7\" (UID: \"c8657989-e2c5-4dfc-b32a-3491ad189395\") " pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-8lgm7" Nov 03 13:26:09.963907 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:09.963879 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8657989-e2c5-4dfc-b32a-3491ad189395-workload-socket\") pod \"e2e-extmod-b-7c192b-openshift-default-5c496b978f-8lgm7\" (UID: \"c8657989-e2c5-4dfc-b32a-3491ad189395\") " pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-8lgm7" Nov 03 13:26:09.964215 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:09.964189 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8657989-e2c5-4dfc-b32a-3491ad189395-istiod-ca-cert\") pod \"e2e-extmod-b-7c192b-openshift-default-5c496b978f-8lgm7\" (UID: \"c8657989-e2c5-4dfc-b32a-3491ad189395\") " pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-8lgm7" Nov 03 13:26:09.965866 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:09.965844 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8657989-e2c5-4dfc-b32a-3491ad189395-istio-envoy\") pod \"e2e-extmod-b-7c192b-openshift-default-5c496b978f-8lgm7\" (UID: \"c8657989-e2c5-4dfc-b32a-3491ad189395\") " pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-8lgm7" Nov 03 13:26:09.966089 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:09.966067 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8657989-e2c5-4dfc-b32a-3491ad189395-istio-podinfo\") pod \"e2e-extmod-b-7c192b-openshift-default-5c496b978f-8lgm7\" (UID: \"c8657989-e2c5-4dfc-b32a-3491ad189395\") " pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-8lgm7" Nov 03 13:26:09.989369 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:09.989265 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8657989-e2c5-4dfc-b32a-3491ad189395-istio-token\") pod \"e2e-extmod-b-7c192b-openshift-default-5c496b978f-8lgm7\" (UID: \"c8657989-e2c5-4dfc-b32a-3491ad189395\") " pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-8lgm7" Nov 03 13:26:09.989574 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:09.989418 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jk96l\" (UniqueName: \"kubernetes.io/projected/c8657989-e2c5-4dfc-b32a-3491ad189395-kube-api-access-jk96l\") pod \"e2e-extmod-b-7c192b-openshift-default-5c496b978f-8lgm7\" (UID: \"c8657989-e2c5-4dfc-b32a-3491ad189395\") " pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-8lgm7" Nov 03 13:26:10.127879 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:10.127595 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-8lgm7" Nov 03 13:26:10.628529 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:10.628487 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-8lgm7"] Nov 03 13:26:10.757006 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:10.756959 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:26:10.757318 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:10.757278 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:26:10.757479 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:10.757463 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:26:11.917306 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:11.913646 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-8lgm7" event={"ID":"c8657989-e2c5-4dfc-b32a-3491ad189395","Type":"ContainerStarted","Data":"7d1b86e4edd6ae09cc252a82929f5f7c328be74f5ff0057cbaa26ff45c752cb4"} Nov 03 13:26:11.917306 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:11.913753 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-8lgm7" event={"ID":"c8657989-e2c5-4dfc-b32a-3491ad189395","Type":"ContainerStarted","Data":"7c6f219cd131a6784235566b21782bd96fe4486658a0a423ecbaa57a0c6346e2"} Nov 03 13:26:11.981994 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:11.981924 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-8lgm7" podStartSLOduration=2.98190012 podStartE2EDuration="2.98190012s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:26:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:26:11.953146826 +0000 UTC m=+1051.961338104" watchObservedRunningTime="2026-11-03 13:26:11.98190012 +0000 UTC m=+1051.990091396" Nov 03 13:26:12.136915 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:12.136862 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-8lgm7" Nov 03 13:26:12.157814 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:12.157776 5024 patch_prober.go:29] interesting pod/e2e-extmod-b-7c192b-openshift-default-5c496b978f-8lgm7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body= Nov 03 13:26:12.158090 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:12.158065 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-8lgm7" podUID="c8657989-e2c5-4dfc-b32a-3491ad189395" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" Nov 03 13:26:13.137311 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:13.131755 5024 patch_prober.go:29] interesting pod/e2e-extmod-b-7c192b-openshift-default-5c496b978f-8lgm7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body= Nov 03 13:26:13.137311 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:13.131824 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-8lgm7" podUID="c8657989-e2c5-4dfc-b32a-3491ad189395" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" Nov 03 13:26:14.157308 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:14.142714 5024 patch_prober.go:29] interesting pod/e2e-extmod-b-7c192b-openshift-default-5c496b978f-8lgm7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body= Nov 03 13:26:14.157308 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:14.142778 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-8lgm7" podUID="c8657989-e2c5-4dfc-b32a-3491ad189395" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" Nov 03 13:26:14.280969 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:14.280162 5024 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-8444wtvfx" podUID="ea6d1245-8491-4503-9a7d-c80b6b04d811" containerName="payload-processing" containerID="cri-o://1c94023325759c305773eaa6eda3b4d7483194c8b118b3253efc058b2666b548" gracePeriod=25 Nov 03 13:26:15.134178 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:15.134135 5024 patch_prober.go:29] interesting pod/e2e-extmod-b-7c192b-openshift-default-5c496b978f-8lgm7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body= Nov 03 13:26:15.134601 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:15.134562 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-8lgm7" podUID="c8657989-e2c5-4dfc-b32a-3491ad189395" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" Nov 03 13:26:15.454309 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:15.453206 5024 generic.go:356] "Generic (PLEG): container finished" podID="ea6d1245-8491-4503-9a7d-c80b6b04d811" containerID="1c94023325759c305773eaa6eda3b4d7483194c8b118b3253efc058b2666b548" exitCode=0 Nov 03 13:26:15.454309 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:15.453503 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-8444wtvfx" event={"ID":"ea6d1245-8491-4503-9a7d-c80b6b04d811","Type":"ContainerDied","Data":"1c94023325759c305773eaa6eda3b4d7483194c8b118b3253efc058b2666b548"} Nov 03 13:26:15.561908 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:15.561481 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-8444wtvfx" Nov 03 13:26:15.656080 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:15.656041 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ea6d1245-8491-4503-9a7d-c80b6b04d811-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ea6d1245-8491-4503-9a7d-c80b6b04d811-routing-overlay-volume\") pod \"ea6d1245-8491-4503-9a7d-c80b6b04d811\" (UID: \"ea6d1245-8491-4503-9a7d-c80b6b04d811\") " Nov 03 13:26:15.660800 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:15.660769 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea6d1245-8491-4503-9a7d-c80b6b04d811-kube-api-access-thgrc\" (UniqueName: \"kubernetes.io/projected/ea6d1245-8491-4503-9a7d-c80b6b04d811-kube-api-access-thgrc\") pod \"ea6d1245-8491-4503-9a7d-c80b6b04d811\" (UID: \"ea6d1245-8491-4503-9a7d-c80b6b04d811\") " Nov 03 13:26:15.671309 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:15.661203 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea6d1245-8491-4503-9a7d-c80b6b04d811-provider-credentials\" (UniqueName: \"kubernetes.io/projected/ea6d1245-8491-4503-9a7d-c80b6b04d811-provider-credentials\") pod \"ea6d1245-8491-4503-9a7d-c80b6b04d811\" (UID: \"ea6d1245-8491-4503-9a7d-c80b6b04d811\") " Nov 03 13:26:15.677442 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:15.677395 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ea6d1245-8491-4503-9a7d-c80b6b04d811-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea6d1245-8491-4503-9a7d-c80b6b04d811-plugins-config-volume\") pod \"ea6d1245-8491-4503-9a7d-c80b6b04d811\" (UID: \"ea6d1245-8491-4503-9a7d-c80b6b04d811\") " Nov 03 13:26:15.697501 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:15.668446 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea6d1245-8491-4503-9a7d-c80b6b04d811-routing-overlay-volume" pod "ea6d1245-8491-4503-9a7d-c80b6b04d811" (UID: "ea6d1245-8491-4503-9a7d-c80b6b04d811"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:26:15.713342 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:15.708555 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea6d1245-8491-4503-9a7d-c80b6b04d811-plugins-config-volume" pod "ea6d1245-8491-4503-9a7d-c80b6b04d811" (UID: "ea6d1245-8491-4503-9a7d-c80b6b04d811"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:26:15.790320 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:15.780632 5024 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea6d1245-8491-4503-9a7d-c80b6b04d811-plugins-config-volume\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:26:15.790788 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:15.790756 5024 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ea6d1245-8491-4503-9a7d-c80b6b04d811-routing-overlay-volume\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:26:15.842365 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:15.806500 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea6d1245-8491-4503-9a7d-c80b6b04d811-kube-api-access-thgrc" pod "ea6d1245-8491-4503-9a7d-c80b6b04d811" (UID: "ea6d1245-8491-4503-9a7d-c80b6b04d811"). InnerVolumeSpecName "kube-api-access-thgrc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:26:15.893762 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:15.893270 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-thgrc\" (UniqueName: \"kubernetes.io/projected/ea6d1245-8491-4503-9a7d-c80b6b04d811-kube-api-access-thgrc\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:26:15.894172 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:15.894119 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea6d1245-8491-4503-9a7d-c80b6b04d811-provider-credentials" pod "ea6d1245-8491-4503-9a7d-c80b6b04d811" (UID: "ea6d1245-8491-4503-9a7d-c80b6b04d811"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:26:16.025972 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:15.997628 5024 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ea6d1245-8491-4503-9a7d-c80b6b04d811-provider-credentials\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:26:16.151116 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:16.150590 5024 patch_prober.go:29] interesting pod/e2e-extmod-b-7c192b-openshift-default-5c496b978f-8lgm7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body= Nov 03 13:26:16.151116 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:16.150659 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-8lgm7" podUID="c8657989-e2c5-4dfc-b32a-3491ad189395" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" Nov 03 13:26:16.526448 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:16.526342 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-8444wtvfx" event={"ID":"ea6d1245-8491-4503-9a7d-c80b6b04d811","Type":"ContainerDied","Data":"d2cdde7d85c6bcb167c8bbb1bcaea8dc7d78036e1b7e43f6b340902bda3b94b5"} Nov 03 13:26:16.527064 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:16.527042 5024 scope.go:122] "RemoveContainer" containerID="1c94023325759c305773eaa6eda3b4d7483194c8b118b3253efc058b2666b548" Nov 03 13:26:16.556013 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:16.527366 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-8444wtvfx" Nov 03 13:26:16.666451 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:16.660331 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-8444wtvfx"] Nov 03 13:26:16.778361 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:16.777651 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-8444wtvfx"] Nov 03 13:26:17.137846 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:17.137805 5024 patch_prober.go:29] interesting pod/e2e-extmod-b-7c192b-openshift-default-5c496b978f-8lgm7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body= Nov 03 13:26:17.138129 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:17.138099 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-8lgm7" podUID="c8657989-e2c5-4dfc-b32a-3491ad189395" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" Nov 03 13:26:18.141317 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:18.141215 5024 patch_prober.go:29] interesting pod/e2e-extmod-b-7c192b-openshift-default-5c496b978f-8lgm7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body= Nov 03 13:26:18.141317 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:18.141278 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-8lgm7" podUID="c8657989-e2c5-4dfc-b32a-3491ad189395" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" Nov 03 13:26:18.148928 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:18.148890 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea6d1245-8491-4503-9a7d-c80b6b04d811" path="/var/lib/kubelet/pods/ea6d1245-8491-4503-9a7d-c80b6b04d811/volumes" Nov 03 13:26:18.445867 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:18.445625 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-2dbf739c-75f684fcd9-zw8cw"] Nov 03 13:26:18.804364 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:18.780174 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-ef1693-openshift-default-66b997ddf7-fqzk5"] Nov 03 13:26:18.829441 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:18.805208 5024 cpu_manager.go:414] "containerMap: removing container" podUID="ea6d1245-8491-4503-9a7d-c80b6b04d811" containerName="payload-processing" Nov 03 13:26:18.829441 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:18.805244 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea6d1245-8491-4503-9a7d-c80b6b04d811" containerName="payload-processing" Nov 03 13:26:18.829441 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:18.805434 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea6d1245-8491-4503-9a7d-c80b6b04d811" containerName="payload-processing" Nov 03 13:26:18.842169 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:18.842136 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-ef1693-openshift-default-66b997ddf7-fqzk5" Nov 03 13:26:18.850648 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:18.850609 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-ef1693-openshift-default-66b997ddf7-fqzk5"] Nov 03 13:26:18.851463 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:18.851440 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-ef1693-openshift-default-dockercfg-fcmw7\"" Nov 03 13:26:19.001728 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:18.998640 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26bee2e8-835c-49d8-9e88-f959627e0ec9-istiod-ca-cert\") pod \"e2e-extmod-a-ef1693-openshift-default-66b997ddf7-fqzk5\" (UID: \"26bee2e8-835c-49d8-9e88-f959627e0ec9\") " pod="openshift-ingress/e2e-extmod-a-ef1693-openshift-default-66b997ddf7-fqzk5" Nov 03 13:26:19.001728 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:18.998699 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26bee2e8-835c-49d8-9e88-f959627e0ec9-istio-podinfo\") pod \"e2e-extmod-a-ef1693-openshift-default-66b997ddf7-fqzk5\" (UID: \"26bee2e8-835c-49d8-9e88-f959627e0ec9\") " pod="openshift-ingress/e2e-extmod-a-ef1693-openshift-default-66b997ddf7-fqzk5" Nov 03 13:26:19.001728 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:18.998770 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26bee2e8-835c-49d8-9e88-f959627e0ec9-credential-socket\") pod \"e2e-extmod-a-ef1693-openshift-default-66b997ddf7-fqzk5\" (UID: \"26bee2e8-835c-49d8-9e88-f959627e0ec9\") " pod="openshift-ingress/e2e-extmod-a-ef1693-openshift-default-66b997ddf7-fqzk5" Nov 03 13:26:19.001728 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:18.998813 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26bee2e8-835c-49d8-9e88-f959627e0ec9-istio-envoy\") pod \"e2e-extmod-a-ef1693-openshift-default-66b997ddf7-fqzk5\" (UID: \"26bee2e8-835c-49d8-9e88-f959627e0ec9\") " pod="openshift-ingress/e2e-extmod-a-ef1693-openshift-default-66b997ddf7-fqzk5" Nov 03 13:26:19.001728 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:18.998891 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26bee2e8-835c-49d8-9e88-f959627e0ec9-workload-socket\") pod \"e2e-extmod-a-ef1693-openshift-default-66b997ddf7-fqzk5\" (UID: \"26bee2e8-835c-49d8-9e88-f959627e0ec9\") " pod="openshift-ingress/e2e-extmod-a-ef1693-openshift-default-66b997ddf7-fqzk5" Nov 03 13:26:19.001728 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:18.998948 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26bee2e8-835c-49d8-9e88-f959627e0ec9-istio-data\") pod \"e2e-extmod-a-ef1693-openshift-default-66b997ddf7-fqzk5\" (UID: \"26bee2e8-835c-49d8-9e88-f959627e0ec9\") " pod="openshift-ingress/e2e-extmod-a-ef1693-openshift-default-66b997ddf7-fqzk5" Nov 03 13:26:19.001728 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:18.998978 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7x2lh\" (UniqueName: \"kubernetes.io/projected/26bee2e8-835c-49d8-9e88-f959627e0ec9-kube-api-access-7x2lh\") pod \"e2e-extmod-a-ef1693-openshift-default-66b997ddf7-fqzk5\" (UID: \"26bee2e8-835c-49d8-9e88-f959627e0ec9\") " pod="openshift-ingress/e2e-extmod-a-ef1693-openshift-default-66b997ddf7-fqzk5" Nov 03 13:26:19.001728 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:18.999021 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26bee2e8-835c-49d8-9e88-f959627e0ec9-istio-token\") pod \"e2e-extmod-a-ef1693-openshift-default-66b997ddf7-fqzk5\" (UID: \"26bee2e8-835c-49d8-9e88-f959627e0ec9\") " pod="openshift-ingress/e2e-extmod-a-ef1693-openshift-default-66b997ddf7-fqzk5" Nov 03 13:26:19.001728 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:18.999085 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26bee2e8-835c-49d8-9e88-f959627e0ec9-workload-certs\") pod \"e2e-extmod-a-ef1693-openshift-default-66b997ddf7-fqzk5\" (UID: \"26bee2e8-835c-49d8-9e88-f959627e0ec9\") " pod="openshift-ingress/e2e-extmod-a-ef1693-openshift-default-66b997ddf7-fqzk5" Nov 03 13:26:19.150312 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:19.121380 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26bee2e8-835c-49d8-9e88-f959627e0ec9-istiod-ca-cert\") pod \"e2e-extmod-a-ef1693-openshift-default-66b997ddf7-fqzk5\" (UID: \"26bee2e8-835c-49d8-9e88-f959627e0ec9\") " pod="openshift-ingress/e2e-extmod-a-ef1693-openshift-default-66b997ddf7-fqzk5" Nov 03 13:26:19.150312 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:19.121442 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26bee2e8-835c-49d8-9e88-f959627e0ec9-istio-podinfo\") pod \"e2e-extmod-a-ef1693-openshift-default-66b997ddf7-fqzk5\" (UID: \"26bee2e8-835c-49d8-9e88-f959627e0ec9\") " pod="openshift-ingress/e2e-extmod-a-ef1693-openshift-default-66b997ddf7-fqzk5" Nov 03 13:26:19.150312 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:19.121533 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26bee2e8-835c-49d8-9e88-f959627e0ec9-credential-socket\") pod \"e2e-extmod-a-ef1693-openshift-default-66b997ddf7-fqzk5\" (UID: \"26bee2e8-835c-49d8-9e88-f959627e0ec9\") " pod="openshift-ingress/e2e-extmod-a-ef1693-openshift-default-66b997ddf7-fqzk5" Nov 03 13:26:19.150312 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:19.121577 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26bee2e8-835c-49d8-9e88-f959627e0ec9-istio-envoy\") pod \"e2e-extmod-a-ef1693-openshift-default-66b997ddf7-fqzk5\" (UID: \"26bee2e8-835c-49d8-9e88-f959627e0ec9\") " pod="openshift-ingress/e2e-extmod-a-ef1693-openshift-default-66b997ddf7-fqzk5" Nov 03 13:26:19.150312 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:19.121677 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26bee2e8-835c-49d8-9e88-f959627e0ec9-workload-socket\") pod \"e2e-extmod-a-ef1693-openshift-default-66b997ddf7-fqzk5\" (UID: \"26bee2e8-835c-49d8-9e88-f959627e0ec9\") " pod="openshift-ingress/e2e-extmod-a-ef1693-openshift-default-66b997ddf7-fqzk5" Nov 03 13:26:19.150312 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:19.121750 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26bee2e8-835c-49d8-9e88-f959627e0ec9-istio-data\") pod \"e2e-extmod-a-ef1693-openshift-default-66b997ddf7-fqzk5\" (UID: \"26bee2e8-835c-49d8-9e88-f959627e0ec9\") " pod="openshift-ingress/e2e-extmod-a-ef1693-openshift-default-66b997ddf7-fqzk5" Nov 03 13:26:19.150312 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:19.121798 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7x2lh\" (UniqueName: \"kubernetes.io/projected/26bee2e8-835c-49d8-9e88-f959627e0ec9-kube-api-access-7x2lh\") pod \"e2e-extmod-a-ef1693-openshift-default-66b997ddf7-fqzk5\" (UID: \"26bee2e8-835c-49d8-9e88-f959627e0ec9\") " pod="openshift-ingress/e2e-extmod-a-ef1693-openshift-default-66b997ddf7-fqzk5" Nov 03 13:26:19.150312 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:19.121850 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26bee2e8-835c-49d8-9e88-f959627e0ec9-istio-token\") pod \"e2e-extmod-a-ef1693-openshift-default-66b997ddf7-fqzk5\" (UID: \"26bee2e8-835c-49d8-9e88-f959627e0ec9\") " pod="openshift-ingress/e2e-extmod-a-ef1693-openshift-default-66b997ddf7-fqzk5" Nov 03 13:26:19.150312 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:19.121973 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26bee2e8-835c-49d8-9e88-f959627e0ec9-workload-certs\") pod \"e2e-extmod-a-ef1693-openshift-default-66b997ddf7-fqzk5\" (UID: \"26bee2e8-835c-49d8-9e88-f959627e0ec9\") " pod="openshift-ingress/e2e-extmod-a-ef1693-openshift-default-66b997ddf7-fqzk5" Nov 03 13:26:19.150312 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:19.122587 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26bee2e8-835c-49d8-9e88-f959627e0ec9-workload-certs\") pod \"e2e-extmod-a-ef1693-openshift-default-66b997ddf7-fqzk5\" (UID: \"26bee2e8-835c-49d8-9e88-f959627e0ec9\") " pod="openshift-ingress/e2e-extmod-a-ef1693-openshift-default-66b997ddf7-fqzk5" Nov 03 13:26:19.150312 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:19.123271 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26bee2e8-835c-49d8-9e88-f959627e0ec9-istiod-ca-cert\") pod \"e2e-extmod-a-ef1693-openshift-default-66b997ddf7-fqzk5\" (UID: \"26bee2e8-835c-49d8-9e88-f959627e0ec9\") " pod="openshift-ingress/e2e-extmod-a-ef1693-openshift-default-66b997ddf7-fqzk5" Nov 03 13:26:19.178179 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:19.153221 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26bee2e8-835c-49d8-9e88-f959627e0ec9-workload-socket\") pod \"e2e-extmod-a-ef1693-openshift-default-66b997ddf7-fqzk5\" (UID: \"26bee2e8-835c-49d8-9e88-f959627e0ec9\") " pod="openshift-ingress/e2e-extmod-a-ef1693-openshift-default-66b997ddf7-fqzk5" Nov 03 13:26:19.178703 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:19.155747 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26bee2e8-835c-49d8-9e88-f959627e0ec9-credential-socket\") pod \"e2e-extmod-a-ef1693-openshift-default-66b997ddf7-fqzk5\" (UID: \"26bee2e8-835c-49d8-9e88-f959627e0ec9\") " pod="openshift-ingress/e2e-extmod-a-ef1693-openshift-default-66b997ddf7-fqzk5" Nov 03 13:26:19.178875 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:19.157835 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26bee2e8-835c-49d8-9e88-f959627e0ec9-istio-data\") pod \"e2e-extmod-a-ef1693-openshift-default-66b997ddf7-fqzk5\" (UID: \"26bee2e8-835c-49d8-9e88-f959627e0ec9\") " pod="openshift-ingress/e2e-extmod-a-ef1693-openshift-default-66b997ddf7-fqzk5" Nov 03 13:26:19.238324 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:19.237881 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26bee2e8-835c-49d8-9e88-f959627e0ec9-istio-podinfo\") pod \"e2e-extmod-a-ef1693-openshift-default-66b997ddf7-fqzk5\" (UID: \"26bee2e8-835c-49d8-9e88-f959627e0ec9\") " pod="openshift-ingress/e2e-extmod-a-ef1693-openshift-default-66b997ddf7-fqzk5" Nov 03 13:26:19.333318 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:19.328798 5024 patch_prober.go:29] interesting pod/e2e-extmod-b-7c192b-openshift-default-5c496b978f-8lgm7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body= Nov 03 13:26:19.333318 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:19.328853 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-8lgm7" podUID="c8657989-e2c5-4dfc-b32a-3491ad189395" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" Nov 03 13:26:19.333318 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:19.332865 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26bee2e8-835c-49d8-9e88-f959627e0ec9-istio-envoy\") pod \"e2e-extmod-a-ef1693-openshift-default-66b997ddf7-fqzk5\" (UID: \"26bee2e8-835c-49d8-9e88-f959627e0ec9\") " pod="openshift-ingress/e2e-extmod-a-ef1693-openshift-default-66b997ddf7-fqzk5" Nov 03 13:26:19.346687 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:19.346616 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26bee2e8-835c-49d8-9e88-f959627e0ec9-istio-token\") pod \"e2e-extmod-a-ef1693-openshift-default-66b997ddf7-fqzk5\" (UID: \"26bee2e8-835c-49d8-9e88-f959627e0ec9\") " pod="openshift-ingress/e2e-extmod-a-ef1693-openshift-default-66b997ddf7-fqzk5" Nov 03 13:26:19.347590 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:19.347565 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7x2lh\" (UniqueName: \"kubernetes.io/projected/26bee2e8-835c-49d8-9e88-f959627e0ec9-kube-api-access-7x2lh\") pod \"e2e-extmod-a-ef1693-openshift-default-66b997ddf7-fqzk5\" (UID: \"26bee2e8-835c-49d8-9e88-f959627e0ec9\") " pod="openshift-ingress/e2e-extmod-a-ef1693-openshift-default-66b997ddf7-fqzk5" Nov 03 13:26:19.518355 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:19.517504 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-ef1693-openshift-default-66b997ddf7-fqzk5" Nov 03 13:26:19.669900 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:19.669868 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-2dbf739c-gw-openshift-default-64b7d84c5c-kmkgg"] Nov 03 13:26:19.670269 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:19.670241 5024 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-2dbf739c-gw-openshift-default-64b7d84c5c-kmkgg" podUID="84095934-31b5-4b42-9495-5ac5286722c2" containerName="istio-proxy" containerID="cri-o://ce2914a031260070d7ab30eb79a765584ed94b6e54403d18756d3fc2cf53f2b0" gracePeriod=30 Nov 03 13:26:20.170559 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:20.170397 5024 patch_prober.go:29] interesting pod/e2e-extmod-b-7c192b-openshift-default-5c496b978f-8lgm7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body= Nov 03 13:26:20.170559 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:20.170466 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-8lgm7" podUID="c8657989-e2c5-4dfc-b32a-3491ad189395" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" Nov 03 13:26:20.477954 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:20.474160 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-ef1693-openshift-default-66b997ddf7-fqzk5"] Nov 03 13:26:20.490043 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:26:20.490005 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26bee2e8_835c_49d8_9e88_f959627e0ec9.slice/crio-ed42652c0866458c33a278a85dab6da65bb773b22d7d7599b7a60d21bdc783f4 WatchSource:0}: Error finding container ed42652c0866458c33a278a85dab6da65bb773b22d7d7599b7a60d21bdc783f4: Status 404 returned error can't find the container with id ed42652c0866458c33a278a85dab6da65bb773b22d7d7599b7a60d21bdc783f4 Nov 03 13:26:20.493898 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:20.493459 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:26:20.493898 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:20.493530 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:26:20.493898 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:20.493559 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:26:21.126809 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:21.125571 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-ef1693-openshift-default-66b997ddf7-fqzk5" event={"ID":"26bee2e8-835c-49d8-9e88-f959627e0ec9","Type":"ContainerStarted","Data":"ecb99fcca1ad99be501b687c96e52f3fceb74db263d52460b29f79fafd21d8c1"} Nov 03 13:26:21.132834 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:21.128639 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-ef1693-openshift-default-66b997ddf7-fqzk5" event={"ID":"26bee2e8-835c-49d8-9e88-f959627e0ec9","Type":"ContainerStarted","Data":"ed42652c0866458c33a278a85dab6da65bb773b22d7d7599b7a60d21bdc783f4"} Nov 03 13:26:21.242066 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:21.241697 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-8lgm7" Nov 03 13:26:21.246128 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:21.244473 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-8lgm7" Nov 03 13:26:21.287328 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:21.280189 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-8lgm7" Nov 03 13:26:21.351327 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:21.345783 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-ef1693-openshift-default-66b997ddf7-fqzk5" podStartSLOduration=3.345753386 podStartE2EDuration="3.345753386s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:26:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:26:21.177979913 +0000 UTC m=+1061.186171370" watchObservedRunningTime="2026-11-03 13:26:21.345753386 +0000 UTC m=+1061.353944663" Nov 03 13:26:21.521956 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:21.521821 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-ef1693-openshift-default-66b997ddf7-fqzk5" Nov 03 13:26:21.523808 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:21.523386 5024 patch_prober.go:29] interesting pod/e2e-extmod-a-ef1693-openshift-default-66b997ddf7-fqzk5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Nov 03 13:26:21.523808 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:21.523468 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-ef1693-openshift-default-66b997ddf7-fqzk5" podUID="26bee2e8-835c-49d8-9e88-f959627e0ec9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Nov 03 13:26:22.520606 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:22.520566 5024 patch_prober.go:29] interesting pod/e2e-extmod-a-ef1693-openshift-default-66b997ddf7-fqzk5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Nov 03 13:26:22.521075 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:22.520642 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-ef1693-openshift-default-66b997ddf7-fqzk5" podUID="26bee2e8-835c-49d8-9e88-f959627e0ec9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Nov 03 13:26:23.487487 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:23.487422 5024 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-2dbf739c-75f684fcd9-zw8cw" podUID="7dae3b52-27f3-42bb-98f1-46770039ab96" containerName="payload-processing" containerID="cri-o://4893976f07c7a369c8a84d120eaf90b9f5c7f49d88d8c65ac5c8e4883cc24df4" gracePeriod=25 Nov 03 13:26:23.521274 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:23.521039 5024 patch_prober.go:29] interesting pod/e2e-extmod-a-ef1693-openshift-default-66b997ddf7-fqzk5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Nov 03 13:26:23.521274 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:23.521107 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-ef1693-openshift-default-66b997ddf7-fqzk5" podUID="26bee2e8-835c-49d8-9e88-f959627e0ec9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Nov 03 13:26:24.118528 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:24.118435 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-2dbf739c-75f684fcd9-zw8cw" Nov 03 13:26:24.162051 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:24.162011 5024 generic.go:356] "Generic (PLEG): container finished" podID="7dae3b52-27f3-42bb-98f1-46770039ab96" containerID="4893976f07c7a369c8a84d120eaf90b9f5c7f49d88d8c65ac5c8e4883cc24df4" exitCode=0 Nov 03 13:26:24.162248 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:24.162087 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-2dbf739c-75f684fcd9-zw8cw" event={"ID":"7dae3b52-27f3-42bb-98f1-46770039ab96","Type":"ContainerDied","Data":"4893976f07c7a369c8a84d120eaf90b9f5c7f49d88d8c65ac5c8e4883cc24df4"} Nov 03 13:26:24.162248 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:24.162127 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-2dbf739c-75f684fcd9-zw8cw" event={"ID":"7dae3b52-27f3-42bb-98f1-46770039ab96","Type":"ContainerDied","Data":"0c662cd5ac1cf8ac9c414e0cd7481dada401316ddf91fc0e958cf11e5dee396d"} Nov 03 13:26:24.162248 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:24.162151 5024 scope.go:122] "RemoveContainer" containerID="4893976f07c7a369c8a84d120eaf90b9f5c7f49d88d8c65ac5c8e4883cc24df4" Nov 03 13:26:24.162464 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:24.162414 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-2dbf739c-75f684fcd9-zw8cw" Nov 03 13:26:24.222892 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:24.222863 5024 scope.go:122] "RemoveContainer" containerID="4893976f07c7a369c8a84d120eaf90b9f5c7f49d88d8c65ac5c8e4883cc24df4" Nov 03 13:26:24.223394 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:26:24.223364 5024 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4893976f07c7a369c8a84d120eaf90b9f5c7f49d88d8c65ac5c8e4883cc24df4\": container with ID starting with 4893976f07c7a369c8a84d120eaf90b9f5c7f49d88d8c65ac5c8e4883cc24df4 not found: ID does not exist" containerID="4893976f07c7a369c8a84d120eaf90b9f5c7f49d88d8c65ac5c8e4883cc24df4" Nov 03 13:26:24.223556 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:24.223530 5024 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4893976f07c7a369c8a84d120eaf90b9f5c7f49d88d8c65ac5c8e4883cc24df4"} err="failed to get container status \"4893976f07c7a369c8a84d120eaf90b9f5c7f49d88d8c65ac5c8e4883cc24df4\": rpc error: code = NotFound desc = could not find container \"4893976f07c7a369c8a84d120eaf90b9f5c7f49d88d8c65ac5c8e4883cc24df4\": container with ID starting with 4893976f07c7a369c8a84d120eaf90b9f5c7f49d88d8c65ac5c8e4883cc24df4 not found: ID does not exist" Nov 03 13:26:24.224877 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:24.224849 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7dae3b52-27f3-42bb-98f1-46770039ab96-kube-api-access-4l586\" (UniqueName: \"kubernetes.io/projected/7dae3b52-27f3-42bb-98f1-46770039ab96-kube-api-access-4l586\") pod \"7dae3b52-27f3-42bb-98f1-46770039ab96\" (UID: \"7dae3b52-27f3-42bb-98f1-46770039ab96\") " Nov 03 13:26:24.225174 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:24.225156 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7dae3b52-27f3-42bb-98f1-46770039ab96-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7dae3b52-27f3-42bb-98f1-46770039ab96-plugins-config-volume\") pod \"7dae3b52-27f3-42bb-98f1-46770039ab96\" (UID: \"7dae3b52-27f3-42bb-98f1-46770039ab96\") " Nov 03 13:26:24.226303 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:24.226266 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7dae3b52-27f3-42bb-98f1-46770039ab96-plugins-config-volume" pod "7dae3b52-27f3-42bb-98f1-46770039ab96" (UID: "7dae3b52-27f3-42bb-98f1-46770039ab96"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:26:24.232922 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:24.232885 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7dae3b52-27f3-42bb-98f1-46770039ab96-kube-api-access-4l586" pod "7dae3b52-27f3-42bb-98f1-46770039ab96" (UID: "7dae3b52-27f3-42bb-98f1-46770039ab96"). InnerVolumeSpecName "kube-api-access-4l586". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:26:24.328168 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:24.328119 5024 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7dae3b52-27f3-42bb-98f1-46770039ab96-plugins-config-volume\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:26:24.328168 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:24.328157 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4l586\" (UniqueName: \"kubernetes.io/projected/7dae3b52-27f3-42bb-98f1-46770039ab96-kube-api-access-4l586\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:26:24.521226 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:24.521186 5024 patch_prober.go:29] interesting pod/e2e-extmod-a-ef1693-openshift-default-66b997ddf7-fqzk5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Nov 03 13:26:24.521383 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:24.521255 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-ef1693-openshift-default-66b997ddf7-fqzk5" podUID="26bee2e8-835c-49d8-9e88-f959627e0ec9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Nov 03 13:26:24.534498 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:24.534461 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-2dbf739c-75f684fcd9-zw8cw"] Nov 03 13:26:24.543179 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:24.543143 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-2dbf739c-75f684fcd9-zw8cw"] Nov 03 13:26:25.521299 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:25.521251 5024 patch_prober.go:29] interesting pod/e2e-extmod-a-ef1693-openshift-default-66b997ddf7-fqzk5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Nov 03 13:26:25.521525 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:25.521325 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-ef1693-openshift-default-66b997ddf7-fqzk5" podUID="26bee2e8-835c-49d8-9e88-f959627e0ec9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Nov 03 13:26:25.969308 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:25.968716 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w0-9add72/auth-test-model-9add72-kserve-54f88d5d5-bm2sv"] Nov 03 13:26:25.973307 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:25.970032 5024 cpu_manager.go:414] "containerMap: removing container" podUID="7dae3b52-27f3-42bb-98f1-46770039ab96" containerName="payload-processing" Nov 03 13:26:25.973307 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:25.970060 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7dae3b52-27f3-42bb-98f1-46770039ab96" containerName="payload-processing" Nov 03 13:26:25.973307 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:25.970237 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="7dae3b52-27f3-42bb-98f1-46770039ab96" containerName="payload-processing" Nov 03 13:26:26.037886 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:26.037840 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w0-9add72/auth-test-model-9add72-kserve-54f88d5d5-bm2sv"] Nov 03 13:26:26.038096 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:26.037898 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-9add72/auth-test-model-9add72-kserve-54f88d5d5-bm2sv" Nov 03 13:26:26.046823 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:26.046460 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w0-9add72\"/\"kube-root-ca.crt\"" Nov 03 13:26:26.048948 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:26.048270 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w0-9add72\"/\"openshift-service-ca.crt\"" Nov 03 13:26:26.048948 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:26.048531 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w0-9add72\"/\"auth-test-model-9add72-kserve-self-signed-certs\"" Nov 03 13:26:26.048948 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:26.048656 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w0-9add72\"/\"default-dockercfg-4k9xb\"" Nov 03 13:26:26.064315 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:26.064248 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/25181c16-f5c3-49bf-aec6-6c6768d87af9-model-cache\") pod \"auth-test-model-9add72-kserve-54f88d5d5-bm2sv\" (UID: \"25181c16-f5c3-49bf-aec6-6c6768d87af9\") " pod="ai-tenant-e2e-shared-b-w0-9add72/auth-test-model-9add72-kserve-54f88d5d5-bm2sv" Nov 03 13:26:26.064478 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:26.064329 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/25181c16-f5c3-49bf-aec6-6c6768d87af9-home\") pod \"auth-test-model-9add72-kserve-54f88d5d5-bm2sv\" (UID: \"25181c16-f5c3-49bf-aec6-6c6768d87af9\") " pod="ai-tenant-e2e-shared-b-w0-9add72/auth-test-model-9add72-kserve-54f88d5d5-bm2sv" Nov 03 13:26:26.064478 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:26.064372 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/25181c16-f5c3-49bf-aec6-6c6768d87af9-dshm\") pod \"auth-test-model-9add72-kserve-54f88d5d5-bm2sv\" (UID: \"25181c16-f5c3-49bf-aec6-6c6768d87af9\") " pod="ai-tenant-e2e-shared-b-w0-9add72/auth-test-model-9add72-kserve-54f88d5d5-bm2sv" Nov 03 13:26:26.064575 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:26.064488 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/25181c16-f5c3-49bf-aec6-6c6768d87af9-tmp-dir\") pod \"auth-test-model-9add72-kserve-54f88d5d5-bm2sv\" (UID: \"25181c16-f5c3-49bf-aec6-6c6768d87af9\") " pod="ai-tenant-e2e-shared-b-w0-9add72/auth-test-model-9add72-kserve-54f88d5d5-bm2sv" Nov 03 13:26:26.064612 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:26.064584 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ln8ln\" (UniqueName: \"kubernetes.io/projected/25181c16-f5c3-49bf-aec6-6c6768d87af9-kube-api-access-ln8ln\") pod \"auth-test-model-9add72-kserve-54f88d5d5-bm2sv\" (UID: \"25181c16-f5c3-49bf-aec6-6c6768d87af9\") " pod="ai-tenant-e2e-shared-b-w0-9add72/auth-test-model-9add72-kserve-54f88d5d5-bm2sv" Nov 03 13:26:26.064659 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:26.064616 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/25181c16-f5c3-49bf-aec6-6c6768d87af9-tls-certs\") pod \"auth-test-model-9add72-kserve-54f88d5d5-bm2sv\" (UID: \"25181c16-f5c3-49bf-aec6-6c6768d87af9\") " pod="ai-tenant-e2e-shared-b-w0-9add72/auth-test-model-9add72-kserve-54f88d5d5-bm2sv" Nov 03 13:26:26.120466 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:26.117271 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7dae3b52-27f3-42bb-98f1-46770039ab96" path="/var/lib/kubelet/pods/7dae3b52-27f3-42bb-98f1-46770039ab96/volumes" Nov 03 13:26:26.166780 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:26.166737 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/25181c16-f5c3-49bf-aec6-6c6768d87af9-model-cache\") pod \"auth-test-model-9add72-kserve-54f88d5d5-bm2sv\" (UID: \"25181c16-f5c3-49bf-aec6-6c6768d87af9\") " pod="ai-tenant-e2e-shared-b-w0-9add72/auth-test-model-9add72-kserve-54f88d5d5-bm2sv" Nov 03 13:26:26.167307 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:26.167240 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/25181c16-f5c3-49bf-aec6-6c6768d87af9-home\") pod \"auth-test-model-9add72-kserve-54f88d5d5-bm2sv\" (UID: \"25181c16-f5c3-49bf-aec6-6c6768d87af9\") " pod="ai-tenant-e2e-shared-b-w0-9add72/auth-test-model-9add72-kserve-54f88d5d5-bm2sv" Nov 03 13:26:26.167307 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:26.167164 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/25181c16-f5c3-49bf-aec6-6c6768d87af9-model-cache\") pod \"auth-test-model-9add72-kserve-54f88d5d5-bm2sv\" (UID: \"25181c16-f5c3-49bf-aec6-6c6768d87af9\") " pod="ai-tenant-e2e-shared-b-w0-9add72/auth-test-model-9add72-kserve-54f88d5d5-bm2sv" Nov 03 13:26:26.170158 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:26.167861 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/25181c16-f5c3-49bf-aec6-6c6768d87af9-home\") pod \"auth-test-model-9add72-kserve-54f88d5d5-bm2sv\" (UID: \"25181c16-f5c3-49bf-aec6-6c6768d87af9\") " pod="ai-tenant-e2e-shared-b-w0-9add72/auth-test-model-9add72-kserve-54f88d5d5-bm2sv" Nov 03 13:26:26.170158 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:26.167934 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/25181c16-f5c3-49bf-aec6-6c6768d87af9-dshm\") pod \"auth-test-model-9add72-kserve-54f88d5d5-bm2sv\" (UID: \"25181c16-f5c3-49bf-aec6-6c6768d87af9\") " pod="ai-tenant-e2e-shared-b-w0-9add72/auth-test-model-9add72-kserve-54f88d5d5-bm2sv" Nov 03 13:26:26.170158 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:26.168046 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/25181c16-f5c3-49bf-aec6-6c6768d87af9-tmp-dir\") pod \"auth-test-model-9add72-kserve-54f88d5d5-bm2sv\" (UID: \"25181c16-f5c3-49bf-aec6-6c6768d87af9\") " pod="ai-tenant-e2e-shared-b-w0-9add72/auth-test-model-9add72-kserve-54f88d5d5-bm2sv" Nov 03 13:26:26.170158 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:26.168172 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ln8ln\" (UniqueName: \"kubernetes.io/projected/25181c16-f5c3-49bf-aec6-6c6768d87af9-kube-api-access-ln8ln\") pod \"auth-test-model-9add72-kserve-54f88d5d5-bm2sv\" (UID: \"25181c16-f5c3-49bf-aec6-6c6768d87af9\") " pod="ai-tenant-e2e-shared-b-w0-9add72/auth-test-model-9add72-kserve-54f88d5d5-bm2sv" Nov 03 13:26:26.170158 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:26.168202 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/25181c16-f5c3-49bf-aec6-6c6768d87af9-tls-certs\") pod \"auth-test-model-9add72-kserve-54f88d5d5-bm2sv\" (UID: \"25181c16-f5c3-49bf-aec6-6c6768d87af9\") " pod="ai-tenant-e2e-shared-b-w0-9add72/auth-test-model-9add72-kserve-54f88d5d5-bm2sv" Nov 03 13:26:26.170158 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:26.169473 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/25181c16-f5c3-49bf-aec6-6c6768d87af9-tmp-dir\") pod \"auth-test-model-9add72-kserve-54f88d5d5-bm2sv\" (UID: \"25181c16-f5c3-49bf-aec6-6c6768d87af9\") " pod="ai-tenant-e2e-shared-b-w0-9add72/auth-test-model-9add72-kserve-54f88d5d5-bm2sv" Nov 03 13:26:26.235685 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:26.232669 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/25181c16-f5c3-49bf-aec6-6c6768d87af9-tls-certs\") pod \"auth-test-model-9add72-kserve-54f88d5d5-bm2sv\" (UID: \"25181c16-f5c3-49bf-aec6-6c6768d87af9\") " pod="ai-tenant-e2e-shared-b-w0-9add72/auth-test-model-9add72-kserve-54f88d5d5-bm2sv" Nov 03 13:26:26.235685 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:26.232989 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/25181c16-f5c3-49bf-aec6-6c6768d87af9-dshm\") pod \"auth-test-model-9add72-kserve-54f88d5d5-bm2sv\" (UID: \"25181c16-f5c3-49bf-aec6-6c6768d87af9\") " pod="ai-tenant-e2e-shared-b-w0-9add72/auth-test-model-9add72-kserve-54f88d5d5-bm2sv" Nov 03 13:26:26.264322 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:26.264235 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ln8ln\" (UniqueName: \"kubernetes.io/projected/25181c16-f5c3-49bf-aec6-6c6768d87af9-kube-api-access-ln8ln\") pod \"auth-test-model-9add72-kserve-54f88d5d5-bm2sv\" (UID: \"25181c16-f5c3-49bf-aec6-6c6768d87af9\") " pod="ai-tenant-e2e-shared-b-w0-9add72/auth-test-model-9add72-kserve-54f88d5d5-bm2sv" Nov 03 13:26:26.391399 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:26.391360 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-9add72/auth-test-model-9add72-kserve-54f88d5d5-bm2sv" Nov 03 13:26:26.541168 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:26.533102 5024 patch_prober.go:29] interesting pod/e2e-extmod-a-ef1693-openshift-default-66b997ddf7-fqzk5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Nov 03 13:26:26.541168 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:26.533179 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-ef1693-openshift-default-66b997ddf7-fqzk5" podUID="26bee2e8-835c-49d8-9e88-f959627e0ec9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Nov 03 13:26:26.966415 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:26.966377 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w0-9add72/auth-test-model-9add72-kserve-54f88d5d5-bm2sv"] Nov 03 13:26:27.356798 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:27.356749 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-9add72/auth-test-model-9add72-kserve-54f88d5d5-bm2sv" event={"ID":"25181c16-f5c3-49bf-aec6-6c6768d87af9","Type":"ContainerStarted","Data":"5aca63baf039a960ae4a1d7a0d74b07c12c101d35216cc0176fe47afc31db294"} Nov 03 13:26:27.356798 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:27.356799 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-9add72/auth-test-model-9add72-kserve-54f88d5d5-bm2sv" event={"ID":"25181c16-f5c3-49bf-aec6-6c6768d87af9","Type":"ContainerStarted","Data":"a63bec35070320a9d12649f898dedf2701b5e25060c3fb1960efbd1ca290f111"} Nov 03 13:26:27.403313 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:27.402072 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w0-9add72/auth-test-model-9add72-kserve-54f88d5d5-bm2sv" Nov 03 13:26:27.414328 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:27.411025 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w0-9add72/auth-test-model-9add72-kserve-54f88d5d5-bm2sv" podStartSLOduration=2.411001409 podStartE2EDuration="2.411001409s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:26:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:26:27.396561565 +0000 UTC m=+1067.404752841" watchObservedRunningTime="2026-11-03 13:26:27.411001409 +0000 UTC m=+1067.419192689" Nov 03 13:26:27.521500 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:27.521462 5024 patch_prober.go:29] interesting pod/e2e-extmod-a-ef1693-openshift-default-66b997ddf7-fqzk5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Nov 03 13:26:27.521877 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:27.521837 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-ef1693-openshift-default-66b997ddf7-fqzk5" podUID="26bee2e8-835c-49d8-9e88-f959627e0ec9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Nov 03 13:26:28.046051 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:28.045690 5024 patch_prober.go:29] interesting pod/e2e-migrate-2dbf739c-gw-openshift-default-64b7d84c5c-kmkgg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 13:26:28.046051 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:28.045760 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-2dbf739c-gw-openshift-default-64b7d84c5c-kmkgg" podUID="84095934-31b5-4b42-9495-5ac5286722c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 13:26:28.138764 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:28.137955 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-g94p9"] Nov 03 13:26:28.176137 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:28.176092 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-g94p9"] Nov 03 13:26:28.176358 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:28.176262 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-g94p9" Nov 03 13:26:28.265969 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:28.265934 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bef13398-28a1-4064-aa3b-ea0f3ebf64fd-credential-socket\") pod \"e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-g94p9\" (UID: \"bef13398-28a1-4064-aa3b-ea0f3ebf64fd\") " pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-g94p9" Nov 03 13:26:28.266155 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:28.265995 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bef13398-28a1-4064-aa3b-ea0f3ebf64fd-workload-certs\") pod \"e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-g94p9\" (UID: \"bef13398-28a1-4064-aa3b-ea0f3ebf64fd\") " pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-g94p9" Nov 03 13:26:28.266155 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:28.266022 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8bbxg\" (UniqueName: \"kubernetes.io/projected/bef13398-28a1-4064-aa3b-ea0f3ebf64fd-kube-api-access-8bbxg\") pod \"e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-g94p9\" (UID: \"bef13398-28a1-4064-aa3b-ea0f3ebf64fd\") " pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-g94p9" Nov 03 13:26:28.266272 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:28.266223 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bef13398-28a1-4064-aa3b-ea0f3ebf64fd-istio-podinfo\") pod \"e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-g94p9\" (UID: \"bef13398-28a1-4064-aa3b-ea0f3ebf64fd\") " pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-g94p9" Nov 03 13:26:28.266419 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:28.266397 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bef13398-28a1-4064-aa3b-ea0f3ebf64fd-workload-socket\") pod \"e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-g94p9\" (UID: \"bef13398-28a1-4064-aa3b-ea0f3ebf64fd\") " pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-g94p9" Nov 03 13:26:28.266481 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:28.266440 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bef13398-28a1-4064-aa3b-ea0f3ebf64fd-istiod-ca-cert\") pod \"e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-g94p9\" (UID: \"bef13398-28a1-4064-aa3b-ea0f3ebf64fd\") " pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-g94p9" Nov 03 13:26:28.266680 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:28.266654 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bef13398-28a1-4064-aa3b-ea0f3ebf64fd-istio-envoy\") pod \"e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-g94p9\" (UID: \"bef13398-28a1-4064-aa3b-ea0f3ebf64fd\") " pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-g94p9" Nov 03 13:26:28.266727 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:28.266702 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bef13398-28a1-4064-aa3b-ea0f3ebf64fd-istio-data\") pod \"e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-g94p9\" (UID: \"bef13398-28a1-4064-aa3b-ea0f3ebf64fd\") " pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-g94p9" Nov 03 13:26:28.266831 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:28.266811 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bef13398-28a1-4064-aa3b-ea0f3ebf64fd-istio-token\") pod \"e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-g94p9\" (UID: \"bef13398-28a1-4064-aa3b-ea0f3ebf64fd\") " pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-g94p9" Nov 03 13:26:28.368278 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:28.368244 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bef13398-28a1-4064-aa3b-ea0f3ebf64fd-istio-envoy\") pod \"e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-g94p9\" (UID: \"bef13398-28a1-4064-aa3b-ea0f3ebf64fd\") " pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-g94p9" Nov 03 13:26:28.368278 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:28.368303 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bef13398-28a1-4064-aa3b-ea0f3ebf64fd-istio-data\") pod \"e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-g94p9\" (UID: \"bef13398-28a1-4064-aa3b-ea0f3ebf64fd\") " pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-g94p9" Nov 03 13:26:28.737537 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:28.368348 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bef13398-28a1-4064-aa3b-ea0f3ebf64fd-istio-token\") pod \"e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-g94p9\" (UID: \"bef13398-28a1-4064-aa3b-ea0f3ebf64fd\") " pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-g94p9" Nov 03 13:26:28.737537 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:28.368420 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bef13398-28a1-4064-aa3b-ea0f3ebf64fd-credential-socket\") pod \"e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-g94p9\" (UID: \"bef13398-28a1-4064-aa3b-ea0f3ebf64fd\") " pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-g94p9" Nov 03 13:26:28.737537 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:28.368464 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bef13398-28a1-4064-aa3b-ea0f3ebf64fd-workload-certs\") pod \"e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-g94p9\" (UID: \"bef13398-28a1-4064-aa3b-ea0f3ebf64fd\") " pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-g94p9" Nov 03 13:26:28.737537 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:28.368625 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8bbxg\" (UniqueName: \"kubernetes.io/projected/bef13398-28a1-4064-aa3b-ea0f3ebf64fd-kube-api-access-8bbxg\") pod \"e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-g94p9\" (UID: \"bef13398-28a1-4064-aa3b-ea0f3ebf64fd\") " pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-g94p9" Nov 03 13:26:28.737537 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:28.368718 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bef13398-28a1-4064-aa3b-ea0f3ebf64fd-istio-podinfo\") pod \"e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-g94p9\" (UID: \"bef13398-28a1-4064-aa3b-ea0f3ebf64fd\") " pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-g94p9" Nov 03 13:26:28.737537 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:28.368754 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bef13398-28a1-4064-aa3b-ea0f3ebf64fd-istio-data\") pod \"e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-g94p9\" (UID: \"bef13398-28a1-4064-aa3b-ea0f3ebf64fd\") " pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-g94p9" Nov 03 13:26:28.737537 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:28.368808 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bef13398-28a1-4064-aa3b-ea0f3ebf64fd-workload-certs\") pod \"e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-g94p9\" (UID: \"bef13398-28a1-4064-aa3b-ea0f3ebf64fd\") " pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-g94p9" Nov 03 13:26:28.737537 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:28.368856 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bef13398-28a1-4064-aa3b-ea0f3ebf64fd-credential-socket\") pod \"e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-g94p9\" (UID: \"bef13398-28a1-4064-aa3b-ea0f3ebf64fd\") " pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-g94p9" Nov 03 13:26:28.737537 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:28.368887 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bef13398-28a1-4064-aa3b-ea0f3ebf64fd-workload-socket\") pod \"e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-g94p9\" (UID: \"bef13398-28a1-4064-aa3b-ea0f3ebf64fd\") " pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-g94p9" Nov 03 13:26:28.737537 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:28.368922 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bef13398-28a1-4064-aa3b-ea0f3ebf64fd-istiod-ca-cert\") pod \"e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-g94p9\" (UID: \"bef13398-28a1-4064-aa3b-ea0f3ebf64fd\") " pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-g94p9" Nov 03 13:26:28.737537 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:28.369202 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bef13398-28a1-4064-aa3b-ea0f3ebf64fd-workload-socket\") pod \"e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-g94p9\" (UID: \"bef13398-28a1-4064-aa3b-ea0f3ebf64fd\") " pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-g94p9" Nov 03 13:26:28.737537 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:28.369683 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bef13398-28a1-4064-aa3b-ea0f3ebf64fd-istiod-ca-cert\") pod \"e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-g94p9\" (UID: \"bef13398-28a1-4064-aa3b-ea0f3ebf64fd\") " pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-g94p9" Nov 03 13:26:28.737537 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:28.371184 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bef13398-28a1-4064-aa3b-ea0f3ebf64fd-istio-envoy\") pod \"e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-g94p9\" (UID: \"bef13398-28a1-4064-aa3b-ea0f3ebf64fd\") " pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-g94p9" Nov 03 13:26:28.737537 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:28.371623 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bef13398-28a1-4064-aa3b-ea0f3ebf64fd-istio-podinfo\") pod \"e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-g94p9\" (UID: \"bef13398-28a1-4064-aa3b-ea0f3ebf64fd\") " pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-g94p9" Nov 03 13:26:28.737537 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:28.378154 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8bbxg\" (UniqueName: \"kubernetes.io/projected/bef13398-28a1-4064-aa3b-ea0f3ebf64fd-kube-api-access-8bbxg\") pod \"e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-g94p9\" (UID: \"bef13398-28a1-4064-aa3b-ea0f3ebf64fd\") " pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-g94p9" Nov 03 13:26:28.737537 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:28.378612 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bef13398-28a1-4064-aa3b-ea0f3ebf64fd-istio-token\") pod \"e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-g94p9\" (UID: \"bef13398-28a1-4064-aa3b-ea0f3ebf64fd\") " pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-g94p9" Nov 03 13:26:28.737537 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:28.519452 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-g94p9" Nov 03 13:26:28.737537 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:28.525108 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-ef1693-openshift-default-66b997ddf7-fqzk5" Nov 03 13:26:28.737537 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:28.525617 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-ef1693-openshift-default-66b997ddf7-fqzk5" Nov 03 13:26:28.737537 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:28.526461 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-ef1693-openshift-default-66b997ddf7-fqzk5" Nov 03 13:26:28.852883 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:28.852819 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-g94p9"] Nov 03 13:26:28.899142 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:28.899099 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:26:28.899257 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:28.899195 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:26:28.899357 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:28.899262 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:26:29.358762 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:29.358733 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-2dbf739c-gw-openshift-default-64b7d84c5c-kmkgg" Nov 03 13:26:29.375054 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:29.375020 5024 generic.go:356] "Generic (PLEG): container finished" podID="84095934-31b5-4b42-9495-5ac5286722c2" containerID="ce2914a031260070d7ab30eb79a765584ed94b6e54403d18756d3fc2cf53f2b0" exitCode=0 Nov 03 13:26:29.375400 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:29.375093 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-2dbf739c-gw-openshift-default-64b7d84c5c-kmkgg" Nov 03 13:26:29.375555 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:29.375115 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-2dbf739c-gw-openshift-default-64b7d84c5c-kmkgg" event={"ID":"84095934-31b5-4b42-9495-5ac5286722c2","Type":"ContainerDied","Data":"ce2914a031260070d7ab30eb79a765584ed94b6e54403d18756d3fc2cf53f2b0"} Nov 03 13:26:29.375626 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:29.375580 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-2dbf739c-gw-openshift-default-64b7d84c5c-kmkgg" event={"ID":"84095934-31b5-4b42-9495-5ac5286722c2","Type":"ContainerDied","Data":"b08f55129f58bfb11be7eacbf0cbb45498626a928a825655663ab41b625c7961"} Nov 03 13:26:29.375626 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:29.375615 5024 scope.go:122] "RemoveContainer" containerID="ce2914a031260070d7ab30eb79a765584ed94b6e54403d18756d3fc2cf53f2b0" Nov 03 13:26:29.408672 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:29.408277 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-g94p9" event={"ID":"bef13398-28a1-4064-aa3b-ea0f3ebf64fd","Type":"ContainerStarted","Data":"2c088d3e468590c47b8bbc3776aa33503b0baa5b9f400d89e231f0781cac3e16"} Nov 03 13:26:29.408672 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:29.408368 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-g94p9" event={"ID":"bef13398-28a1-4064-aa3b-ea0f3ebf64fd","Type":"ContainerStarted","Data":"dcf091dd73cb6ec98ee2ad274d97a9464cf67046727af464ad6413f53ce9bf78"} Nov 03 13:26:29.435310 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:29.431603 5024 generic.go:356] "Generic (PLEG): container finished" podID="7d77cd17-8ce7-4fe7-9b61-7e4c7cdc720b" containerID="d7b5b985f79627cac6c18273d4df80d60d9c163fd8078d539b655cd26becda5a" exitCode=0 Nov 03 13:26:29.435310 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:29.432130 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-6z9tv" event={"ID":"7d77cd17-8ce7-4fe7-9b61-7e4c7cdc720b","Type":"ContainerDied","Data":"d7b5b985f79627cac6c18273d4df80d60d9c163fd8078d539b655cd26becda5a"} Nov 03 13:26:29.435310 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:29.432586 5024 scope.go:122] "RemoveContainer" containerID="d7b5b985f79627cac6c18273d4df80d60d9c163fd8078d539b655cd26becda5a" Nov 03 13:26:29.435310 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:29.433161 5024 scope.go:122] "RemoveContainer" containerID="ce2914a031260070d7ab30eb79a765584ed94b6e54403d18756d3fc2cf53f2b0" Nov 03 13:26:29.435310 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:29.434691 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:26:29.435310 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:29.434758 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:26:29.435310 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:29.434792 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:26:29.435973 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:26:29.435459 5024 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce2914a031260070d7ab30eb79a765584ed94b6e54403d18756d3fc2cf53f2b0\": container with ID starting with ce2914a031260070d7ab30eb79a765584ed94b6e54403d18756d3fc2cf53f2b0 not found: ID does not exist" containerID="ce2914a031260070d7ab30eb79a765584ed94b6e54403d18756d3fc2cf53f2b0" Nov 03 13:26:29.435973 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:29.435490 5024 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce2914a031260070d7ab30eb79a765584ed94b6e54403d18756d3fc2cf53f2b0"} err="failed to get container status \"ce2914a031260070d7ab30eb79a765584ed94b6e54403d18756d3fc2cf53f2b0\": rpc error: code = NotFound desc = could not find container \"ce2914a031260070d7ab30eb79a765584ed94b6e54403d18756d3fc2cf53f2b0\": container with ID starting with ce2914a031260070d7ab30eb79a765584ed94b6e54403d18756d3fc2cf53f2b0 not found: ID does not exist" Nov 03 13:26:29.444656 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:29.444622 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/84095934-31b5-4b42-9495-5ac5286722c2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84095934-31b5-4b42-9495-5ac5286722c2-istio-podinfo\") pod \"84095934-31b5-4b42-9495-5ac5286722c2\" (UID: \"84095934-31b5-4b42-9495-5ac5286722c2\") " Nov 03 13:26:29.444829 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:29.444680 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84095934-31b5-4b42-9495-5ac5286722c2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84095934-31b5-4b42-9495-5ac5286722c2-istiod-ca-cert\") pod \"84095934-31b5-4b42-9495-5ac5286722c2\" (UID: \"84095934-31b5-4b42-9495-5ac5286722c2\") " Nov 03 13:26:29.444829 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:29.444734 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84095934-31b5-4b42-9495-5ac5286722c2-istio-token\" (UniqueName: \"kubernetes.io/projected/84095934-31b5-4b42-9495-5ac5286722c2-istio-token\") pod \"84095934-31b5-4b42-9495-5ac5286722c2\" (UID: \"84095934-31b5-4b42-9495-5ac5286722c2\") " Nov 03 13:26:29.445047 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:29.444843 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84095934-31b5-4b42-9495-5ac5286722c2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84095934-31b5-4b42-9495-5ac5286722c2-workload-socket\") pod \"84095934-31b5-4b42-9495-5ac5286722c2\" (UID: \"84095934-31b5-4b42-9495-5ac5286722c2\") " Nov 03 13:26:29.445047 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:29.444984 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84095934-31b5-4b42-9495-5ac5286722c2-kube-api-access-jzjgh\" (UniqueName: \"kubernetes.io/projected/84095934-31b5-4b42-9495-5ac5286722c2-kube-api-access-jzjgh\") pod \"84095934-31b5-4b42-9495-5ac5286722c2\" (UID: \"84095934-31b5-4b42-9495-5ac5286722c2\") " Nov 03 13:26:29.445047 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:29.445036 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84095934-31b5-4b42-9495-5ac5286722c2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84095934-31b5-4b42-9495-5ac5286722c2-credential-socket\") pod \"84095934-31b5-4b42-9495-5ac5286722c2\" (UID: \"84095934-31b5-4b42-9495-5ac5286722c2\") " Nov 03 13:26:29.445206 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:29.445083 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84095934-31b5-4b42-9495-5ac5286722c2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84095934-31b5-4b42-9495-5ac5286722c2-istio-envoy\") pod \"84095934-31b5-4b42-9495-5ac5286722c2\" (UID: \"84095934-31b5-4b42-9495-5ac5286722c2\") " Nov 03 13:26:29.445206 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:29.445192 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84095934-31b5-4b42-9495-5ac5286722c2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84095934-31b5-4b42-9495-5ac5286722c2-istio-data\") pod \"84095934-31b5-4b42-9495-5ac5286722c2\" (UID: \"84095934-31b5-4b42-9495-5ac5286722c2\") " Nov 03 13:26:29.445358 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:29.445333 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84095934-31b5-4b42-9495-5ac5286722c2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84095934-31b5-4b42-9495-5ac5286722c2-workload-certs\") pod \"84095934-31b5-4b42-9495-5ac5286722c2\" (UID: \"84095934-31b5-4b42-9495-5ac5286722c2\") " Nov 03 13:26:29.446254 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:29.446224 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84095934-31b5-4b42-9495-5ac5286722c2-workload-certs" pod "84095934-31b5-4b42-9495-5ac5286722c2" (UID: "84095934-31b5-4b42-9495-5ac5286722c2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:26:29.446512 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:29.446490 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84095934-31b5-4b42-9495-5ac5286722c2-credential-socket" pod "84095934-31b5-4b42-9495-5ac5286722c2" (UID: "84095934-31b5-4b42-9495-5ac5286722c2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:26:29.446670 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:29.446645 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84095934-31b5-4b42-9495-5ac5286722c2-workload-socket" pod "84095934-31b5-4b42-9495-5ac5286722c2" (UID: "84095934-31b5-4b42-9495-5ac5286722c2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:26:29.446986 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:29.446958 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84095934-31b5-4b42-9495-5ac5286722c2-istio-data" pod "84095934-31b5-4b42-9495-5ac5286722c2" (UID: "84095934-31b5-4b42-9495-5ac5286722c2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:26:29.447142 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:29.447124 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84095934-31b5-4b42-9495-5ac5286722c2-istiod-ca-cert" pod "84095934-31b5-4b42-9495-5ac5286722c2" (UID: "84095934-31b5-4b42-9495-5ac5286722c2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:26:29.447955 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:29.447926 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84095934-31b5-4b42-9495-5ac5286722c2-kube-api-access-jzjgh" pod "84095934-31b5-4b42-9495-5ac5286722c2" (UID: "84095934-31b5-4b42-9495-5ac5286722c2"). InnerVolumeSpecName "kube-api-access-jzjgh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:26:29.449102 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:29.449079 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84095934-31b5-4b42-9495-5ac5286722c2-istio-envoy" pod "84095934-31b5-4b42-9495-5ac5286722c2" (UID: "84095934-31b5-4b42-9495-5ac5286722c2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:26:29.449472 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:29.449443 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84095934-31b5-4b42-9495-5ac5286722c2-istio-token" pod "84095934-31b5-4b42-9495-5ac5286722c2" (UID: "84095934-31b5-4b42-9495-5ac5286722c2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:26:29.451436 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:29.451408 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/84095934-31b5-4b42-9495-5ac5286722c2-istio-podinfo" pod "84095934-31b5-4b42-9495-5ac5286722c2" (UID: "84095934-31b5-4b42-9495-5ac5286722c2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 13:26:29.453132 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:29.453076 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-g94p9" podStartSLOduration=1.453057103 podStartE2EDuration="1.453057103s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:26:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:26:29.445823555 +0000 UTC m=+1069.454014843" watchObservedRunningTime="2026-11-03 13:26:29.453057103 +0000 UTC m=+1069.461248379" Nov 03 13:26:29.520612 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:29.520573 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-g94p9" Nov 03 13:26:29.550467 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:29.550426 5024 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84095934-31b5-4b42-9495-5ac5286722c2-workload-certs\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:26:29.550566 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:29.550469 5024 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84095934-31b5-4b42-9495-5ac5286722c2-istio-podinfo\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:26:29.550566 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:29.550484 5024 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84095934-31b5-4b42-9495-5ac5286722c2-istiod-ca-cert\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:26:29.550566 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:29.550498 5024 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84095934-31b5-4b42-9495-5ac5286722c2-istio-token\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:26:29.550566 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:29.550514 5024 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84095934-31b5-4b42-9495-5ac5286722c2-workload-socket\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:26:29.550566 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:29.550528 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jzjgh\" (UniqueName: \"kubernetes.io/projected/84095934-31b5-4b42-9495-5ac5286722c2-kube-api-access-jzjgh\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:26:29.550566 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:29.550544 5024 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84095934-31b5-4b42-9495-5ac5286722c2-credential-socket\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:26:29.550566 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:29.550557 5024 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84095934-31b5-4b42-9495-5ac5286722c2-istio-envoy\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:26:29.550779 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:29.550571 5024 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84095934-31b5-4b42-9495-5ac5286722c2-istio-data\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:26:29.708468 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:29.708430 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-2dbf739c-gw-openshift-default-64b7d84c5c-kmkgg"] Nov 03 13:26:29.714407 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:29.714366 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-2dbf739c-gw-openshift-default-64b7d84c5c-kmkgg"] Nov 03 13:26:30.114130 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:30.114098 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="84095934-31b5-4b42-9495-5ac5286722c2" path="/var/lib/kubelet/pods/84095934-31b5-4b42-9495-5ac5286722c2/volumes" Nov 03 13:26:30.451036 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:30.450998 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-6z9tv" event={"ID":"7d77cd17-8ce7-4fe7-9b61-7e4c7cdc720b","Type":"ContainerStarted","Data":"80d72223e1f18c419b2b9bde76343f979e9fad8deaab42e8d5bbca63c6a5f150"} Nov 03 13:26:30.526687 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:30.526352 5024 patch_prober.go:29] interesting pod/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-g94p9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Nov 03 13:26:30.526687 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:30.526416 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-g94p9" podUID="bef13398-28a1-4064-aa3b-ea0f3ebf64fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Nov 03 13:26:30.715725 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:30.715189 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-6z9tv" Nov 03 13:26:30.715725 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:30.715354 5024 patch_prober.go:29] interesting pod/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-6z9tv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Nov 03 13:26:30.715725 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:30.715403 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-6z9tv" podUID="7d77cd17-8ce7-4fe7-9b61-7e4c7cdc720b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Nov 03 13:26:30.737953 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:30.737919 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-6z9tv" Nov 03 13:26:31.527977 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:31.527931 5024 patch_prober.go:29] interesting pod/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-g94p9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Nov 03 13:26:31.528727 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:31.528692 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-g94p9" podUID="bef13398-28a1-4064-aa3b-ea0f3ebf64fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Nov 03 13:26:31.728694 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:31.728659 5024 patch_prober.go:29] interesting pod/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-6z9tv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Nov 03 13:26:31.728973 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:31.728942 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-6z9tv" podUID="7d77cd17-8ce7-4fe7-9b61-7e4c7cdc720b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Nov 03 13:26:32.528311 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:32.527700 5024 patch_prober.go:29] interesting pod/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-g94p9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Nov 03 13:26:32.528311 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:32.527768 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-g94p9" podUID="bef13398-28a1-4064-aa3b-ea0f3ebf64fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Nov 03 13:26:32.738489 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:32.738437 5024 patch_prober.go:29] interesting pod/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-6z9tv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Nov 03 13:26:32.781374 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:32.738740 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-6z9tv" podUID="7d77cd17-8ce7-4fe7-9b61-7e4c7cdc720b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Nov 03 13:26:33.527311 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:33.525810 5024 patch_prober.go:29] interesting pod/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-g94p9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Nov 03 13:26:33.527311 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:33.525882 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-g94p9" podUID="bef13398-28a1-4064-aa3b-ea0f3ebf64fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Nov 03 13:26:33.722357 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:33.711666 5024 patch_prober.go:29] interesting pod/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-6z9tv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Nov 03 13:26:33.722357 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:33.711741 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-6z9tv" podUID="7d77cd17-8ce7-4fe7-9b61-7e4c7cdc720b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Nov 03 13:26:34.527309 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:34.527225 5024 patch_prober.go:29] interesting pod/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-g94p9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Nov 03 13:26:34.527309 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:34.527307 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-g94p9" podUID="bef13398-28a1-4064-aa3b-ea0f3ebf64fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Nov 03 13:26:34.725578 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:34.724820 5024 patch_prober.go:29] interesting pod/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-6z9tv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Nov 03 13:26:34.725578 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:34.724890 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-6z9tv" podUID="7d77cd17-8ce7-4fe7-9b61-7e4c7cdc720b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Nov 03 13:26:35.749349 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:35.747014 5024 patch_prober.go:29] interesting pod/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-g94p9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Nov 03 13:26:35.749349 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:35.747082 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-g94p9" podUID="bef13398-28a1-4064-aa3b-ea0f3ebf64fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Nov 03 13:26:35.749349 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:35.747507 5024 patch_prober.go:29] interesting pod/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-6z9tv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Nov 03 13:26:35.749349 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:35.747542 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-6z9tv" podUID="7d77cd17-8ce7-4fe7-9b61-7e4c7cdc720b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Nov 03 13:26:36.420389 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:36.420350 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w0-9add72/auth-test-model-9add72-kserve-54f88d5d5-bm2sv" Nov 03 13:26:36.502433 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:36.502373 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-mq7ng"] Nov 03 13:26:36.537348 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:36.513450 5024 cpu_manager.go:414] "containerMap: removing container" podUID="84095934-31b5-4b42-9495-5ac5286722c2" containerName="istio-proxy" Nov 03 13:26:36.537631 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:36.537604 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84095934-31b5-4b42-9495-5ac5286722c2" containerName="istio-proxy" Nov 03 13:26:36.567340 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:36.541354 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="84095934-31b5-4b42-9495-5ac5286722c2" containerName="istio-proxy" Nov 03 13:26:36.915372 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:36.890686 5024 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-d3ca48/premium-w3-d3ca48-kserve-75ffc959b-zvjbc" podUID="add1e8b9-e7fd-4e4d-8d10-f97d0e2612ba" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 13:26:37.040850 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:37.040809 5024 patch_prober.go:29] interesting pod/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-g94p9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Nov 03 13:26:37.041426 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:37.041392 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-g94p9" podUID="bef13398-28a1-4064-aa3b-ea0f3ebf64fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Nov 03 13:26:37.043214 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:37.043188 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-mq7ng"] Nov 03 13:26:37.043551 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:37.043534 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-mq7ng" Nov 03 13:26:37.047578 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:37.047551 5024 patch_prober.go:29] interesting pod/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-6z9tv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Nov 03 13:26:37.047761 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:37.047738 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-6z9tv" podUID="7d77cd17-8ce7-4fe7-9b61-7e4c7cdc720b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Nov 03 13:26:37.064659 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:37.064469 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca19960c-df6f-44d1-b6b3-c20e06564315-workload-certs\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-mq7ng\" (UID: \"ca19960c-df6f-44d1-b6b3-c20e06564315\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-mq7ng" Nov 03 13:26:37.064932 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:37.064910 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca19960c-df6f-44d1-b6b3-c20e06564315-credential-socket\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-mq7ng\" (UID: \"ca19960c-df6f-44d1-b6b3-c20e06564315\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-mq7ng" Nov 03 13:26:37.065209 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:37.065065 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca19960c-df6f-44d1-b6b3-c20e06564315-istio-envoy\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-mq7ng\" (UID: \"ca19960c-df6f-44d1-b6b3-c20e06564315\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-mq7ng" Nov 03 13:26:37.065552 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:37.065531 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca19960c-df6f-44d1-b6b3-c20e06564315-istio-data\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-mq7ng\" (UID: \"ca19960c-df6f-44d1-b6b3-c20e06564315\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-mq7ng" Nov 03 13:26:37.065826 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:37.065807 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca19960c-df6f-44d1-b6b3-c20e06564315-workload-socket\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-mq7ng\" (UID: \"ca19960c-df6f-44d1-b6b3-c20e06564315\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-mq7ng" Nov 03 13:26:37.171365 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:37.170596 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca19960c-df6f-44d1-b6b3-c20e06564315-workload-certs\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-mq7ng\" (UID: \"ca19960c-df6f-44d1-b6b3-c20e06564315\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-mq7ng" Nov 03 13:26:37.171365 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:37.170669 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca19960c-df6f-44d1-b6b3-c20e06564315-istiod-ca-cert\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-mq7ng\" (UID: \"ca19960c-df6f-44d1-b6b3-c20e06564315\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-mq7ng" Nov 03 13:26:37.171365 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:37.170705 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca19960c-df6f-44d1-b6b3-c20e06564315-credential-socket\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-mq7ng\" (UID: \"ca19960c-df6f-44d1-b6b3-c20e06564315\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-mq7ng" Nov 03 13:26:37.171365 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:37.170738 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca19960c-df6f-44d1-b6b3-c20e06564315-istio-envoy\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-mq7ng\" (UID: \"ca19960c-df6f-44d1-b6b3-c20e06564315\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-mq7ng" Nov 03 13:26:37.171365 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:37.170872 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca19960c-df6f-44d1-b6b3-c20e06564315-istio-token\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-mq7ng\" (UID: \"ca19960c-df6f-44d1-b6b3-c20e06564315\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-mq7ng" Nov 03 13:26:37.171365 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:37.170973 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca19960c-df6f-44d1-b6b3-c20e06564315-istio-data\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-mq7ng\" (UID: \"ca19960c-df6f-44d1-b6b3-c20e06564315\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-mq7ng" Nov 03 13:26:37.171365 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:37.171004 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vmthd\" (UniqueName: \"kubernetes.io/projected/ca19960c-df6f-44d1-b6b3-c20e06564315-kube-api-access-vmthd\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-mq7ng\" (UID: \"ca19960c-df6f-44d1-b6b3-c20e06564315\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-mq7ng" Nov 03 13:26:37.171365 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:37.171044 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca19960c-df6f-44d1-b6b3-c20e06564315-workload-socket\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-mq7ng\" (UID: \"ca19960c-df6f-44d1-b6b3-c20e06564315\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-mq7ng" Nov 03 13:26:37.171365 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:37.171109 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca19960c-df6f-44d1-b6b3-c20e06564315-istio-podinfo\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-mq7ng\" (UID: \"ca19960c-df6f-44d1-b6b3-c20e06564315\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-mq7ng" Nov 03 13:26:37.171992 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:37.171692 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca19960c-df6f-44d1-b6b3-c20e06564315-workload-certs\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-mq7ng\" (UID: \"ca19960c-df6f-44d1-b6b3-c20e06564315\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-mq7ng" Nov 03 13:26:37.171992 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:37.171971 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca19960c-df6f-44d1-b6b3-c20e06564315-credential-socket\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-mq7ng\" (UID: \"ca19960c-df6f-44d1-b6b3-c20e06564315\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-mq7ng" Nov 03 13:26:37.187815 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:37.180833 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca19960c-df6f-44d1-b6b3-c20e06564315-workload-socket\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-mq7ng\" (UID: \"ca19960c-df6f-44d1-b6b3-c20e06564315\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-mq7ng" Nov 03 13:26:37.187815 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:37.181149 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca19960c-df6f-44d1-b6b3-c20e06564315-istio-data\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-mq7ng\" (UID: \"ca19960c-df6f-44d1-b6b3-c20e06564315\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-mq7ng" Nov 03 13:26:37.231064 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:37.229962 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca19960c-df6f-44d1-b6b3-c20e06564315-istio-envoy\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-mq7ng\" (UID: \"ca19960c-df6f-44d1-b6b3-c20e06564315\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-mq7ng" Nov 03 13:26:37.236306 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:37.234783 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w0-9add72/auth-test-model-9add72-kserve-54f88d5d5-bm2sv" Nov 03 13:26:37.256673 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:37.256643 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w0-9add72/auth-test-model-9add72-kserve-54f88d5d5-bm2sv" Nov 03 13:26:37.343327 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:37.340448 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca19960c-df6f-44d1-b6b3-c20e06564315-istio-podinfo\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-mq7ng\" (UID: \"ca19960c-df6f-44d1-b6b3-c20e06564315\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-mq7ng" Nov 03 13:26:37.343327 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:37.340663 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca19960c-df6f-44d1-b6b3-c20e06564315-istiod-ca-cert\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-mq7ng\" (UID: \"ca19960c-df6f-44d1-b6b3-c20e06564315\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-mq7ng" Nov 03 13:26:37.343327 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:37.340831 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca19960c-df6f-44d1-b6b3-c20e06564315-istio-token\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-mq7ng\" (UID: \"ca19960c-df6f-44d1-b6b3-c20e06564315\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-mq7ng" Nov 03 13:26:37.343327 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:37.340942 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vmthd\" (UniqueName: \"kubernetes.io/projected/ca19960c-df6f-44d1-b6b3-c20e06564315-kube-api-access-vmthd\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-mq7ng\" (UID: \"ca19960c-df6f-44d1-b6b3-c20e06564315\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-mq7ng" Nov 03 13:26:37.343327 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:37.343160 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca19960c-df6f-44d1-b6b3-c20e06564315-istiod-ca-cert\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-mq7ng\" (UID: \"ca19960c-df6f-44d1-b6b3-c20e06564315\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-mq7ng" Nov 03 13:26:37.346862 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:37.346834 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca19960c-df6f-44d1-b6b3-c20e06564315-istio-podinfo\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-mq7ng\" (UID: \"ca19960c-df6f-44d1-b6b3-c20e06564315\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-mq7ng" Nov 03 13:26:37.614806 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:37.614775 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-g94p9" Nov 03 13:26:37.638214 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:37.637867 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-g94p9" Nov 03 13:26:37.723953 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:37.723918 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca19960c-df6f-44d1-b6b3-c20e06564315-istio-token\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-mq7ng\" (UID: \"ca19960c-df6f-44d1-b6b3-c20e06564315\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-mq7ng" Nov 03 13:26:37.724615 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:37.724583 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-g94p9" Nov 03 13:26:37.731354 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:37.725450 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vmthd\" (UniqueName: \"kubernetes.io/projected/ca19960c-df6f-44d1-b6b3-c20e06564315-kube-api-access-vmthd\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-mq7ng\" (UID: \"ca19960c-df6f-44d1-b6b3-c20e06564315\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-mq7ng" Nov 03 13:26:37.743073 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:37.743042 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-6z9tv" Nov 03 13:26:37.809948 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:37.809905 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-6z9tv" Nov 03 13:26:37.993686 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:37.993592 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-mq7ng" Nov 03 13:26:38.509527 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:38.507589 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-mq7ng"] Nov 03 13:26:38.511856 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:38.511823 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:26:38.511954 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:38.511898 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:26:38.511954 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:38.511929 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:26:39.303305 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:39.303241 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-mq7ng" event={"ID":"ca19960c-df6f-44d1-b6b3-c20e06564315","Type":"ContainerStarted","Data":"b92d1a3a091f65c21db96f359a7f560892bbc3869f93eba34c250c2e957b6b8c"} Nov 03 13:26:39.303800 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:39.303336 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-mq7ng" event={"ID":"ca19960c-df6f-44d1-b6b3-c20e06564315","Type":"ContainerStarted","Data":"8512f180aac54a29ea18cc9f23cc0d6224997188580c7efa28df4886824fff5c"} Nov 03 13:26:39.384310 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:39.364827 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-mq7ng" podStartSLOduration=3.364802099 podStartE2EDuration="3.364802099s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:26:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:26:39.338267345 +0000 UTC m=+1079.346458623" watchObservedRunningTime="2026-11-03 13:26:39.364802099 +0000 UTC m=+1079.372993377" Nov 03 13:26:39.533966 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:26:39.533911 5024 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d7mwz" containerName="extractor" Nov 03 13:26:40.001347 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:39.997444 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-mq7ng" Nov 03 13:26:40.001347 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:39.997669 5024 patch_prober.go:29] interesting pod/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-mq7ng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Nov 03 13:26:40.001347 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:39.997726 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-mq7ng" podUID="ca19960c-df6f-44d1-b6b3-c20e06564315" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Nov 03 13:26:41.009315 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:41.008444 5024 patch_prober.go:29] interesting pod/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-mq7ng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Nov 03 13:26:41.009315 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:41.008515 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-mq7ng" podUID="ca19960c-df6f-44d1-b6b3-c20e06564315" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Nov 03 13:26:41.997913 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:41.997875 5024 patch_prober.go:29] interesting pod/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-mq7ng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Nov 03 13:26:41.998256 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:41.998220 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-mq7ng" podUID="ca19960c-df6f-44d1-b6b3-c20e06564315" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Nov 03 13:26:43.005308 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:42.997709 5024 patch_prober.go:29] interesting pod/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-mq7ng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Nov 03 13:26:43.005308 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:42.997791 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-mq7ng" podUID="ca19960c-df6f-44d1-b6b3-c20e06564315" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Nov 03 13:26:44.002320 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:44.001748 5024 patch_prober.go:29] interesting pod/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-mq7ng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Nov 03 13:26:44.002320 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:44.001822 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-mq7ng" podUID="ca19960c-df6f-44d1-b6b3-c20e06564315" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Nov 03 13:26:44.999729 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:44.999674 5024 patch_prober.go:29] interesting pod/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-mq7ng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Nov 03 13:26:45.000250 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:44.999744 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-mq7ng" podUID="ca19960c-df6f-44d1-b6b3-c20e06564315" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Nov 03 13:26:46.004311 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:46.000733 5024 patch_prober.go:29] interesting pod/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-mq7ng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Nov 03 13:26:46.004311 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:46.000818 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-mq7ng" podUID="ca19960c-df6f-44d1-b6b3-c20e06564315" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Nov 03 13:26:47.196546 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:47.196509 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-mq7ng" Nov 03 13:26:47.198098 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:47.198073 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-mq7ng" Nov 03 13:26:47.199505 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:47.199477 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-mq7ng" Nov 03 13:26:48.593408 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:48.593313 5024 generic.go:356] "Generic (PLEG): container finished" podID="3c69ea47-20cb-40de-94fb-add95bd74c10" containerID="fb1b7bcd2ac51511eb46ebfa4e730d348cddc59d0a252d65cd58e6b9191402d5" exitCode=0 Nov 03 13:26:48.593408 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:48.593389 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-csthf" event={"ID":"3c69ea47-20cb-40de-94fb-add95bd74c10","Type":"ContainerDied","Data":"fb1b7bcd2ac51511eb46ebfa4e730d348cddc59d0a252d65cd58e6b9191402d5"} Nov 03 13:26:48.594165 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:48.594142 5024 scope.go:122] "RemoveContainer" containerID="fb1b7bcd2ac51511eb46ebfa4e730d348cddc59d0a252d65cd58e6b9191402d5" Nov 03 13:26:48.595818 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:48.595787 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:26:48.595941 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:48.595857 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:26:48.595941 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:48.595889 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:26:49.610702 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:49.610666 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-csthf" event={"ID":"3c69ea47-20cb-40de-94fb-add95bd74c10","Type":"ContainerStarted","Data":"e5e570b28483fbb69bd7d73ee91848f5f59c37b0211abca330431bad5432b4d5"} Nov 03 13:26:50.097567 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:50.097470 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-csthf" Nov 03 13:26:50.099679 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:50.099644 5024 patch_prober.go:29] interesting pod/e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-csthf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Nov 03 13:26:50.099935 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:50.099907 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-csthf" podUID="3c69ea47-20cb-40de-94fb-add95bd74c10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Nov 03 13:26:50.409783 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:50.409749 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-csthf" Nov 03 13:26:51.095307 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:51.094741 5024 patch_prober.go:29] interesting pod/e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-csthf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Nov 03 13:26:51.095307 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:51.094810 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-csthf" podUID="3c69ea47-20cb-40de-94fb-add95bd74c10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Nov 03 13:26:52.101837 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:52.101794 5024 patch_prober.go:29] interesting pod/e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-csthf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Nov 03 13:26:52.102489 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:52.102439 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-csthf" podUID="3c69ea47-20cb-40de-94fb-add95bd74c10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Nov 03 13:26:53.096315 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:53.095492 5024 patch_prober.go:29] interesting pod/e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-csthf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Nov 03 13:26:53.096315 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:53.095567 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-csthf" podUID="3c69ea47-20cb-40de-94fb-add95bd74c10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Nov 03 13:26:54.101313 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:54.097982 5024 patch_prober.go:29] interesting pod/e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-csthf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Nov 03 13:26:54.101313 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:54.098043 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-csthf" podUID="3c69ea47-20cb-40de-94fb-add95bd74c10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Nov 03 13:26:55.092826 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:55.092777 5024 patch_prober.go:29] interesting pod/e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-csthf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Nov 03 13:26:55.093037 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:55.092852 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-csthf" podUID="3c69ea47-20cb-40de-94fb-add95bd74c10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Nov 03 13:26:56.097516 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:56.097478 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-csthf" Nov 03 13:26:56.098748 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:26:56.098721 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-csthf" Nov 03 13:27:00.600175 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:00.600067 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-c87701ad-openshift-default-58f78ff59-b664t"] Nov 03 13:27:00.607965 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:00.607925 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-c87701ad-openshift-default-58f78ff59-b664t" Nov 03 13:27:00.642305 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:00.641763 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-c87701ad-openshift-default-dockercfg-z57x2\"" Nov 03 13:27:00.705397 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:00.705332 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-c87701ad-openshift-default-58f78ff59-b664t"] Nov 03 13:27:00.821496 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:00.821461 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8a61536-4baf-4b92-ae7f-574b72135f52-istio-podinfo\") pod \"e2e-ait-c87701ad-openshift-default-58f78ff59-b664t\" (UID: \"c8a61536-4baf-4b92-ae7f-574b72135f52\") " pod="openshift-ingress/e2e-ait-c87701ad-openshift-default-58f78ff59-b664t" Nov 03 13:27:00.821819 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:00.821794 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8a61536-4baf-4b92-ae7f-574b72135f52-workload-certs\") pod \"e2e-ait-c87701ad-openshift-default-58f78ff59-b664t\" (UID: \"c8a61536-4baf-4b92-ae7f-574b72135f52\") " pod="openshift-ingress/e2e-ait-c87701ad-openshift-default-58f78ff59-b664t" Nov 03 13:27:00.821974 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:00.821960 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8a61536-4baf-4b92-ae7f-574b72135f52-istio-envoy\") pod \"e2e-ait-c87701ad-openshift-default-58f78ff59-b664t\" (UID: \"c8a61536-4baf-4b92-ae7f-574b72135f52\") " pod="openshift-ingress/e2e-ait-c87701ad-openshift-default-58f78ff59-b664t" Nov 03 13:27:00.822118 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:00.822103 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8a61536-4baf-4b92-ae7f-574b72135f52-credential-socket\") pod \"e2e-ait-c87701ad-openshift-default-58f78ff59-b664t\" (UID: \"c8a61536-4baf-4b92-ae7f-574b72135f52\") " pod="openshift-ingress/e2e-ait-c87701ad-openshift-default-58f78ff59-b664t" Nov 03 13:27:00.822440 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:00.822338 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vh9rq\" (UniqueName: \"kubernetes.io/projected/c8a61536-4baf-4b92-ae7f-574b72135f52-kube-api-access-vh9rq\") pod \"e2e-ait-c87701ad-openshift-default-58f78ff59-b664t\" (UID: \"c8a61536-4baf-4b92-ae7f-574b72135f52\") " pod="openshift-ingress/e2e-ait-c87701ad-openshift-default-58f78ff59-b664t" Nov 03 13:27:00.822726 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:00.822706 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8a61536-4baf-4b92-ae7f-574b72135f52-istio-data\") pod \"e2e-ait-c87701ad-openshift-default-58f78ff59-b664t\" (UID: \"c8a61536-4baf-4b92-ae7f-574b72135f52\") " pod="openshift-ingress/e2e-ait-c87701ad-openshift-default-58f78ff59-b664t" Nov 03 13:27:00.822937 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:00.822917 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8a61536-4baf-4b92-ae7f-574b72135f52-workload-socket\") pod \"e2e-ait-c87701ad-openshift-default-58f78ff59-b664t\" (UID: \"c8a61536-4baf-4b92-ae7f-574b72135f52\") " pod="openshift-ingress/e2e-ait-c87701ad-openshift-default-58f78ff59-b664t" Nov 03 13:27:00.823186 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:00.823166 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8a61536-4baf-4b92-ae7f-574b72135f52-istiod-ca-cert\") pod \"e2e-ait-c87701ad-openshift-default-58f78ff59-b664t\" (UID: \"c8a61536-4baf-4b92-ae7f-574b72135f52\") " pod="openshift-ingress/e2e-ait-c87701ad-openshift-default-58f78ff59-b664t" Nov 03 13:27:00.823441 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:00.823419 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8a61536-4baf-4b92-ae7f-574b72135f52-istio-token\") pod \"e2e-ait-c87701ad-openshift-default-58f78ff59-b664t\" (UID: \"c8a61536-4baf-4b92-ae7f-574b72135f52\") " pod="openshift-ingress/e2e-ait-c87701ad-openshift-default-58f78ff59-b664t" Nov 03 13:27:00.928373 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:00.926315 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8a61536-4baf-4b92-ae7f-574b72135f52-istio-data\") pod \"e2e-ait-c87701ad-openshift-default-58f78ff59-b664t\" (UID: \"c8a61536-4baf-4b92-ae7f-574b72135f52\") " pod="openshift-ingress/e2e-ait-c87701ad-openshift-default-58f78ff59-b664t" Nov 03 13:27:00.928373 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:00.926407 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8a61536-4baf-4b92-ae7f-574b72135f52-workload-socket\") pod \"e2e-ait-c87701ad-openshift-default-58f78ff59-b664t\" (UID: \"c8a61536-4baf-4b92-ae7f-574b72135f52\") " pod="openshift-ingress/e2e-ait-c87701ad-openshift-default-58f78ff59-b664t" Nov 03 13:27:00.928373 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:00.926527 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8a61536-4baf-4b92-ae7f-574b72135f52-istiod-ca-cert\") pod \"e2e-ait-c87701ad-openshift-default-58f78ff59-b664t\" (UID: \"c8a61536-4baf-4b92-ae7f-574b72135f52\") " pod="openshift-ingress/e2e-ait-c87701ad-openshift-default-58f78ff59-b664t" Nov 03 13:27:00.928373 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:00.926589 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8a61536-4baf-4b92-ae7f-574b72135f52-istio-token\") pod \"e2e-ait-c87701ad-openshift-default-58f78ff59-b664t\" (UID: \"c8a61536-4baf-4b92-ae7f-574b72135f52\") " pod="openshift-ingress/e2e-ait-c87701ad-openshift-default-58f78ff59-b664t" Nov 03 13:27:00.928373 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:00.926676 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8a61536-4baf-4b92-ae7f-574b72135f52-istio-podinfo\") pod \"e2e-ait-c87701ad-openshift-default-58f78ff59-b664t\" (UID: \"c8a61536-4baf-4b92-ae7f-574b72135f52\") " pod="openshift-ingress/e2e-ait-c87701ad-openshift-default-58f78ff59-b664t" Nov 03 13:27:00.928373 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:00.926740 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8a61536-4baf-4b92-ae7f-574b72135f52-workload-certs\") pod \"e2e-ait-c87701ad-openshift-default-58f78ff59-b664t\" (UID: \"c8a61536-4baf-4b92-ae7f-574b72135f52\") " pod="openshift-ingress/e2e-ait-c87701ad-openshift-default-58f78ff59-b664t" Nov 03 13:27:00.928373 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:00.926791 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8a61536-4baf-4b92-ae7f-574b72135f52-istio-envoy\") pod \"e2e-ait-c87701ad-openshift-default-58f78ff59-b664t\" (UID: \"c8a61536-4baf-4b92-ae7f-574b72135f52\") " pod="openshift-ingress/e2e-ait-c87701ad-openshift-default-58f78ff59-b664t" Nov 03 13:27:00.928373 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:00.926826 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8a61536-4baf-4b92-ae7f-574b72135f52-credential-socket\") pod \"e2e-ait-c87701ad-openshift-default-58f78ff59-b664t\" (UID: \"c8a61536-4baf-4b92-ae7f-574b72135f52\") " pod="openshift-ingress/e2e-ait-c87701ad-openshift-default-58f78ff59-b664t" Nov 03 13:27:00.928373 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:00.926921 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vh9rq\" (UniqueName: \"kubernetes.io/projected/c8a61536-4baf-4b92-ae7f-574b72135f52-kube-api-access-vh9rq\") pod \"e2e-ait-c87701ad-openshift-default-58f78ff59-b664t\" (UID: \"c8a61536-4baf-4b92-ae7f-574b72135f52\") " pod="openshift-ingress/e2e-ait-c87701ad-openshift-default-58f78ff59-b664t" Nov 03 13:27:00.928373 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:00.927863 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8a61536-4baf-4b92-ae7f-574b72135f52-istio-data\") pod \"e2e-ait-c87701ad-openshift-default-58f78ff59-b664t\" (UID: \"c8a61536-4baf-4b92-ae7f-574b72135f52\") " pod="openshift-ingress/e2e-ait-c87701ad-openshift-default-58f78ff59-b664t" Nov 03 13:27:00.928373 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:00.928116 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8a61536-4baf-4b92-ae7f-574b72135f52-workload-socket\") pod \"e2e-ait-c87701ad-openshift-default-58f78ff59-b664t\" (UID: \"c8a61536-4baf-4b92-ae7f-574b72135f52\") " pod="openshift-ingress/e2e-ait-c87701ad-openshift-default-58f78ff59-b664t" Nov 03 13:27:00.929088 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:00.928753 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8a61536-4baf-4b92-ae7f-574b72135f52-istiod-ca-cert\") pod \"e2e-ait-c87701ad-openshift-default-58f78ff59-b664t\" (UID: \"c8a61536-4baf-4b92-ae7f-574b72135f52\") " pod="openshift-ingress/e2e-ait-c87701ad-openshift-default-58f78ff59-b664t" Nov 03 13:27:00.962349 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:00.957809 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8a61536-4baf-4b92-ae7f-574b72135f52-workload-certs\") pod \"e2e-ait-c87701ad-openshift-default-58f78ff59-b664t\" (UID: \"c8a61536-4baf-4b92-ae7f-574b72135f52\") " pod="openshift-ingress/e2e-ait-c87701ad-openshift-default-58f78ff59-b664t" Nov 03 13:27:00.962349 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:00.958957 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8a61536-4baf-4b92-ae7f-574b72135f52-credential-socket\") pod \"e2e-ait-c87701ad-openshift-default-58f78ff59-b664t\" (UID: \"c8a61536-4baf-4b92-ae7f-574b72135f52\") " pod="openshift-ingress/e2e-ait-c87701ad-openshift-default-58f78ff59-b664t" Nov 03 13:27:00.987476 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:00.968846 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8a61536-4baf-4b92-ae7f-574b72135f52-istio-token\") pod \"e2e-ait-c87701ad-openshift-default-58f78ff59-b664t\" (UID: \"c8a61536-4baf-4b92-ae7f-574b72135f52\") " pod="openshift-ingress/e2e-ait-c87701ad-openshift-default-58f78ff59-b664t" Nov 03 13:27:00.988509 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:00.988217 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8a61536-4baf-4b92-ae7f-574b72135f52-istio-podinfo\") pod \"e2e-ait-c87701ad-openshift-default-58f78ff59-b664t\" (UID: \"c8a61536-4baf-4b92-ae7f-574b72135f52\") " pod="openshift-ingress/e2e-ait-c87701ad-openshift-default-58f78ff59-b664t" Nov 03 13:27:00.989482 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:00.989457 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8a61536-4baf-4b92-ae7f-574b72135f52-istio-envoy\") pod \"e2e-ait-c87701ad-openshift-default-58f78ff59-b664t\" (UID: \"c8a61536-4baf-4b92-ae7f-574b72135f52\") " pod="openshift-ingress/e2e-ait-c87701ad-openshift-default-58f78ff59-b664t" Nov 03 13:27:00.994712 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:00.994519 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vh9rq\" (UniqueName: \"kubernetes.io/projected/c8a61536-4baf-4b92-ae7f-574b72135f52-kube-api-access-vh9rq\") pod \"e2e-ait-c87701ad-openshift-default-58f78ff59-b664t\" (UID: \"c8a61536-4baf-4b92-ae7f-574b72135f52\") " pod="openshift-ingress/e2e-ait-c87701ad-openshift-default-58f78ff59-b664t" Nov 03 13:27:01.037306 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:01.035653 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-c87701ad-openshift-default-58f78ff59-b664t" Nov 03 13:27:01.518411 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:01.518309 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-c87701ad-openshift-default-58f78ff59-b664t"] Nov 03 13:27:01.543899 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:27:01.543826 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8a61536_4baf_4b92_ae7f_574b72135f52.slice/crio-2689a543dd1ce0e4b488895142909fa393584cc8ae8a9f4c91aa2501120d3712 WatchSource:0}: Error finding container 2689a543dd1ce0e4b488895142909fa393584cc8ae8a9f4c91aa2501120d3712: Status 404 returned error can't find the container with id 2689a543dd1ce0e4b488895142909fa393584cc8ae8a9f4c91aa2501120d3712 Nov 03 13:27:01.546588 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:01.546552 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:27:01.546708 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:01.546624 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:27:01.546708 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:01.546653 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:27:01.955398 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:01.955347 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c87701ad-openshift-default-58f78ff59-b664t" event={"ID":"c8a61536-4baf-4b92-ae7f-574b72135f52","Type":"ContainerStarted","Data":"8f3aed864cd2b5ea0e77670639cce8ee6a0e426732dc6796fb8e345a0b9be0ad"} Nov 03 13:27:02.027025 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:02.026977 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c87701ad-openshift-default-58f78ff59-b664t" event={"ID":"c8a61536-4baf-4b92-ae7f-574b72135f52","Type":"ContainerStarted","Data":"2689a543dd1ce0e4b488895142909fa393584cc8ae8a9f4c91aa2501120d3712"} Nov 03 13:27:02.027221 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:02.024394 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-c87701ad-openshift-default-58f78ff59-b664t" podStartSLOduration=2.024373394 podStartE2EDuration="2.024373394s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:27:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:27:02.023353833 +0000 UTC m=+1102.031545167" watchObservedRunningTime="2026-11-03 13:27:02.024373394 +0000 UTC m=+1102.032564672" Nov 03 13:27:02.042579 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:02.042542 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-c87701ad-openshift-default-58f78ff59-b664t" Nov 03 13:27:02.042940 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:02.042908 5024 patch_prober.go:29] interesting pod/e2e-ait-c87701ad-openshift-default-58f78ff59-b664t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body= Nov 03 13:27:02.043164 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:02.043129 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-c87701ad-openshift-default-58f78ff59-b664t" podUID="c8a61536-4baf-4b92-ae7f-574b72135f52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" Nov 03 13:27:03.091358 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:03.089524 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-c87701ad-openshift-default-58f78ff59-b664t" Nov 03 13:27:04.033358 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:04.033328 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-c87701ad-openshift-default-58f78ff59-b664t" Nov 03 13:27:04.040416 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:04.040388 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-c87701ad-openshift-default-58f78ff59-b664t" Nov 03 13:27:09.517115 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:27:09.517008 5024 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d7mwz" containerName="extractor" Nov 03 13:27:13.224844 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:13.224803 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-4w96b"] Nov 03 13:27:13.231243 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:13.231215 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-4w96b" Nov 03 13:27:13.263944 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:13.263906 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-4w96b"] Nov 03 13:27:13.305987 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:13.305950 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cabfe6c9-07d8-420b-bb66-143190883f61-workload-certs\") pod \"e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-4w96b\" (UID: \"cabfe6c9-07d8-420b-bb66-143190883f61\") " pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-4w96b" Nov 03 13:27:13.306261 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:13.306003 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cabfe6c9-07d8-420b-bb66-143190883f61-istio-envoy\") pod \"e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-4w96b\" (UID: \"cabfe6c9-07d8-420b-bb66-143190883f61\") " pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-4w96b" Nov 03 13:27:13.306261 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:13.306086 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m55hj\" (UniqueName: \"kubernetes.io/projected/cabfe6c9-07d8-420b-bb66-143190883f61-kube-api-access-m55hj\") pod \"e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-4w96b\" (UID: \"cabfe6c9-07d8-420b-bb66-143190883f61\") " pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-4w96b" Nov 03 13:27:13.306261 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:13.306127 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cabfe6c9-07d8-420b-bb66-143190883f61-workload-socket\") pod \"e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-4w96b\" (UID: \"cabfe6c9-07d8-420b-bb66-143190883f61\") " pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-4w96b" Nov 03 13:27:13.306261 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:13.306164 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cabfe6c9-07d8-420b-bb66-143190883f61-istio-data\") pod \"e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-4w96b\" (UID: \"cabfe6c9-07d8-420b-bb66-143190883f61\") " pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-4w96b" Nov 03 13:27:13.306261 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:13.306222 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cabfe6c9-07d8-420b-bb66-143190883f61-istio-podinfo\") pod \"e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-4w96b\" (UID: \"cabfe6c9-07d8-420b-bb66-143190883f61\") " pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-4w96b" Nov 03 13:27:13.306703 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:13.306339 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cabfe6c9-07d8-420b-bb66-143190883f61-istio-token\") pod \"e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-4w96b\" (UID: \"cabfe6c9-07d8-420b-bb66-143190883f61\") " pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-4w96b" Nov 03 13:27:13.306703 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:13.306360 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cabfe6c9-07d8-420b-bb66-143190883f61-istiod-ca-cert\") pod \"e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-4w96b\" (UID: \"cabfe6c9-07d8-420b-bb66-143190883f61\") " pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-4w96b" Nov 03 13:27:13.306703 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:13.306414 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cabfe6c9-07d8-420b-bb66-143190883f61-credential-socket\") pod \"e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-4w96b\" (UID: \"cabfe6c9-07d8-420b-bb66-143190883f61\") " pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-4w96b" Nov 03 13:27:13.453331 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:13.450665 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cabfe6c9-07d8-420b-bb66-143190883f61-workload-certs\") pod \"e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-4w96b\" (UID: \"cabfe6c9-07d8-420b-bb66-143190883f61\") " pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-4w96b" Nov 03 13:27:13.453331 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:13.450770 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cabfe6c9-07d8-420b-bb66-143190883f61-istio-envoy\") pod \"e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-4w96b\" (UID: \"cabfe6c9-07d8-420b-bb66-143190883f61\") " pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-4w96b" Nov 03 13:27:13.453331 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:13.450836 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m55hj\" (UniqueName: \"kubernetes.io/projected/cabfe6c9-07d8-420b-bb66-143190883f61-kube-api-access-m55hj\") pod \"e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-4w96b\" (UID: \"cabfe6c9-07d8-420b-bb66-143190883f61\") " pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-4w96b" Nov 03 13:27:13.453331 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:13.450919 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cabfe6c9-07d8-420b-bb66-143190883f61-workload-socket\") pod \"e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-4w96b\" (UID: \"cabfe6c9-07d8-420b-bb66-143190883f61\") " pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-4w96b" Nov 03 13:27:13.453331 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:13.450994 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cabfe6c9-07d8-420b-bb66-143190883f61-istio-data\") pod \"e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-4w96b\" (UID: \"cabfe6c9-07d8-420b-bb66-143190883f61\") " pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-4w96b" Nov 03 13:27:13.453331 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:13.451122 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cabfe6c9-07d8-420b-bb66-143190883f61-istio-podinfo\") pod \"e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-4w96b\" (UID: \"cabfe6c9-07d8-420b-bb66-143190883f61\") " pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-4w96b" Nov 03 13:27:13.453331 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:13.451339 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cabfe6c9-07d8-420b-bb66-143190883f61-istio-token\") pod \"e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-4w96b\" (UID: \"cabfe6c9-07d8-420b-bb66-143190883f61\") " pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-4w96b" Nov 03 13:27:13.453331 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:13.451389 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cabfe6c9-07d8-420b-bb66-143190883f61-istiod-ca-cert\") pod \"e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-4w96b\" (UID: \"cabfe6c9-07d8-420b-bb66-143190883f61\") " pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-4w96b" Nov 03 13:27:13.453331 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:13.451647 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cabfe6c9-07d8-420b-bb66-143190883f61-credential-socket\") pod \"e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-4w96b\" (UID: \"cabfe6c9-07d8-420b-bb66-143190883f61\") " pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-4w96b" Nov 03 13:27:13.454328 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:13.454114 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cabfe6c9-07d8-420b-bb66-143190883f61-workload-socket\") pod \"e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-4w96b\" (UID: \"cabfe6c9-07d8-420b-bb66-143190883f61\") " pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-4w96b" Nov 03 13:27:13.454445 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:13.454420 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cabfe6c9-07d8-420b-bb66-143190883f61-workload-certs\") pod \"e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-4w96b\" (UID: \"cabfe6c9-07d8-420b-bb66-143190883f61\") " pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-4w96b" Nov 03 13:27:13.481357 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:13.468137 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cabfe6c9-07d8-420b-bb66-143190883f61-istio-data\") pod \"e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-4w96b\" (UID: \"cabfe6c9-07d8-420b-bb66-143190883f61\") " pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-4w96b" Nov 03 13:27:13.481357 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:13.475181 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cabfe6c9-07d8-420b-bb66-143190883f61-istiod-ca-cert\") pod \"e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-4w96b\" (UID: \"cabfe6c9-07d8-420b-bb66-143190883f61\") " pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-4w96b" Nov 03 13:27:13.481357 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:13.475602 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cabfe6c9-07d8-420b-bb66-143190883f61-credential-socket\") pod \"e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-4w96b\" (UID: \"cabfe6c9-07d8-420b-bb66-143190883f61\") " pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-4w96b" Nov 03 13:27:13.488335 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:13.484326 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m55hj\" (UniqueName: \"kubernetes.io/projected/cabfe6c9-07d8-420b-bb66-143190883f61-kube-api-access-m55hj\") pod \"e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-4w96b\" (UID: \"cabfe6c9-07d8-420b-bb66-143190883f61\") " pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-4w96b" Nov 03 13:27:13.488335 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:13.486072 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cabfe6c9-07d8-420b-bb66-143190883f61-istio-podinfo\") pod \"e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-4w96b\" (UID: \"cabfe6c9-07d8-420b-bb66-143190883f61\") " pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-4w96b" Nov 03 13:27:13.491451 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:13.491400 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cabfe6c9-07d8-420b-bb66-143190883f61-istio-envoy\") pod \"e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-4w96b\" (UID: \"cabfe6c9-07d8-420b-bb66-143190883f61\") " pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-4w96b" Nov 03 13:27:13.494348 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:13.494273 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cabfe6c9-07d8-420b-bb66-143190883f61-istio-token\") pod \"e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-4w96b\" (UID: \"cabfe6c9-07d8-420b-bb66-143190883f61\") " pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-4w96b" Nov 03 13:27:13.548502 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:13.548459 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-4w96b" Nov 03 13:27:13.838150 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:13.838120 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-4w96b"] Nov 03 13:27:13.838879 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:27:13.838854 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcabfe6c9_07d8_420b_bb66_143190883f61.slice/crio-1d053ded65a95f1497d6165338d1797b8d3d4bff13cccac6f0577faa9794a631 WatchSource:0}: Error finding container 1d053ded65a95f1497d6165338d1797b8d3d4bff13cccac6f0577faa9794a631: Status 404 returned error can't find the container with id 1d053ded65a95f1497d6165338d1797b8d3d4bff13cccac6f0577faa9794a631 Nov 03 13:27:13.841569 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:13.841536 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:27:13.841664 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:13.841602 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:27:13.841664 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:13.841634 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:27:14.280344 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:14.280305 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-4w96b" event={"ID":"cabfe6c9-07d8-420b-bb66-143190883f61","Type":"ContainerStarted","Data":"6662648feedad8d9f19220ad83effd7ebfb71a18ab245fed582074b8f870b602"} Nov 03 13:27:14.280868 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:14.280373 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-4w96b" event={"ID":"cabfe6c9-07d8-420b-bb66-143190883f61","Type":"ContainerStarted","Data":"1d053ded65a95f1497d6165338d1797b8d3d4bff13cccac6f0577faa9794a631"} Nov 03 13:27:14.310159 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:14.310087 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-4w96b" podStartSLOduration=1.310065647 podStartE2EDuration="1.310065647s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:27:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:27:14.305821035 +0000 UTC m=+1114.314012313" watchObservedRunningTime="2026-11-03 13:27:14.310065647 +0000 UTC m=+1114.318256926" Nov 03 13:27:14.549278 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:14.549173 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-4w96b" Nov 03 13:27:14.550817 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:14.550791 5024 patch_prober.go:29] interesting pod/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-4w96b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body= Nov 03 13:27:14.550894 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:14.550845 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-4w96b" podUID="cabfe6c9-07d8-420b-bb66-143190883f61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" Nov 03 13:27:15.549771 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:15.549730 5024 patch_prober.go:29] interesting pod/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-4w96b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body= Nov 03 13:27:15.550229 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:15.549798 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-4w96b" podUID="cabfe6c9-07d8-420b-bb66-143190883f61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" Nov 03 13:27:16.549877 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:16.549836 5024 patch_prober.go:29] interesting pod/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-4w96b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body= Nov 03 13:27:16.550541 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:16.550505 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-4w96b" podUID="cabfe6c9-07d8-420b-bb66-143190883f61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" Nov 03 13:27:17.135536 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:17.135453 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-cd4b9r87m"] Nov 03 13:27:17.155788 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:17.154574 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-cd4b9r87m" Nov 03 13:27:17.442870 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:17.442769 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dxlqz\" (UniqueName: \"kubernetes.io/projected/138f6437-e9a2-4270-8583-220e170165f4-kube-api-access-dxlqz\") pod \"payload-processing-external-model-e2e-extmod-b-7c192b-cd4b9r87m\" (UID: \"138f6437-e9a2-4270-8583-220e170165f4\") " pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-cd4b9r87m" Nov 03 13:27:17.459696 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:17.459655 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/138f6437-e9a2-4270-8583-220e170165f4-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-7c192b-cd4b9r87m\" (UID: \"138f6437-e9a2-4270-8583-220e170165f4\") " pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-cd4b9r87m" Nov 03 13:27:17.459998 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:17.459978 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/138f6437-e9a2-4270-8583-220e170165f4-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-7c192b-cd4b9r87m\" (UID: \"138f6437-e9a2-4270-8583-220e170165f4\") " pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-cd4b9r87m" Nov 03 13:27:17.460256 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:17.460237 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/138f6437-e9a2-4270-8583-220e170165f4-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-7c192b-cd4b9r87m\" (UID: \"138f6437-e9a2-4270-8583-220e170165f4\") " pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-cd4b9r87m" Nov 03 13:27:17.592501 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:17.556138 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-cd4b9r87m"] Nov 03 13:27:17.592501 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:17.585947 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxlqz\" (UniqueName: \"kubernetes.io/projected/138f6437-e9a2-4270-8583-220e170165f4-kube-api-access-dxlqz\") pod \"payload-processing-external-model-e2e-extmod-b-7c192b-cd4b9r87m\" (UID: \"138f6437-e9a2-4270-8583-220e170165f4\") " pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-cd4b9r87m" Nov 03 13:27:17.592501 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:17.586025 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/138f6437-e9a2-4270-8583-220e170165f4-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-7c192b-cd4b9r87m\" (UID: \"138f6437-e9a2-4270-8583-220e170165f4\") " pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-cd4b9r87m" Nov 03 13:27:17.592501 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:17.586102 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/138f6437-e9a2-4270-8583-220e170165f4-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-7c192b-cd4b9r87m\" (UID: \"138f6437-e9a2-4270-8583-220e170165f4\") " pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-cd4b9r87m" Nov 03 13:27:17.629369 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:17.605046 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/138f6437-e9a2-4270-8583-220e170165f4-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-7c192b-cd4b9r87m\" (UID: \"138f6437-e9a2-4270-8583-220e170165f4\") " pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-cd4b9r87m" Nov 03 13:27:17.630172 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:17.613703 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/138f6437-e9a2-4270-8583-220e170165f4-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-7c192b-cd4b9r87m\" (UID: \"138f6437-e9a2-4270-8583-220e170165f4\") " pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-cd4b9r87m" Nov 03 13:27:17.739367 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:17.684126 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/138f6437-e9a2-4270-8583-220e170165f4-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-7c192b-cd4b9r87m\" (UID: \"138f6437-e9a2-4270-8583-220e170165f4\") " pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-cd4b9r87m" Nov 03 13:27:17.739367 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:17.719072 5024 patch_prober.go:29] interesting pod/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-4w96b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body= Nov 03 13:27:17.739367 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:17.719142 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-4w96b" podUID="cabfe6c9-07d8-420b-bb66-143190883f61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" Nov 03 13:27:17.750913 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:17.750872 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/138f6437-e9a2-4270-8583-220e170165f4-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-7c192b-cd4b9r87m\" (UID: \"138f6437-e9a2-4270-8583-220e170165f4\") " pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-cd4b9r87m" Nov 03 13:27:17.775246 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:17.775207 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dxlqz\" (UniqueName: \"kubernetes.io/projected/138f6437-e9a2-4270-8583-220e170165f4-kube-api-access-dxlqz\") pod \"payload-processing-external-model-e2e-extmod-b-7c192b-cd4b9r87m\" (UID: \"138f6437-e9a2-4270-8583-220e170165f4\") " pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-cd4b9r87m" Nov 03 13:27:17.846914 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:17.846876 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-cd4b9r87m" Nov 03 13:27:18.054113 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:18.054016 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-2jswl"] Nov 03 13:27:18.055479 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:18.055444 5024 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-2jswl" podUID="ed014864-f935-4abc-8cac-8f989d8b679b" containerName="istio-proxy" containerID="cri-o://ffd5a28f72afdbb8e31c47fc97af0390cac40756db95832eff4882f784417a76" gracePeriod=30 Nov 03 13:27:18.069160 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:18.069120 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-n4t7b"] Nov 03 13:27:18.069968 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:18.069945 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-lldw4"] Nov 03 13:27:18.070370 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:18.070337 5024 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-lldw4" podUID="995deaed-6195-4e09-8ef4-8f1d7aa53d13" containerName="istio-proxy" containerID="cri-o://04d69eb53d00bd5adb24aa2a51c8747e0c8b36a2d661df1d1c5ef84aa47b1503" gracePeriod=30 Nov 03 13:27:18.070821 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:18.070773 5024 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-n4t7b" podUID="0da6e9d5-8520-418d-bbd6-525f6adfa582" containerName="istio-proxy" containerID="cri-o://cc197516ce142f9ee3b6d09e57c19dbcdfb6b544edf0ca7a9d2bd425436491d7" gracePeriod=30 Nov 03 13:27:18.551297 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:18.550802 5024 patch_prober.go:29] interesting pod/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-4w96b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body= Nov 03 13:27:18.551297 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:18.550874 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-4w96b" podUID="cabfe6c9-07d8-420b-bb66-143190883f61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" Nov 03 13:27:19.108858 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:19.108688 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-cd4b9r87m"] Nov 03 13:27:19.549298 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:19.549185 5024 patch_prober.go:29] interesting pod/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-4w96b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body= Nov 03 13:27:19.549825 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:19.549788 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-4w96b" podUID="cabfe6c9-07d8-420b-bb66-143190883f61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" Nov 03 13:27:19.962639 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:19.962598 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-cd4b9r87m" event={"ID":"138f6437-e9a2-4270-8583-220e170165f4","Type":"ContainerStarted","Data":"fd3e90f4a4e8be89850155ac1af02d6da28b634133713b2a7855e9ba8ef41e06"} Nov 03 13:27:19.962824 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:19.962658 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-cd4b9r87m" event={"ID":"138f6437-e9a2-4270-8583-220e170165f4","Type":"ContainerStarted","Data":"8dc917343b7c0495f1e756efafac70fbcf6dcc336619d841de046f79d94f1cff"} Nov 03 13:27:19.962824 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:19.962684 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-cd4b9r87m" Nov 03 13:27:19.991279 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:19.991199 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-cd4b9r87m" podStartSLOduration=2.991175455 podStartE2EDuration="2.991175455s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:27:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:27:19.989871185 +0000 UTC m=+1119.998062457" watchObservedRunningTime="2026-11-03 13:27:19.991175455 +0000 UTC m=+1119.999366733" Nov 03 13:27:20.550625 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:20.550586 5024 patch_prober.go:29] interesting pod/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-4w96b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body= Nov 03 13:27:20.551234 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:20.551199 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-4w96b" podUID="cabfe6c9-07d8-420b-bb66-143190883f61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" Nov 03 13:27:21.584739 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:21.584707 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-4w96b" Nov 03 13:27:21.586339 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:21.586153 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-4w96b" Nov 03 13:27:21.587988 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:21.587959 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-4w96b" Nov 03 13:27:23.422210 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:23.420811 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-2jswl" Nov 03 13:27:23.473731 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:23.473672 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-lldw4" Nov 03 13:27:23.542637 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:23.542611 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-n4t7b" Nov 03 13:27:23.584030 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:23.583988 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ed014864-f935-4abc-8cac-8f989d8b679b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed014864-f935-4abc-8cac-8f989d8b679b-istiod-ca-cert\") pod \"ed014864-f935-4abc-8cac-8f989d8b679b\" (UID: \"ed014864-f935-4abc-8cac-8f989d8b679b\") " Nov 03 13:27:23.584219 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:23.584065 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed014864-f935-4abc-8cac-8f989d8b679b-istio-token\" (UniqueName: \"kubernetes.io/projected/ed014864-f935-4abc-8cac-8f989d8b679b-istio-token\") pod \"ed014864-f935-4abc-8cac-8f989d8b679b\" (UID: \"ed014864-f935-4abc-8cac-8f989d8b679b\") " Nov 03 13:27:23.584219 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:23.584153 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/995deaed-6195-4e09-8ef4-8f1d7aa53d13-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/995deaed-6195-4e09-8ef4-8f1d7aa53d13-istio-data\") pod \"995deaed-6195-4e09-8ef4-8f1d7aa53d13\" (UID: \"995deaed-6195-4e09-8ef4-8f1d7aa53d13\") " Nov 03 13:27:23.584219 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:23.584181 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/995deaed-6195-4e09-8ef4-8f1d7aa53d13-kube-api-access-zb6g7\" (UniqueName: \"kubernetes.io/projected/995deaed-6195-4e09-8ef4-8f1d7aa53d13-kube-api-access-zb6g7\") pod \"995deaed-6195-4e09-8ef4-8f1d7aa53d13\" (UID: \"995deaed-6195-4e09-8ef4-8f1d7aa53d13\") " Nov 03 13:27:23.584416 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:23.584245 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed014864-f935-4abc-8cac-8f989d8b679b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed014864-f935-4abc-8cac-8f989d8b679b-istio-data\") pod \"ed014864-f935-4abc-8cac-8f989d8b679b\" (UID: \"ed014864-f935-4abc-8cac-8f989d8b679b\") " Nov 03 13:27:23.584416 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:23.584330 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed014864-f935-4abc-8cac-8f989d8b679b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed014864-f935-4abc-8cac-8f989d8b679b-istio-envoy\") pod \"ed014864-f935-4abc-8cac-8f989d8b679b\" (UID: \"ed014864-f935-4abc-8cac-8f989d8b679b\") " Nov 03 13:27:23.584517 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:23.584450 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed014864-f935-4abc-8cac-8f989d8b679b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed014864-f935-4abc-8cac-8f989d8b679b-workload-certs\") pod \"ed014864-f935-4abc-8cac-8f989d8b679b\" (UID: \"ed014864-f935-4abc-8cac-8f989d8b679b\") " Nov 03 13:27:23.584517 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:23.584494 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/995deaed-6195-4e09-8ef4-8f1d7aa53d13-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/995deaed-6195-4e09-8ef4-8f1d7aa53d13-istiod-ca-cert\") pod \"995deaed-6195-4e09-8ef4-8f1d7aa53d13\" (UID: \"995deaed-6195-4e09-8ef4-8f1d7aa53d13\") " Nov 03 13:27:23.584604 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:23.584554 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/995deaed-6195-4e09-8ef4-8f1d7aa53d13-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/995deaed-6195-4e09-8ef4-8f1d7aa53d13-istio-podinfo\") pod \"995deaed-6195-4e09-8ef4-8f1d7aa53d13\" (UID: \"995deaed-6195-4e09-8ef4-8f1d7aa53d13\") " Nov 03 13:27:23.584656 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:23.584632 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed014864-f935-4abc-8cac-8f989d8b679b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed014864-f935-4abc-8cac-8f989d8b679b-workload-socket\") pod \"ed014864-f935-4abc-8cac-8f989d8b679b\" (UID: \"ed014864-f935-4abc-8cac-8f989d8b679b\") " Nov 03 13:27:23.584731 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:23.584712 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed014864-f935-4abc-8cac-8f989d8b679b-kube-api-access-xjkht\" (UniqueName: \"kubernetes.io/projected/ed014864-f935-4abc-8cac-8f989d8b679b-kube-api-access-xjkht\") pod \"ed014864-f935-4abc-8cac-8f989d8b679b\" (UID: \"ed014864-f935-4abc-8cac-8f989d8b679b\") " Nov 03 13:27:23.584774 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:23.584749 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed014864-f935-4abc-8cac-8f989d8b679b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed014864-f935-4abc-8cac-8f989d8b679b-credential-socket\") pod \"ed014864-f935-4abc-8cac-8f989d8b679b\" (UID: \"ed014864-f935-4abc-8cac-8f989d8b679b\") " Nov 03 13:27:23.584816 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:23.584791 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/995deaed-6195-4e09-8ef4-8f1d7aa53d13-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/995deaed-6195-4e09-8ef4-8f1d7aa53d13-workload-certs\") pod \"995deaed-6195-4e09-8ef4-8f1d7aa53d13\" (UID: \"995deaed-6195-4e09-8ef4-8f1d7aa53d13\") " Nov 03 13:27:23.584860 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:23.584825 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/995deaed-6195-4e09-8ef4-8f1d7aa53d13-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/995deaed-6195-4e09-8ef4-8f1d7aa53d13-istio-envoy\") pod \"995deaed-6195-4e09-8ef4-8f1d7aa53d13\" (UID: \"995deaed-6195-4e09-8ef4-8f1d7aa53d13\") " Nov 03 13:27:23.584918 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:23.584902 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ed014864-f935-4abc-8cac-8f989d8b679b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed014864-f935-4abc-8cac-8f989d8b679b-istio-podinfo\") pod \"ed014864-f935-4abc-8cac-8f989d8b679b\" (UID: \"ed014864-f935-4abc-8cac-8f989d8b679b\") " Nov 03 13:27:23.584979 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:23.584965 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/995deaed-6195-4e09-8ef4-8f1d7aa53d13-istio-token\" (UniqueName: \"kubernetes.io/projected/995deaed-6195-4e09-8ef4-8f1d7aa53d13-istio-token\") pod \"995deaed-6195-4e09-8ef4-8f1d7aa53d13\" (UID: \"995deaed-6195-4e09-8ef4-8f1d7aa53d13\") " Nov 03 13:27:23.585105 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:23.585089 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/995deaed-6195-4e09-8ef4-8f1d7aa53d13-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/995deaed-6195-4e09-8ef4-8f1d7aa53d13-workload-socket\") pod \"995deaed-6195-4e09-8ef4-8f1d7aa53d13\" (UID: \"995deaed-6195-4e09-8ef4-8f1d7aa53d13\") " Nov 03 13:27:23.585261 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:23.585238 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/995deaed-6195-4e09-8ef4-8f1d7aa53d13-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/995deaed-6195-4e09-8ef4-8f1d7aa53d13-credential-socket\") pod \"995deaed-6195-4e09-8ef4-8f1d7aa53d13\" (UID: \"995deaed-6195-4e09-8ef4-8f1d7aa53d13\") " Nov 03 13:27:23.586494 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:23.586466 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/995deaed-6195-4e09-8ef4-8f1d7aa53d13-credential-socket" pod "995deaed-6195-4e09-8ef4-8f1d7aa53d13" (UID: "995deaed-6195-4e09-8ef4-8f1d7aa53d13"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:27:23.588096 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:23.587070 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed014864-f935-4abc-8cac-8f989d8b679b-istiod-ca-cert" pod "ed014864-f935-4abc-8cac-8f989d8b679b" (UID: "ed014864-f935-4abc-8cac-8f989d8b679b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:27:23.588096 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:23.587218 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed014864-f935-4abc-8cac-8f989d8b679b-istio-data" pod "ed014864-f935-4abc-8cac-8f989d8b679b" (UID: "ed014864-f935-4abc-8cac-8f989d8b679b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:27:23.588096 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:23.587532 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed014864-f935-4abc-8cac-8f989d8b679b-credential-socket" pod "ed014864-f935-4abc-8cac-8f989d8b679b" (UID: "ed014864-f935-4abc-8cac-8f989d8b679b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:27:23.588096 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:23.587712 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed014864-f935-4abc-8cac-8f989d8b679b-workload-socket" pod "ed014864-f935-4abc-8cac-8f989d8b679b" (UID: "ed014864-f935-4abc-8cac-8f989d8b679b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:27:23.588096 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:23.587877 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed014864-f935-4abc-8cac-8f989d8b679b-workload-certs" pod "ed014864-f935-4abc-8cac-8f989d8b679b" (UID: "ed014864-f935-4abc-8cac-8f989d8b679b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:27:23.588823 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:23.588553 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/995deaed-6195-4e09-8ef4-8f1d7aa53d13-workload-certs" pod "995deaed-6195-4e09-8ef4-8f1d7aa53d13" (UID: "995deaed-6195-4e09-8ef4-8f1d7aa53d13"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:27:23.588823 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:23.588568 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/995deaed-6195-4e09-8ef4-8f1d7aa53d13-istio-data" pod "995deaed-6195-4e09-8ef4-8f1d7aa53d13" (UID: "995deaed-6195-4e09-8ef4-8f1d7aa53d13"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:27:23.588990 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:23.588963 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/995deaed-6195-4e09-8ef4-8f1d7aa53d13-istiod-ca-cert" pod "995deaed-6195-4e09-8ef4-8f1d7aa53d13" (UID: "995deaed-6195-4e09-8ef4-8f1d7aa53d13"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:27:23.590548 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:23.590002 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/995deaed-6195-4e09-8ef4-8f1d7aa53d13-istio-envoy" pod "995deaed-6195-4e09-8ef4-8f1d7aa53d13" (UID: "995deaed-6195-4e09-8ef4-8f1d7aa53d13"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:27:23.590548 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:23.590210 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/995deaed-6195-4e09-8ef4-8f1d7aa53d13-workload-socket" pod "995deaed-6195-4e09-8ef4-8f1d7aa53d13" (UID: "995deaed-6195-4e09-8ef4-8f1d7aa53d13"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:27:23.590548 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:23.590332 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed014864-f935-4abc-8cac-8f989d8b679b-istio-token" pod "ed014864-f935-4abc-8cac-8f989d8b679b" (UID: "ed014864-f935-4abc-8cac-8f989d8b679b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:27:23.592538 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:23.592420 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed014864-f935-4abc-8cac-8f989d8b679b-istio-envoy" pod "ed014864-f935-4abc-8cac-8f989d8b679b" (UID: "ed014864-f935-4abc-8cac-8f989d8b679b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:27:23.593927 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:23.593360 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/995deaed-6195-4e09-8ef4-8f1d7aa53d13-kube-api-access-zb6g7" pod "995deaed-6195-4e09-8ef4-8f1d7aa53d13" (UID: "995deaed-6195-4e09-8ef4-8f1d7aa53d13"). InnerVolumeSpecName "kube-api-access-zb6g7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:27:23.594030 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:23.593934 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/995deaed-6195-4e09-8ef4-8f1d7aa53d13-istio-token" pod "995deaed-6195-4e09-8ef4-8f1d7aa53d13" (UID: "995deaed-6195-4e09-8ef4-8f1d7aa53d13"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:27:23.594726 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:23.594699 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ed014864-f935-4abc-8cac-8f989d8b679b-istio-podinfo" pod "ed014864-f935-4abc-8cac-8f989d8b679b" (UID: "ed014864-f935-4abc-8cac-8f989d8b679b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 13:27:23.596526 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:23.596498 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed014864-f935-4abc-8cac-8f989d8b679b-kube-api-access-xjkht" pod "ed014864-f935-4abc-8cac-8f989d8b679b" (UID: "ed014864-f935-4abc-8cac-8f989d8b679b"). InnerVolumeSpecName "kube-api-access-xjkht". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:27:23.597733 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:23.597710 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/995deaed-6195-4e09-8ef4-8f1d7aa53d13-istio-podinfo" pod "995deaed-6195-4e09-8ef4-8f1d7aa53d13" (UID: "995deaed-6195-4e09-8ef4-8f1d7aa53d13"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 13:27:23.687630 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:23.687527 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0da6e9d5-8520-418d-bbd6-525f6adfa582-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0da6e9d5-8520-418d-bbd6-525f6adfa582-istio-envoy\") pod \"0da6e9d5-8520-418d-bbd6-525f6adfa582\" (UID: \"0da6e9d5-8520-418d-bbd6-525f6adfa582\") " Nov 03 13:27:23.687812 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:23.687645 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0da6e9d5-8520-418d-bbd6-525f6adfa582-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0da6e9d5-8520-418d-bbd6-525f6adfa582-credential-socket\") pod \"0da6e9d5-8520-418d-bbd6-525f6adfa582\" (UID: \"0da6e9d5-8520-418d-bbd6-525f6adfa582\") " Nov 03 13:27:23.687812 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:23.687729 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0da6e9d5-8520-418d-bbd6-525f6adfa582-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0da6e9d5-8520-418d-bbd6-525f6adfa582-workload-certs\") pod \"0da6e9d5-8520-418d-bbd6-525f6adfa582\" (UID: \"0da6e9d5-8520-418d-bbd6-525f6adfa582\") " Nov 03 13:27:23.687984 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:23.687869 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0da6e9d5-8520-418d-bbd6-525f6adfa582-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0da6e9d5-8520-418d-bbd6-525f6adfa582-workload-socket\") pod \"0da6e9d5-8520-418d-bbd6-525f6adfa582\" (UID: \"0da6e9d5-8520-418d-bbd6-525f6adfa582\") " Nov 03 13:27:23.687984 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:23.687926 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0da6e9d5-8520-418d-bbd6-525f6adfa582-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0da6e9d5-8520-418d-bbd6-525f6adfa582-istio-data\") pod \"0da6e9d5-8520-418d-bbd6-525f6adfa582\" (UID: \"0da6e9d5-8520-418d-bbd6-525f6adfa582\") " Nov 03 13:27:23.688096 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:23.688030 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0da6e9d5-8520-418d-bbd6-525f6adfa582-credential-socket" pod "0da6e9d5-8520-418d-bbd6-525f6adfa582" (UID: "0da6e9d5-8520-418d-bbd6-525f6adfa582"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:27:23.688096 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:23.688046 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0da6e9d5-8520-418d-bbd6-525f6adfa582-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0da6e9d5-8520-418d-bbd6-525f6adfa582-istio-podinfo\") pod \"0da6e9d5-8520-418d-bbd6-525f6adfa582\" (UID: \"0da6e9d5-8520-418d-bbd6-525f6adfa582\") " Nov 03 13:27:23.688214 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:23.688154 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0da6e9d5-8520-418d-bbd6-525f6adfa582-istio-token\" (UniqueName: \"kubernetes.io/projected/0da6e9d5-8520-418d-bbd6-525f6adfa582-istio-token\") pod \"0da6e9d5-8520-418d-bbd6-525f6adfa582\" (UID: \"0da6e9d5-8520-418d-bbd6-525f6adfa582\") " Nov 03 13:27:23.688261 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:23.688227 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0da6e9d5-8520-418d-bbd6-525f6adfa582-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0da6e9d5-8520-418d-bbd6-525f6adfa582-istiod-ca-cert\") pod \"0da6e9d5-8520-418d-bbd6-525f6adfa582\" (UID: \"0da6e9d5-8520-418d-bbd6-525f6adfa582\") " Nov 03 13:27:23.688349 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:23.688267 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0da6e9d5-8520-418d-bbd6-525f6adfa582-kube-api-access-rptbx\" (UniqueName: \"kubernetes.io/projected/0da6e9d5-8520-418d-bbd6-525f6adfa582-kube-api-access-rptbx\") pod \"0da6e9d5-8520-418d-bbd6-525f6adfa582\" (UID: \"0da6e9d5-8520-418d-bbd6-525f6adfa582\") " Nov 03 13:27:23.688349 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:23.688273 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0da6e9d5-8520-418d-bbd6-525f6adfa582-workload-certs" pod "0da6e9d5-8520-418d-bbd6-525f6adfa582" (UID: "0da6e9d5-8520-418d-bbd6-525f6adfa582"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:27:23.689428 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:23.688514 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0da6e9d5-8520-418d-bbd6-525f6adfa582-workload-socket" pod "0da6e9d5-8520-418d-bbd6-525f6adfa582" (UID: "0da6e9d5-8520-418d-bbd6-525f6adfa582"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:27:23.689428 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:23.688589 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0da6e9d5-8520-418d-bbd6-525f6adfa582-istio-data" pod "0da6e9d5-8520-418d-bbd6-525f6adfa582" (UID: "0da6e9d5-8520-418d-bbd6-525f6adfa582"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:27:23.689428 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:23.688846 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0da6e9d5-8520-418d-bbd6-525f6adfa582-istiod-ca-cert" pod "0da6e9d5-8520-418d-bbd6-525f6adfa582" (UID: "0da6e9d5-8520-418d-bbd6-525f6adfa582"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:27:23.689428 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:23.689125 5024 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0da6e9d5-8520-418d-bbd6-525f6adfa582-workload-socket\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:27:23.689428 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:23.689149 5024 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/995deaed-6195-4e09-8ef4-8f1d7aa53d13-credential-socket\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:27:23.689428 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:23.689164 5024 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0da6e9d5-8520-418d-bbd6-525f6adfa582-istio-data\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:27:23.689428 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:23.689181 5024 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed014864-f935-4abc-8cac-8f989d8b679b-istiod-ca-cert\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:27:23.689428 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:23.689196 5024 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed014864-f935-4abc-8cac-8f989d8b679b-istio-token\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:27:23.689428 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:23.689211 5024 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/995deaed-6195-4e09-8ef4-8f1d7aa53d13-istio-data\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:27:23.689428 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:23.689226 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zb6g7\" (UniqueName: \"kubernetes.io/projected/995deaed-6195-4e09-8ef4-8f1d7aa53d13-kube-api-access-zb6g7\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:27:23.689428 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:23.689242 5024 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed014864-f935-4abc-8cac-8f989d8b679b-istio-data\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:27:23.689428 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:23.689256 5024 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed014864-f935-4abc-8cac-8f989d8b679b-istio-envoy\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:27:23.689428 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:23.689270 5024 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0da6e9d5-8520-418d-bbd6-525f6adfa582-istiod-ca-cert\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:27:23.689428 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:23.689300 5024 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed014864-f935-4abc-8cac-8f989d8b679b-workload-certs\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:27:23.689428 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:23.689317 5024 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/995deaed-6195-4e09-8ef4-8f1d7aa53d13-istiod-ca-cert\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:27:23.689428 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:23.689334 5024 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/995deaed-6195-4e09-8ef4-8f1d7aa53d13-istio-podinfo\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:27:23.689428 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:23.689358 5024 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0da6e9d5-8520-418d-bbd6-525f6adfa582-credential-socket\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:27:23.689428 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:23.689375 5024 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed014864-f935-4abc-8cac-8f989d8b679b-workload-socket\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:27:23.689428 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:23.689391 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xjkht\" (UniqueName: \"kubernetes.io/projected/ed014864-f935-4abc-8cac-8f989d8b679b-kube-api-access-xjkht\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:27:23.689428 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:23.689407 5024 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed014864-f935-4abc-8cac-8f989d8b679b-credential-socket\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:27:23.689428 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:23.689423 5024 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/995deaed-6195-4e09-8ef4-8f1d7aa53d13-workload-certs\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:27:23.689428 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:23.689438 5024 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/995deaed-6195-4e09-8ef4-8f1d7aa53d13-istio-envoy\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:27:23.690513 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:23.689451 5024 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0da6e9d5-8520-418d-bbd6-525f6adfa582-workload-certs\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:27:23.690513 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:23.689464 5024 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed014864-f935-4abc-8cac-8f989d8b679b-istio-podinfo\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:27:23.690513 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:23.689479 5024 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/995deaed-6195-4e09-8ef4-8f1d7aa53d13-istio-token\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:27:23.690513 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:23.689491 5024 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/995deaed-6195-4e09-8ef4-8f1d7aa53d13-workload-socket\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:27:23.690838 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:23.690811 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0da6e9d5-8520-418d-bbd6-525f6adfa582-istio-envoy" pod "0da6e9d5-8520-418d-bbd6-525f6adfa582" (UID: "0da6e9d5-8520-418d-bbd6-525f6adfa582"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:27:23.691085 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:23.691056 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0da6e9d5-8520-418d-bbd6-525f6adfa582-istio-podinfo" pod "0da6e9d5-8520-418d-bbd6-525f6adfa582" (UID: "0da6e9d5-8520-418d-bbd6-525f6adfa582"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 13:27:23.691182 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:23.691118 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0da6e9d5-8520-418d-bbd6-525f6adfa582-istio-token" pod "0da6e9d5-8520-418d-bbd6-525f6adfa582" (UID: "0da6e9d5-8520-418d-bbd6-525f6adfa582"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:27:23.691850 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:23.691829 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0da6e9d5-8520-418d-bbd6-525f6adfa582-kube-api-access-rptbx" pod "0da6e9d5-8520-418d-bbd6-525f6adfa582" (UID: "0da6e9d5-8520-418d-bbd6-525f6adfa582"). InnerVolumeSpecName "kube-api-access-rptbx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:27:23.799399 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:23.796314 5024 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0da6e9d5-8520-418d-bbd6-525f6adfa582-istio-podinfo\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:27:23.799399 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:23.796360 5024 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0da6e9d5-8520-418d-bbd6-525f6adfa582-istio-token\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:27:23.799399 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:23.796377 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rptbx\" (UniqueName: \"kubernetes.io/projected/0da6e9d5-8520-418d-bbd6-525f6adfa582-kube-api-access-rptbx\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:27:23.799399 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:23.796393 5024 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0da6e9d5-8520-418d-bbd6-525f6adfa582-istio-envoy\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:27:24.000672 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:24.000577 5024 generic.go:356] "Generic (PLEG): container finished" podID="ed014864-f935-4abc-8cac-8f989d8b679b" containerID="ffd5a28f72afdbb8e31c47fc97af0390cac40756db95832eff4882f784417a76" exitCode=0 Nov 03 13:27:24.000672 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:24.000630 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-2jswl" Nov 03 13:27:24.000672 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:24.000655 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-2jswl" event={"ID":"ed014864-f935-4abc-8cac-8f989d8b679b","Type":"ContainerDied","Data":"ffd5a28f72afdbb8e31c47fc97af0390cac40756db95832eff4882f784417a76"} Nov 03 13:27:24.000971 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:24.000716 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-2jswl" event={"ID":"ed014864-f935-4abc-8cac-8f989d8b679b","Type":"ContainerDied","Data":"c027506fed1472c7aee15f47e8b2dfa093a7ac0bd24289588ae13e0bdaa669a3"} Nov 03 13:27:24.000971 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:24.000745 5024 scope.go:122] "RemoveContainer" containerID="ffd5a28f72afdbb8e31c47fc97af0390cac40756db95832eff4882f784417a76" Nov 03 13:27:24.002404 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:24.002374 5024 generic.go:356] "Generic (PLEG): container finished" podID="0da6e9d5-8520-418d-bbd6-525f6adfa582" containerID="cc197516ce142f9ee3b6d09e57c19dbcdfb6b544edf0ca7a9d2bd425436491d7" exitCode=0 Nov 03 13:27:24.002557 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:24.002434 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-n4t7b" Nov 03 13:27:24.002663 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:24.002463 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-n4t7b" event={"ID":"0da6e9d5-8520-418d-bbd6-525f6adfa582","Type":"ContainerDied","Data":"cc197516ce142f9ee3b6d09e57c19dbcdfb6b544edf0ca7a9d2bd425436491d7"} Nov 03 13:27:24.002723 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:24.002702 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-n4t7b" event={"ID":"0da6e9d5-8520-418d-bbd6-525f6adfa582","Type":"ContainerDied","Data":"7f517fea1a6a7687790a4689e9b253dc8db3493a533459686171373e1531c173"} Nov 03 13:27:24.005703 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:24.005387 5024 generic.go:356] "Generic (PLEG): container finished" podID="995deaed-6195-4e09-8ef4-8f1d7aa53d13" containerID="04d69eb53d00bd5adb24aa2a51c8747e0c8b36a2d661df1d1c5ef84aa47b1503" exitCode=0 Nov 03 13:27:24.005703 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:24.005459 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-lldw4" Nov 03 13:27:24.005703 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:24.005479 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-lldw4" event={"ID":"995deaed-6195-4e09-8ef4-8f1d7aa53d13","Type":"ContainerDied","Data":"04d69eb53d00bd5adb24aa2a51c8747e0c8b36a2d661df1d1c5ef84aa47b1503"} Nov 03 13:27:24.005703 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:24.005537 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-lldw4" event={"ID":"995deaed-6195-4e09-8ef4-8f1d7aa53d13","Type":"ContainerDied","Data":"d17cdf854df56c71154ba658270ca1ff5c9bfd85a6c66172e5ad188d5af4c526"} Nov 03 13:27:24.015107 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:24.014991 5024 scope.go:122] "RemoveContainer" containerID="ffd5a28f72afdbb8e31c47fc97af0390cac40756db95832eff4882f784417a76" Nov 03 13:27:24.015420 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:27:24.015397 5024 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ffd5a28f72afdbb8e31c47fc97af0390cac40756db95832eff4882f784417a76\": container with ID starting with ffd5a28f72afdbb8e31c47fc97af0390cac40756db95832eff4882f784417a76 not found: ID does not exist" containerID="ffd5a28f72afdbb8e31c47fc97af0390cac40756db95832eff4882f784417a76" Nov 03 13:27:24.015524 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:24.015434 5024 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ffd5a28f72afdbb8e31c47fc97af0390cac40756db95832eff4882f784417a76"} err="failed to get container status \"ffd5a28f72afdbb8e31c47fc97af0390cac40756db95832eff4882f784417a76\": rpc error: code = NotFound desc = could not find container \"ffd5a28f72afdbb8e31c47fc97af0390cac40756db95832eff4882f784417a76\": container with ID starting with ffd5a28f72afdbb8e31c47fc97af0390cac40756db95832eff4882f784417a76 not found: ID does not exist" Nov 03 13:27:24.015524 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:24.015454 5024 scope.go:122] "RemoveContainer" containerID="cc197516ce142f9ee3b6d09e57c19dbcdfb6b544edf0ca7a9d2bd425436491d7" Nov 03 13:27:24.027188 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:24.027163 5024 scope.go:122] "RemoveContainer" containerID="cc197516ce142f9ee3b6d09e57c19dbcdfb6b544edf0ca7a9d2bd425436491d7" Nov 03 13:27:24.027511 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:27:24.027486 5024 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc197516ce142f9ee3b6d09e57c19dbcdfb6b544edf0ca7a9d2bd425436491d7\": container with ID starting with cc197516ce142f9ee3b6d09e57c19dbcdfb6b544edf0ca7a9d2bd425436491d7 not found: ID does not exist" containerID="cc197516ce142f9ee3b6d09e57c19dbcdfb6b544edf0ca7a9d2bd425436491d7" Nov 03 13:27:24.027565 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:24.027520 5024 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc197516ce142f9ee3b6d09e57c19dbcdfb6b544edf0ca7a9d2bd425436491d7"} err="failed to get container status \"cc197516ce142f9ee3b6d09e57c19dbcdfb6b544edf0ca7a9d2bd425436491d7\": rpc error: code = NotFound desc = could not find container \"cc197516ce142f9ee3b6d09e57c19dbcdfb6b544edf0ca7a9d2bd425436491d7\": container with ID starting with cc197516ce142f9ee3b6d09e57c19dbcdfb6b544edf0ca7a9d2bd425436491d7 not found: ID does not exist" Nov 03 13:27:24.027565 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:24.027538 5024 scope.go:122] "RemoveContainer" containerID="04d69eb53d00bd5adb24aa2a51c8747e0c8b36a2d661df1d1c5ef84aa47b1503" Nov 03 13:27:24.037018 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:24.035256 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-2jswl"] Nov 03 13:27:24.039133 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:24.039109 5024 scope.go:122] "RemoveContainer" containerID="e4e8ec926494865695cb452bb85ab4f496dd0f534cc7ffd08ec37bfac6acbb55" Nov 03 13:27:24.040559 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:24.040536 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-2jswl"] Nov 03 13:27:24.049076 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:24.049053 5024 scope.go:122] "RemoveContainer" containerID="04d69eb53d00bd5adb24aa2a51c8747e0c8b36a2d661df1d1c5ef84aa47b1503" Nov 03 13:27:24.049552 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:27:24.049529 5024 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"04d69eb53d00bd5adb24aa2a51c8747e0c8b36a2d661df1d1c5ef84aa47b1503\": container with ID starting with 04d69eb53d00bd5adb24aa2a51c8747e0c8b36a2d661df1d1c5ef84aa47b1503 not found: ID does not exist" containerID="04d69eb53d00bd5adb24aa2a51c8747e0c8b36a2d661df1d1c5ef84aa47b1503" Nov 03 13:27:24.049705 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:24.049563 5024 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04d69eb53d00bd5adb24aa2a51c8747e0c8b36a2d661df1d1c5ef84aa47b1503"} err="failed to get container status \"04d69eb53d00bd5adb24aa2a51c8747e0c8b36a2d661df1d1c5ef84aa47b1503\": rpc error: code = NotFound desc = could not find container \"04d69eb53d00bd5adb24aa2a51c8747e0c8b36a2d661df1d1c5ef84aa47b1503\": container with ID starting with 04d69eb53d00bd5adb24aa2a51c8747e0c8b36a2d661df1d1c5ef84aa47b1503 not found: ID does not exist" Nov 03 13:27:24.049705 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:24.049594 5024 scope.go:122] "RemoveContainer" containerID="e4e8ec926494865695cb452bb85ab4f496dd0f534cc7ffd08ec37bfac6acbb55" Nov 03 13:27:24.050012 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:27:24.049986 5024 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4e8ec926494865695cb452bb85ab4f496dd0f534cc7ffd08ec37bfac6acbb55\": container with ID starting with e4e8ec926494865695cb452bb85ab4f496dd0f534cc7ffd08ec37bfac6acbb55 not found: ID does not exist" containerID="e4e8ec926494865695cb452bb85ab4f496dd0f534cc7ffd08ec37bfac6acbb55" Nov 03 13:27:24.050119 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:24.050019 5024 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4e8ec926494865695cb452bb85ab4f496dd0f534cc7ffd08ec37bfac6acbb55"} err="failed to get container status \"e4e8ec926494865695cb452bb85ab4f496dd0f534cc7ffd08ec37bfac6acbb55\": rpc error: code = NotFound desc = could not find container \"e4e8ec926494865695cb452bb85ab4f496dd0f534cc7ffd08ec37bfac6acbb55\": container with ID starting with e4e8ec926494865695cb452bb85ab4f496dd0f534cc7ffd08ec37bfac6acbb55 not found: ID does not exist" Nov 03 13:27:24.055038 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:24.055011 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-n4t7b"] Nov 03 13:27:24.060525 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:24.060471 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-n4t7b"] Nov 03 13:27:24.080351 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:24.080279 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-lldw4"] Nov 03 13:27:24.084608 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:24.084578 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-lldw4"] Nov 03 13:27:24.120369 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:24.112686 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0da6e9d5-8520-418d-bbd6-525f6adfa582" path="/var/lib/kubelet/pods/0da6e9d5-8520-418d-bbd6-525f6adfa582/volumes" Nov 03 13:27:24.120369 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:24.113725 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="995deaed-6195-4e09-8ef4-8f1d7aa53d13" path="/var/lib/kubelet/pods/995deaed-6195-4e09-8ef4-8f1d7aa53d13/volumes" Nov 03 13:27:24.120369 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:24.114575 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed014864-f935-4abc-8cac-8f989d8b679b" path="/var/lib/kubelet/pods/ed014864-f935-4abc-8cac-8f989d8b679b/volumes" Nov 03 13:27:30.973809 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:30.973775 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-cd4b9r87m" Nov 03 13:27:31.030133 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:31.030097 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-7b5bt8chm"] Nov 03 13:27:34.424302 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:34.424245 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-a62d22-openshift-default-d9c57bf65-pvtj7"] Nov 03 13:27:34.425186 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:34.425164 5024 cpu_manager.go:414] "containerMap: removing container" podUID="0da6e9d5-8520-418d-bbd6-525f6adfa582" containerName="istio-proxy" Nov 03 13:27:34.425321 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:34.425188 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0da6e9d5-8520-418d-bbd6-525f6adfa582" containerName="istio-proxy" Nov 03 13:27:34.425321 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:34.425207 5024 cpu_manager.go:414] "containerMap: removing container" podUID="995deaed-6195-4e09-8ef4-8f1d7aa53d13" containerName="istio-proxy" Nov 03 13:27:34.425321 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:34.425214 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="995deaed-6195-4e09-8ef4-8f1d7aa53d13" containerName="istio-proxy" Nov 03 13:27:34.425321 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:34.425243 5024 cpu_manager.go:414] "containerMap: removing container" podUID="995deaed-6195-4e09-8ef4-8f1d7aa53d13" containerName="istio-proxy" Nov 03 13:27:34.425321 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:34.425251 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="995deaed-6195-4e09-8ef4-8f1d7aa53d13" containerName="istio-proxy" Nov 03 13:27:34.425663 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:34.425329 5024 cpu_manager.go:414] "containerMap: removing container" podUID="ed014864-f935-4abc-8cac-8f989d8b679b" containerName="istio-proxy" Nov 03 13:27:34.425663 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:34.425337 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed014864-f935-4abc-8cac-8f989d8b679b" containerName="istio-proxy" Nov 03 13:27:34.425663 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:34.425518 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="0da6e9d5-8520-418d-bbd6-525f6adfa582" containerName="istio-proxy" Nov 03 13:27:34.425663 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:34.425533 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="995deaed-6195-4e09-8ef4-8f1d7aa53d13" containerName="istio-proxy" Nov 03 13:27:34.425663 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:34.425555 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed014864-f935-4abc-8cac-8f989d8b679b" containerName="istio-proxy" Nov 03 13:27:34.431839 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:34.431803 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-a62d22-openshift-default-d9c57bf65-pvtj7"] Nov 03 13:27:34.431984 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:34.431914 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-a62d22-openshift-default-d9c57bf65-pvtj7" Nov 03 13:27:34.436908 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:34.436877 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-a62d22-openshift-default-dockercfg-r9srg\"" Nov 03 13:27:34.461437 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:34.461400 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/78398b98-5396-4f98-bce4-682198b80f4b-istio-envoy\") pod \"e2e-extmod-a-a62d22-openshift-default-d9c57bf65-pvtj7\" (UID: \"78398b98-5396-4f98-bce4-682198b80f4b\") " pod="openshift-ingress/e2e-extmod-a-a62d22-openshift-default-d9c57bf65-pvtj7" Nov 03 13:27:34.461620 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:34.461450 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/78398b98-5396-4f98-bce4-682198b80f4b-istiod-ca-cert\") pod \"e2e-extmod-a-a62d22-openshift-default-d9c57bf65-pvtj7\" (UID: \"78398b98-5396-4f98-bce4-682198b80f4b\") " pod="openshift-ingress/e2e-extmod-a-a62d22-openshift-default-d9c57bf65-pvtj7" Nov 03 13:27:34.461620 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:34.461606 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/78398b98-5396-4f98-bce4-682198b80f4b-workload-certs\") pod \"e2e-extmod-a-a62d22-openshift-default-d9c57bf65-pvtj7\" (UID: \"78398b98-5396-4f98-bce4-682198b80f4b\") " pod="openshift-ingress/e2e-extmod-a-a62d22-openshift-default-d9c57bf65-pvtj7" Nov 03 13:27:34.461766 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:34.461647 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/78398b98-5396-4f98-bce4-682198b80f4b-istio-token\") pod \"e2e-extmod-a-a62d22-openshift-default-d9c57bf65-pvtj7\" (UID: \"78398b98-5396-4f98-bce4-682198b80f4b\") " pod="openshift-ingress/e2e-extmod-a-a62d22-openshift-default-d9c57bf65-pvtj7" Nov 03 13:27:34.461766 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:34.461729 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/78398b98-5396-4f98-bce4-682198b80f4b-istio-data\") pod \"e2e-extmod-a-a62d22-openshift-default-d9c57bf65-pvtj7\" (UID: \"78398b98-5396-4f98-bce4-682198b80f4b\") " pod="openshift-ingress/e2e-extmod-a-a62d22-openshift-default-d9c57bf65-pvtj7" Nov 03 13:27:34.461876 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:34.461768 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4pc72\" (UniqueName: \"kubernetes.io/projected/78398b98-5396-4f98-bce4-682198b80f4b-kube-api-access-4pc72\") pod \"e2e-extmod-a-a62d22-openshift-default-d9c57bf65-pvtj7\" (UID: \"78398b98-5396-4f98-bce4-682198b80f4b\") " pod="openshift-ingress/e2e-extmod-a-a62d22-openshift-default-d9c57bf65-pvtj7" Nov 03 13:27:34.461927 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:34.461897 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/78398b98-5396-4f98-bce4-682198b80f4b-workload-socket\") pod \"e2e-extmod-a-a62d22-openshift-default-d9c57bf65-pvtj7\" (UID: \"78398b98-5396-4f98-bce4-682198b80f4b\") " pod="openshift-ingress/e2e-extmod-a-a62d22-openshift-default-d9c57bf65-pvtj7" Nov 03 13:27:34.461927 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:34.461919 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/78398b98-5396-4f98-bce4-682198b80f4b-credential-socket\") pod \"e2e-extmod-a-a62d22-openshift-default-d9c57bf65-pvtj7\" (UID: \"78398b98-5396-4f98-bce4-682198b80f4b\") " pod="openshift-ingress/e2e-extmod-a-a62d22-openshift-default-d9c57bf65-pvtj7" Nov 03 13:27:34.462037 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:34.461938 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/78398b98-5396-4f98-bce4-682198b80f4b-istio-podinfo\") pod \"e2e-extmod-a-a62d22-openshift-default-d9c57bf65-pvtj7\" (UID: \"78398b98-5396-4f98-bce4-682198b80f4b\") " pod="openshift-ingress/e2e-extmod-a-a62d22-openshift-default-d9c57bf65-pvtj7" Nov 03 13:27:34.563447 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:34.563404 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/78398b98-5396-4f98-bce4-682198b80f4b-istio-envoy\") pod \"e2e-extmod-a-a62d22-openshift-default-d9c57bf65-pvtj7\" (UID: \"78398b98-5396-4f98-bce4-682198b80f4b\") " pod="openshift-ingress/e2e-extmod-a-a62d22-openshift-default-d9c57bf65-pvtj7" Nov 03 13:27:34.563447 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:34.563450 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/78398b98-5396-4f98-bce4-682198b80f4b-istiod-ca-cert\") pod \"e2e-extmod-a-a62d22-openshift-default-d9c57bf65-pvtj7\" (UID: \"78398b98-5396-4f98-bce4-682198b80f4b\") " pod="openshift-ingress/e2e-extmod-a-a62d22-openshift-default-d9c57bf65-pvtj7" Nov 03 13:27:34.563663 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:34.563519 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/78398b98-5396-4f98-bce4-682198b80f4b-workload-certs\") pod \"e2e-extmod-a-a62d22-openshift-default-d9c57bf65-pvtj7\" (UID: \"78398b98-5396-4f98-bce4-682198b80f4b\") " pod="openshift-ingress/e2e-extmod-a-a62d22-openshift-default-d9c57bf65-pvtj7" Nov 03 13:27:34.563663 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:34.563549 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/78398b98-5396-4f98-bce4-682198b80f4b-istio-token\") pod \"e2e-extmod-a-a62d22-openshift-default-d9c57bf65-pvtj7\" (UID: \"78398b98-5396-4f98-bce4-682198b80f4b\") " pod="openshift-ingress/e2e-extmod-a-a62d22-openshift-default-d9c57bf65-pvtj7" Nov 03 13:27:34.563663 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:34.563590 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/78398b98-5396-4f98-bce4-682198b80f4b-istio-data\") pod \"e2e-extmod-a-a62d22-openshift-default-d9c57bf65-pvtj7\" (UID: \"78398b98-5396-4f98-bce4-682198b80f4b\") " pod="openshift-ingress/e2e-extmod-a-a62d22-openshift-default-d9c57bf65-pvtj7" Nov 03 13:27:34.563663 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:34.563621 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4pc72\" (UniqueName: \"kubernetes.io/projected/78398b98-5396-4f98-bce4-682198b80f4b-kube-api-access-4pc72\") pod \"e2e-extmod-a-a62d22-openshift-default-d9c57bf65-pvtj7\" (UID: \"78398b98-5396-4f98-bce4-682198b80f4b\") " pod="openshift-ingress/e2e-extmod-a-a62d22-openshift-default-d9c57bf65-pvtj7" Nov 03 13:27:34.563847 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:34.563756 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/78398b98-5396-4f98-bce4-682198b80f4b-workload-socket\") pod \"e2e-extmod-a-a62d22-openshift-default-d9c57bf65-pvtj7\" (UID: \"78398b98-5396-4f98-bce4-682198b80f4b\") " pod="openshift-ingress/e2e-extmod-a-a62d22-openshift-default-d9c57bf65-pvtj7" Nov 03 13:27:34.563847 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:34.563791 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/78398b98-5396-4f98-bce4-682198b80f4b-credential-socket\") pod \"e2e-extmod-a-a62d22-openshift-default-d9c57bf65-pvtj7\" (UID: \"78398b98-5396-4f98-bce4-682198b80f4b\") " pod="openshift-ingress/e2e-extmod-a-a62d22-openshift-default-d9c57bf65-pvtj7" Nov 03 13:27:34.563847 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:34.563815 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/78398b98-5396-4f98-bce4-682198b80f4b-istio-podinfo\") pod \"e2e-extmod-a-a62d22-openshift-default-d9c57bf65-pvtj7\" (UID: \"78398b98-5396-4f98-bce4-682198b80f4b\") " pod="openshift-ingress/e2e-extmod-a-a62d22-openshift-default-d9c57bf65-pvtj7" Nov 03 13:27:34.564001 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:34.563908 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/78398b98-5396-4f98-bce4-682198b80f4b-workload-certs\") pod \"e2e-extmod-a-a62d22-openshift-default-d9c57bf65-pvtj7\" (UID: \"78398b98-5396-4f98-bce4-682198b80f4b\") " pod="openshift-ingress/e2e-extmod-a-a62d22-openshift-default-d9c57bf65-pvtj7" Nov 03 13:27:34.564001 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:34.563957 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/78398b98-5396-4f98-bce4-682198b80f4b-istio-data\") pod \"e2e-extmod-a-a62d22-openshift-default-d9c57bf65-pvtj7\" (UID: \"78398b98-5396-4f98-bce4-682198b80f4b\") " pod="openshift-ingress/e2e-extmod-a-a62d22-openshift-default-d9c57bf65-pvtj7" Nov 03 13:27:34.564231 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:34.564203 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/78398b98-5396-4f98-bce4-682198b80f4b-credential-socket\") pod \"e2e-extmod-a-a62d22-openshift-default-d9c57bf65-pvtj7\" (UID: \"78398b98-5396-4f98-bce4-682198b80f4b\") " pod="openshift-ingress/e2e-extmod-a-a62d22-openshift-default-d9c57bf65-pvtj7" Nov 03 13:27:34.564386 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:34.564243 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/78398b98-5396-4f98-bce4-682198b80f4b-workload-socket\") pod \"e2e-extmod-a-a62d22-openshift-default-d9c57bf65-pvtj7\" (UID: \"78398b98-5396-4f98-bce4-682198b80f4b\") " pod="openshift-ingress/e2e-extmod-a-a62d22-openshift-default-d9c57bf65-pvtj7" Nov 03 13:27:34.564386 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:34.564363 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/78398b98-5396-4f98-bce4-682198b80f4b-istiod-ca-cert\") pod \"e2e-extmod-a-a62d22-openshift-default-d9c57bf65-pvtj7\" (UID: \"78398b98-5396-4f98-bce4-682198b80f4b\") " pod="openshift-ingress/e2e-extmod-a-a62d22-openshift-default-d9c57bf65-pvtj7" Nov 03 13:27:34.565987 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:34.565958 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/78398b98-5396-4f98-bce4-682198b80f4b-istio-envoy\") pod \"e2e-extmod-a-a62d22-openshift-default-d9c57bf65-pvtj7\" (UID: \"78398b98-5396-4f98-bce4-682198b80f4b\") " pod="openshift-ingress/e2e-extmod-a-a62d22-openshift-default-d9c57bf65-pvtj7" Nov 03 13:27:34.566249 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:34.566229 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/78398b98-5396-4f98-bce4-682198b80f4b-istio-podinfo\") pod \"e2e-extmod-a-a62d22-openshift-default-d9c57bf65-pvtj7\" (UID: \"78398b98-5396-4f98-bce4-682198b80f4b\") " pod="openshift-ingress/e2e-extmod-a-a62d22-openshift-default-d9c57bf65-pvtj7" Nov 03 13:27:34.615320 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:34.612819 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/78398b98-5396-4f98-bce4-682198b80f4b-istio-token\") pod \"e2e-extmod-a-a62d22-openshift-default-d9c57bf65-pvtj7\" (UID: \"78398b98-5396-4f98-bce4-682198b80f4b\") " pod="openshift-ingress/e2e-extmod-a-a62d22-openshift-default-d9c57bf65-pvtj7" Nov 03 13:27:34.615320 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:34.613494 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4pc72\" (UniqueName: \"kubernetes.io/projected/78398b98-5396-4f98-bce4-682198b80f4b-kube-api-access-4pc72\") pod \"e2e-extmod-a-a62d22-openshift-default-d9c57bf65-pvtj7\" (UID: \"78398b98-5396-4f98-bce4-682198b80f4b\") " pod="openshift-ingress/e2e-extmod-a-a62d22-openshift-default-d9c57bf65-pvtj7" Nov 03 13:27:34.758374 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:34.751589 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-a62d22-openshift-default-d9c57bf65-pvtj7" Nov 03 13:27:35.266668 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:35.266636 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-a62d22-openshift-default-d9c57bf65-pvtj7"] Nov 03 13:27:35.269531 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:35.269499 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:27:35.269625 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:35.269578 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:27:35.269751 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:35.269671 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:27:36.031014 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:36.030971 5024 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-7b5bt8chm" podUID="aad3011d-09ea-4a06-8979-f234cde784bf" containerName="payload-processing" containerID="cri-o://1481840671a84baf66ea96d1a55d37385142951e5025cd18eb927e8473c6a140" gracePeriod=25 Nov 03 13:27:36.141184 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:36.141143 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-a62d22-openshift-default-d9c57bf65-pvtj7" event={"ID":"78398b98-5396-4f98-bce4-682198b80f4b","Type":"ContainerStarted","Data":"8be48a1471b5009f52dfa6f88215e1419deebb1457e0e7b59bf17f1cd67a6869"} Nov 03 13:27:36.141402 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:36.141209 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-a62d22-openshift-default-d9c57bf65-pvtj7" event={"ID":"78398b98-5396-4f98-bce4-682198b80f4b","Type":"ContainerStarted","Data":"4a4383630542f5ab5f3c6bde60d167e5527cf98a6cb75107f7acab11d9e0695d"} Nov 03 13:27:36.172434 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:36.172315 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-a62d22-openshift-default-d9c57bf65-pvtj7" podStartSLOduration=2.172279993 podStartE2EDuration="2.172279993s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:27:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:27:36.166323397 +0000 UTC m=+1136.174514675" watchObservedRunningTime="2026-11-03 13:27:36.172279993 +0000 UTC m=+1136.180471270" Nov 03 13:27:36.444007 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:36.443671 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-7b5bt8chm" Nov 03 13:27:36.486078 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:36.486041 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-nqkls"] Nov 03 13:27:36.499883 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:36.499849 5024 cpu_manager.go:414] "containerMap: removing container" podUID="aad3011d-09ea-4a06-8979-f234cde784bf" containerName="payload-processing" Nov 03 13:27:36.500099 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:36.500082 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aad3011d-09ea-4a06-8979-f234cde784bf" containerName="payload-processing" Nov 03 13:27:36.500642 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:36.500622 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="aad3011d-09ea-4a06-8979-f234cde784bf" containerName="payload-processing" Nov 03 13:27:36.500909 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:36.500894 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="995deaed-6195-4e09-8ef4-8f1d7aa53d13" containerName="istio-proxy" Nov 03 13:27:36.503278 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:36.503246 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aad3011d-09ea-4a06-8979-f234cde784bf-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/aad3011d-09ea-4a06-8979-f234cde784bf-routing-overlay-volume\") pod \"aad3011d-09ea-4a06-8979-f234cde784bf\" (UID: \"aad3011d-09ea-4a06-8979-f234cde784bf\") " Nov 03 13:27:36.503398 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:36.503329 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aad3011d-09ea-4a06-8979-f234cde784bf-kube-api-access-phswp\" (UniqueName: \"kubernetes.io/projected/aad3011d-09ea-4a06-8979-f234cde784bf-kube-api-access-phswp\") pod \"aad3011d-09ea-4a06-8979-f234cde784bf\" (UID: \"aad3011d-09ea-4a06-8979-f234cde784bf\") " Nov 03 13:27:36.503483 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:36.503465 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aad3011d-09ea-4a06-8979-f234cde784bf-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aad3011d-09ea-4a06-8979-f234cde784bf-plugins-config-volume\") pod \"aad3011d-09ea-4a06-8979-f234cde784bf\" (UID: \"aad3011d-09ea-4a06-8979-f234cde784bf\") " Nov 03 13:27:36.503566 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:36.503546 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aad3011d-09ea-4a06-8979-f234cde784bf-provider-credentials\" (UniqueName: \"kubernetes.io/projected/aad3011d-09ea-4a06-8979-f234cde784bf-provider-credentials\") pod \"aad3011d-09ea-4a06-8979-f234cde784bf\" (UID: \"aad3011d-09ea-4a06-8979-f234cde784bf\") " Nov 03 13:27:36.503998 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:36.503971 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aad3011d-09ea-4a06-8979-f234cde784bf-routing-overlay-volume" pod "aad3011d-09ea-4a06-8979-f234cde784bf" (UID: "aad3011d-09ea-4a06-8979-f234cde784bf"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:27:36.504746 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:36.504722 5024 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/aad3011d-09ea-4a06-8979-f234cde784bf-routing-overlay-volume\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:27:36.513601 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:36.513565 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aad3011d-09ea-4a06-8979-f234cde784bf-plugins-config-volume" pod "aad3011d-09ea-4a06-8979-f234cde784bf" (UID: "aad3011d-09ea-4a06-8979-f234cde784bf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:27:36.528760 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:36.528160 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aad3011d-09ea-4a06-8979-f234cde784bf-kube-api-access-phswp" pod "aad3011d-09ea-4a06-8979-f234cde784bf" (UID: "aad3011d-09ea-4a06-8979-f234cde784bf"). InnerVolumeSpecName "kube-api-access-phswp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:27:36.528760 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:36.528684 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aad3011d-09ea-4a06-8979-f234cde784bf-provider-credentials" pod "aad3011d-09ea-4a06-8979-f234cde784bf" (UID: "aad3011d-09ea-4a06-8979-f234cde784bf"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:27:36.533720 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:36.533695 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-nqkls" Nov 03 13:27:36.540127 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:36.540092 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-rzhn5"] Nov 03 13:27:36.547390 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:36.546387 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-rzhn5" Nov 03 13:27:36.547390 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:36.546770 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-nqkls"] Nov 03 13:27:36.553304 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:36.553265 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-rzhn5"] Nov 03 13:27:36.606524 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:36.606491 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1727674a-caa9-4d85-8ee8-45bee8538927-credential-socket\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-rzhn5\" (UID: \"1727674a-caa9-4d85-8ee8-45bee8538927\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-rzhn5" Nov 03 13:27:36.606727 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:36.606699 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1727674a-caa9-4d85-8ee8-45bee8538927-istio-data\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-rzhn5\" (UID: \"1727674a-caa9-4d85-8ee8-45bee8538927\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-rzhn5" Nov 03 13:27:36.606801 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:36.606755 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19ea0757-f048-4f73-9091-cf3a0d86dd12-istiod-ca-cert\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-nqkls\" (UID: \"19ea0757-f048-4f73-9091-cf3a0d86dd12\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-nqkls" Nov 03 13:27:36.606801 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:36.606784 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjdtq\" (UniqueName: \"kubernetes.io/projected/19ea0757-f048-4f73-9091-cf3a0d86dd12-kube-api-access-zjdtq\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-nqkls\" (UID: \"19ea0757-f048-4f73-9091-cf3a0d86dd12\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-nqkls" Nov 03 13:27:36.606982 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:36.606820 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19ea0757-f048-4f73-9091-cf3a0d86dd12-istio-token\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-nqkls\" (UID: \"19ea0757-f048-4f73-9091-cf3a0d86dd12\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-nqkls" Nov 03 13:27:36.606982 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:36.606864 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1727674a-caa9-4d85-8ee8-45bee8538927-workload-certs\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-rzhn5\" (UID: \"1727674a-caa9-4d85-8ee8-45bee8538927\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-rzhn5" Nov 03 13:27:36.606982 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:36.606896 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1727674a-caa9-4d85-8ee8-45bee8538927-istio-envoy\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-rzhn5\" (UID: \"1727674a-caa9-4d85-8ee8-45bee8538927\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-rzhn5" Nov 03 13:27:36.607140 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:36.606982 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8qgrx\" (UniqueName: \"kubernetes.io/projected/1727674a-caa9-4d85-8ee8-45bee8538927-kube-api-access-8qgrx\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-rzhn5\" (UID: \"1727674a-caa9-4d85-8ee8-45bee8538927\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-rzhn5" Nov 03 13:27:36.607140 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:36.607029 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19ea0757-f048-4f73-9091-cf3a0d86dd12-credential-socket\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-nqkls\" (UID: \"19ea0757-f048-4f73-9091-cf3a0d86dd12\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-nqkls" Nov 03 13:27:36.607140 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:36.607058 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1727674a-caa9-4d85-8ee8-45bee8538927-istio-podinfo\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-rzhn5\" (UID: \"1727674a-caa9-4d85-8ee8-45bee8538927\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-rzhn5" Nov 03 13:27:36.607310 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:36.607180 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19ea0757-f048-4f73-9091-cf3a0d86dd12-workload-socket\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-nqkls\" (UID: \"19ea0757-f048-4f73-9091-cf3a0d86dd12\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-nqkls" Nov 03 13:27:36.607310 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:36.607215 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19ea0757-f048-4f73-9091-cf3a0d86dd12-istio-envoy\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-nqkls\" (UID: \"19ea0757-f048-4f73-9091-cf3a0d86dd12\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-nqkls" Nov 03 13:27:36.607310 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:36.607252 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19ea0757-f048-4f73-9091-cf3a0d86dd12-istio-data\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-nqkls\" (UID: \"19ea0757-f048-4f73-9091-cf3a0d86dd12\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-nqkls" Nov 03 13:27:36.607478 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:36.607353 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1727674a-caa9-4d85-8ee8-45bee8538927-istiod-ca-cert\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-rzhn5\" (UID: \"1727674a-caa9-4d85-8ee8-45bee8538927\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-rzhn5" Nov 03 13:27:36.607478 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:36.607410 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1727674a-caa9-4d85-8ee8-45bee8538927-workload-socket\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-rzhn5\" (UID: \"1727674a-caa9-4d85-8ee8-45bee8538927\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-rzhn5" Nov 03 13:27:36.607585 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:36.607545 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1727674a-caa9-4d85-8ee8-45bee8538927-istio-token\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-rzhn5\" (UID: \"1727674a-caa9-4d85-8ee8-45bee8538927\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-rzhn5" Nov 03 13:27:36.607762 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:36.607741 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19ea0757-f048-4f73-9091-cf3a0d86dd12-istio-podinfo\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-nqkls\" (UID: \"19ea0757-f048-4f73-9091-cf3a0d86dd12\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-nqkls" Nov 03 13:27:36.607850 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:36.607834 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19ea0757-f048-4f73-9091-cf3a0d86dd12-workload-certs\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-nqkls\" (UID: \"19ea0757-f048-4f73-9091-cf3a0d86dd12\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-nqkls" Nov 03 13:27:36.607973 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:36.607932 5024 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aad3011d-09ea-4a06-8979-f234cde784bf-plugins-config-volume\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:27:36.607973 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:36.607952 5024 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/aad3011d-09ea-4a06-8979-f234cde784bf-provider-credentials\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:27:36.607973 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:36.607971 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-phswp\" (UniqueName: \"kubernetes.io/projected/aad3011d-09ea-4a06-8979-f234cde784bf-kube-api-access-phswp\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:27:36.709403 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:36.709364 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1727674a-caa9-4d85-8ee8-45bee8538927-workload-certs\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-rzhn5\" (UID: \"1727674a-caa9-4d85-8ee8-45bee8538927\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-rzhn5" Nov 03 13:27:36.709403 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:36.709410 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1727674a-caa9-4d85-8ee8-45bee8538927-istio-envoy\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-rzhn5\" (UID: \"1727674a-caa9-4d85-8ee8-45bee8538927\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-rzhn5" Nov 03 13:27:36.709670 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:36.709459 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8qgrx\" (UniqueName: \"kubernetes.io/projected/1727674a-caa9-4d85-8ee8-45bee8538927-kube-api-access-8qgrx\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-rzhn5\" (UID: \"1727674a-caa9-4d85-8ee8-45bee8538927\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-rzhn5" Nov 03 13:27:36.709670 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:36.709491 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19ea0757-f048-4f73-9091-cf3a0d86dd12-credential-socket\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-nqkls\" (UID: \"19ea0757-f048-4f73-9091-cf3a0d86dd12\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-nqkls" Nov 03 13:27:36.709670 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:36.709518 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1727674a-caa9-4d85-8ee8-45bee8538927-istio-podinfo\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-rzhn5\" (UID: \"1727674a-caa9-4d85-8ee8-45bee8538927\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-rzhn5" Nov 03 13:27:36.709670 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:36.709620 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19ea0757-f048-4f73-9091-cf3a0d86dd12-workload-socket\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-nqkls\" (UID: \"19ea0757-f048-4f73-9091-cf3a0d86dd12\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-nqkls" Nov 03 13:27:36.709670 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:36.709654 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19ea0757-f048-4f73-9091-cf3a0d86dd12-istio-envoy\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-nqkls\" (UID: \"19ea0757-f048-4f73-9091-cf3a0d86dd12\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-nqkls" Nov 03 13:27:36.709919 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:36.709687 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19ea0757-f048-4f73-9091-cf3a0d86dd12-istio-data\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-nqkls\" (UID: \"19ea0757-f048-4f73-9091-cf3a0d86dd12\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-nqkls" Nov 03 13:27:36.709919 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:36.709743 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1727674a-caa9-4d85-8ee8-45bee8538927-istiod-ca-cert\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-rzhn5\" (UID: \"1727674a-caa9-4d85-8ee8-45bee8538927\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-rzhn5" Nov 03 13:27:36.709919 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:36.709782 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1727674a-caa9-4d85-8ee8-45bee8538927-workload-socket\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-rzhn5\" (UID: \"1727674a-caa9-4d85-8ee8-45bee8538927\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-rzhn5" Nov 03 13:27:36.709919 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:36.709814 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1727674a-caa9-4d85-8ee8-45bee8538927-istio-token\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-rzhn5\" (UID: \"1727674a-caa9-4d85-8ee8-45bee8538927\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-rzhn5" Nov 03 13:27:36.709919 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:36.709816 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1727674a-caa9-4d85-8ee8-45bee8538927-workload-certs\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-rzhn5\" (UID: \"1727674a-caa9-4d85-8ee8-45bee8538927\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-rzhn5" Nov 03 13:27:36.709919 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:36.709886 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19ea0757-f048-4f73-9091-cf3a0d86dd12-istio-podinfo\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-nqkls\" (UID: \"19ea0757-f048-4f73-9091-cf3a0d86dd12\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-nqkls" Nov 03 13:27:36.710251 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:36.709924 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19ea0757-f048-4f73-9091-cf3a0d86dd12-credential-socket\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-nqkls\" (UID: \"19ea0757-f048-4f73-9091-cf3a0d86dd12\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-nqkls" Nov 03 13:27:36.710251 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:36.709984 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19ea0757-f048-4f73-9091-cf3a0d86dd12-workload-certs\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-nqkls\" (UID: \"19ea0757-f048-4f73-9091-cf3a0d86dd12\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-nqkls" Nov 03 13:27:36.710251 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:36.710023 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1727674a-caa9-4d85-8ee8-45bee8538927-credential-socket\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-rzhn5\" (UID: \"1727674a-caa9-4d85-8ee8-45bee8538927\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-rzhn5" Nov 03 13:27:36.710251 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:36.710135 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1727674a-caa9-4d85-8ee8-45bee8538927-istio-data\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-rzhn5\" (UID: \"1727674a-caa9-4d85-8ee8-45bee8538927\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-rzhn5" Nov 03 13:27:36.710251 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:36.710147 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19ea0757-f048-4f73-9091-cf3a0d86dd12-workload-socket\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-nqkls\" (UID: \"19ea0757-f048-4f73-9091-cf3a0d86dd12\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-nqkls" Nov 03 13:27:36.710251 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:36.710178 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19ea0757-f048-4f73-9091-cf3a0d86dd12-istiod-ca-cert\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-nqkls\" (UID: \"19ea0757-f048-4f73-9091-cf3a0d86dd12\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-nqkls" Nov 03 13:27:36.710251 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:36.710204 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjdtq\" (UniqueName: \"kubernetes.io/projected/19ea0757-f048-4f73-9091-cf3a0d86dd12-kube-api-access-zjdtq\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-nqkls\" (UID: \"19ea0757-f048-4f73-9091-cf3a0d86dd12\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-nqkls" Nov 03 13:27:36.710251 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:36.710241 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19ea0757-f048-4f73-9091-cf3a0d86dd12-istio-token\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-nqkls\" (UID: \"19ea0757-f048-4f73-9091-cf3a0d86dd12\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-nqkls" Nov 03 13:27:36.710822 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:36.710440 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1727674a-caa9-4d85-8ee8-45bee8538927-credential-socket\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-rzhn5\" (UID: \"1727674a-caa9-4d85-8ee8-45bee8538927\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-rzhn5" Nov 03 13:27:36.710822 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:36.710713 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19ea0757-f048-4f73-9091-cf3a0d86dd12-workload-certs\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-nqkls\" (UID: \"19ea0757-f048-4f73-9091-cf3a0d86dd12\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-nqkls" Nov 03 13:27:36.710936 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:36.710832 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1727674a-caa9-4d85-8ee8-45bee8538927-istio-data\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-rzhn5\" (UID: \"1727674a-caa9-4d85-8ee8-45bee8538927\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-rzhn5" Nov 03 13:27:36.710936 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:36.710854 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1727674a-caa9-4d85-8ee8-45bee8538927-istiod-ca-cert\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-rzhn5\" (UID: \"1727674a-caa9-4d85-8ee8-45bee8538927\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-rzhn5" Nov 03 13:27:36.711153 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:36.711128 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1727674a-caa9-4d85-8ee8-45bee8538927-workload-socket\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-rzhn5\" (UID: \"1727674a-caa9-4d85-8ee8-45bee8538927\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-rzhn5" Nov 03 13:27:36.711362 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:36.711305 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19ea0757-f048-4f73-9091-cf3a0d86dd12-istio-data\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-nqkls\" (UID: \"19ea0757-f048-4f73-9091-cf3a0d86dd12\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-nqkls" Nov 03 13:27:36.711493 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:36.711420 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19ea0757-f048-4f73-9091-cf3a0d86dd12-istiod-ca-cert\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-nqkls\" (UID: \"19ea0757-f048-4f73-9091-cf3a0d86dd12\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-nqkls" Nov 03 13:27:36.712351 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:36.712310 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1727674a-caa9-4d85-8ee8-45bee8538927-istio-envoy\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-rzhn5\" (UID: \"1727674a-caa9-4d85-8ee8-45bee8538927\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-rzhn5" Nov 03 13:27:36.712706 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:36.712681 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1727674a-caa9-4d85-8ee8-45bee8538927-istio-podinfo\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-rzhn5\" (UID: \"1727674a-caa9-4d85-8ee8-45bee8538927\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-rzhn5" Nov 03 13:27:36.712706 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:36.712699 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19ea0757-f048-4f73-9091-cf3a0d86dd12-istio-podinfo\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-nqkls\" (UID: \"19ea0757-f048-4f73-9091-cf3a0d86dd12\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-nqkls" Nov 03 13:27:36.713089 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:36.713061 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19ea0757-f048-4f73-9091-cf3a0d86dd12-istio-envoy\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-nqkls\" (UID: \"19ea0757-f048-4f73-9091-cf3a0d86dd12\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-nqkls" Nov 03 13:27:36.723362 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:36.723335 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjdtq\" (UniqueName: \"kubernetes.io/projected/19ea0757-f048-4f73-9091-cf3a0d86dd12-kube-api-access-zjdtq\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-nqkls\" (UID: \"19ea0757-f048-4f73-9091-cf3a0d86dd12\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-nqkls" Nov 03 13:27:36.723703 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:36.723677 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1727674a-caa9-4d85-8ee8-45bee8538927-istio-token\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-rzhn5\" (UID: \"1727674a-caa9-4d85-8ee8-45bee8538927\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-rzhn5" Nov 03 13:27:36.723703 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:36.723695 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19ea0757-f048-4f73-9091-cf3a0d86dd12-istio-token\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-nqkls\" (UID: \"19ea0757-f048-4f73-9091-cf3a0d86dd12\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-nqkls" Nov 03 13:27:36.723834 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:36.723747 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8qgrx\" (UniqueName: \"kubernetes.io/projected/1727674a-caa9-4d85-8ee8-45bee8538927-kube-api-access-8qgrx\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-rzhn5\" (UID: \"1727674a-caa9-4d85-8ee8-45bee8538927\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-rzhn5" Nov 03 13:27:36.752878 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:36.752843 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-a62d22-openshift-default-d9c57bf65-pvtj7" Nov 03 13:27:36.759390 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:36.759359 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-a62d22-openshift-default-d9c57bf65-pvtj7" Nov 03 13:27:36.867441 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:36.867402 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-nqkls" Nov 03 13:27:36.894727 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:36.894645 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-rzhn5" Nov 03 13:27:37.535623 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:37.535585 5024 generic.go:356] "Generic (PLEG): container finished" podID="aad3011d-09ea-4a06-8979-f234cde784bf" containerID="1481840671a84baf66ea96d1a55d37385142951e5025cd18eb927e8473c6a140" exitCode=0 Nov 03 13:27:37.537407 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:37.537385 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-7b5bt8chm" Nov 03 13:27:37.538585 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:37.538557 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-7b5bt8chm" event={"ID":"aad3011d-09ea-4a06-8979-f234cde784bf","Type":"ContainerDied","Data":"1481840671a84baf66ea96d1a55d37385142951e5025cd18eb927e8473c6a140"} Nov 03 13:27:37.538687 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:37.538614 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-a62d22-openshift-default-d9c57bf65-pvtj7" Nov 03 13:27:37.538687 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:37.538633 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-7b5bt8chm" event={"ID":"aad3011d-09ea-4a06-8979-f234cde784bf","Type":"ContainerDied","Data":"743d0472ba8ee72e8c2e704528114806e22aa079b97f3952b7da785e789931f5"} Nov 03 13:27:37.538687 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:37.538654 5024 scope.go:122] "RemoveContainer" containerID="1481840671a84baf66ea96d1a55d37385142951e5025cd18eb927e8473c6a140" Nov 03 13:27:37.539929 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:37.539913 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-a62d22-openshift-default-d9c57bf65-pvtj7" Nov 03 13:27:37.559969 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:37.559947 5024 scope.go:122] "RemoveContainer" containerID="1481840671a84baf66ea96d1a55d37385142951e5025cd18eb927e8473c6a140" Nov 03 13:27:37.560419 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:27:37.560384 5024 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1481840671a84baf66ea96d1a55d37385142951e5025cd18eb927e8473c6a140\": container with ID starting with 1481840671a84baf66ea96d1a55d37385142951e5025cd18eb927e8473c6a140 not found: ID does not exist" containerID="1481840671a84baf66ea96d1a55d37385142951e5025cd18eb927e8473c6a140" Nov 03 13:27:37.560577 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:37.560429 5024 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1481840671a84baf66ea96d1a55d37385142951e5025cd18eb927e8473c6a140"} err="failed to get container status \"1481840671a84baf66ea96d1a55d37385142951e5025cd18eb927e8473c6a140\": rpc error: code = NotFound desc = could not find container \"1481840671a84baf66ea96d1a55d37385142951e5025cd18eb927e8473c6a140\": container with ID starting with 1481840671a84baf66ea96d1a55d37385142951e5025cd18eb927e8473c6a140 not found: ID does not exist" Nov 03 13:27:37.644335 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:27:37.641216 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19ea0757_f048_4f73_9091_cf3a0d86dd12.slice/crio-036f17e36691ddd7eff3b486f13084f9cbb83c3a5dccc390c854aa0419ea03e5 WatchSource:0}: Error finding container 036f17e36691ddd7eff3b486f13084f9cbb83c3a5dccc390c854aa0419ea03e5: Status 404 returned error can't find the container with id 036f17e36691ddd7eff3b486f13084f9cbb83c3a5dccc390c854aa0419ea03e5 Nov 03 13:27:37.658087 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:27:37.655267 5024 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaad3011d_09ea_4a06_8979_f234cde784bf.slice\": RecentStats: unable to find data in memory cache]" Nov 03 13:27:37.658087 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:37.657960 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-7b5bt8chm"] Nov 03 13:27:37.660845 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:37.659112 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:27:37.660845 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:37.659230 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:27:37.660845 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:37.659325 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:27:37.661379 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:37.661325 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-7b5bt8chm"] Nov 03 13:27:37.666406 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:37.666239 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-nqkls"] Nov 03 13:27:37.832323 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:37.832230 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-rzhn5"] Nov 03 13:27:38.004019 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:38.003198 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:27:38.004019 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:38.003314 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:27:38.004019 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:38.003361 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:27:38.480357 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:38.472711 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aad3011d-09ea-4a06-8979-f234cde784bf" path="/var/lib/kubelet/pods/aad3011d-09ea-4a06-8979-f234cde784bf/volumes" Nov 03 13:27:38.796254 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:38.775326 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-nqkls" event={"ID":"19ea0757-f048-4f73-9091-cf3a0d86dd12","Type":"ContainerStarted","Data":"87f6449ca7b71baa86b1ff7cabe3d3f44555837eb4a4a31e5028ddf20e66573a"} Nov 03 13:27:38.796254 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:38.775444 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-nqkls" event={"ID":"19ea0757-f048-4f73-9091-cf3a0d86dd12","Type":"ContainerStarted","Data":"036f17e36691ddd7eff3b486f13084f9cbb83c3a5dccc390c854aa0419ea03e5"} Nov 03 13:27:38.937413 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:38.934951 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-nqkls" Nov 03 13:27:38.992309 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:38.986581 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-nqkls" podStartSLOduration=2.9865570420000003 podStartE2EDuration="2.986557042s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:27:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:27:38.922245385 +0000 UTC m=+1138.930436662" watchObservedRunningTime="2026-11-03 13:27:38.986557042 +0000 UTC m=+1138.994748320" Nov 03 13:27:38.996754 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:38.996717 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-rzhn5" event={"ID":"1727674a-caa9-4d85-8ee8-45bee8538927","Type":"ContainerStarted","Data":"671164c163498b87c4f866190db94eaed327154dd2bad4be4d4e75107711d25b"} Nov 03 13:27:39.381198 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:39.376478 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d3ca48/premium-w3-d3ca48-kserve-75ffc959b-zvjbc" podUID="add1e8b9-e7fd-4e4d-8d10-f97d0e2612ba" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 13:27:39.796399 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:27:39.796121 5024 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d7mwz" containerName="extractor" Nov 03 13:27:39.879222 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:39.879179 5024 patch_prober.go:29] interesting pod/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-nqkls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body= Nov 03 13:27:39.879520 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:39.879490 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-nqkls" podUID="19ea0757-f048-4f73-9091-cf3a0d86dd12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" Nov 03 13:27:40.062603 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:40.062506 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-rzhn5" event={"ID":"1727674a-caa9-4d85-8ee8-45bee8538927","Type":"ContainerStarted","Data":"6f03585f11e71d5baf89675f2619debaaa20d61dc8cc67e809ba057de9da739a"} Nov 03 13:27:40.887572 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:40.887530 5024 patch_prober.go:29] interesting pod/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-nqkls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body= Nov 03 13:27:40.889041 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:40.888988 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-nqkls" podUID="19ea0757-f048-4f73-9091-cf3a0d86dd12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" Nov 03 13:27:40.924312 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:40.907029 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-rzhn5" Nov 03 13:27:40.934323 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:40.933876 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-rzhn5" podStartSLOduration=4.933853331 podStartE2EDuration="4.933853331s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:27:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:27:40.145675627 +0000 UTC m=+1140.153866905" watchObservedRunningTime="2026-11-03 13:27:40.933853331 +0000 UTC m=+1140.942044612" Nov 03 13:27:41.009352 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:41.009314 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-a62d22-6746df564f-wv77r"] Nov 03 13:27:41.259484 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:41.255500 5024 patch_prober.go:29] interesting pod/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-rzhn5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body= Nov 03 13:27:41.259484 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:41.255589 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-rzhn5" podUID="1727674a-caa9-4d85-8ee8-45bee8538927" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" Nov 03 13:27:41.539441 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:41.458522 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-a62d22-6746df564f-wv77r" Nov 03 13:27:41.539441 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:41.487251 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-a62d22-dockercfg-xl6qp\"" Nov 03 13:27:41.653838 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:41.653792 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-a62d22\"" Nov 03 13:27:41.748846 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:41.748543 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mbdgb\" (UniqueName: \"kubernetes.io/projected/327a26e7-752e-4271-a3eb-1100182124a0-kube-api-access-mbdgb\") pod \"payload-processing-e2e-extmod-a-a62d22-6746df564f-wv77r\" (UID: \"327a26e7-752e-4271-a3eb-1100182124a0\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-a62d22-6746df564f-wv77r" Nov 03 13:27:41.748846 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:41.748715 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/327a26e7-752e-4271-a3eb-1100182124a0-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-a62d22-6746df564f-wv77r\" (UID: \"327a26e7-752e-4271-a3eb-1100182124a0\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-a62d22-6746df564f-wv77r" Nov 03 13:27:41.798494 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:41.793322 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-a62d22-6746df564f-wv77r"] Nov 03 13:27:41.870662 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:41.870615 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbdgb\" (UniqueName: \"kubernetes.io/projected/327a26e7-752e-4271-a3eb-1100182124a0-kube-api-access-mbdgb\") pod \"payload-processing-e2e-extmod-a-a62d22-6746df564f-wv77r\" (UID: \"327a26e7-752e-4271-a3eb-1100182124a0\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-a62d22-6746df564f-wv77r" Nov 03 13:27:41.871047 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:41.870998 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/327a26e7-752e-4271-a3eb-1100182124a0-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-a62d22-6746df564f-wv77r\" (UID: \"327a26e7-752e-4271-a3eb-1100182124a0\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-a62d22-6746df564f-wv77r" Nov 03 13:27:41.888277 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:41.888232 5024 patch_prober.go:29] interesting pod/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-nqkls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body= Nov 03 13:27:41.889236 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:41.889208 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/327a26e7-752e-4271-a3eb-1100182124a0-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-a62d22-6746df564f-wv77r\" (UID: \"327a26e7-752e-4271-a3eb-1100182124a0\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-a62d22-6746df564f-wv77r" Nov 03 13:27:41.893562 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:41.893532 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-a62d22-69fbf9644c-2svhb"] Nov 03 13:27:41.953337 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:41.918360 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-nqkls" podUID="19ea0757-f048-4f73-9091-cf3a0d86dd12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" Nov 03 13:27:41.953337 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:41.919107 5024 patch_prober.go:29] interesting pod/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-rzhn5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body= Nov 03 13:27:41.953337 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:41.919151 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-rzhn5" podUID="1727674a-caa9-4d85-8ee8-45bee8538927" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" Nov 03 13:27:42.000863 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:42.000817 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mbdgb\" (UniqueName: \"kubernetes.io/projected/327a26e7-752e-4271-a3eb-1100182124a0-kube-api-access-mbdgb\") pod \"payload-processing-e2e-extmod-a-a62d22-6746df564f-wv77r\" (UID: \"327a26e7-752e-4271-a3eb-1100182124a0\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-a62d22-6746df564f-wv77r" Nov 03 13:27:42.026937 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:42.017158 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-a62d22-69fbf9644c-2svhb"] Nov 03 13:27:42.026937 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:42.017352 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a62d22-69fbf9644c-2svhb" Nov 03 13:27:42.026937 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:42.017642 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-c87701ad-openshift-default-58f78ff59-b664t"] Nov 03 13:27:42.026937 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:42.017842 5024 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-c87701ad-openshift-default-58f78ff59-b664t" podUID="c8a61536-4baf-4b92-ae7f-574b72135f52" containerName="istio-proxy" containerID="cri-o://8f3aed864cd2b5ea0e77670639cce8ee6a0e426732dc6796fb8e345a0b9be0ad" gracePeriod=30 Nov 03 13:27:42.229855 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:42.219700 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2gkn9\" (UniqueName: \"kubernetes.io/projected/8a432763-9702-42b0-97e4-c2294b4fe866-kube-api-access-2gkn9\") pod \"payload-pre-processing-e2e-extmod-a-a62d22-69fbf9644c-2svhb\" (UID: \"8a432763-9702-42b0-97e4-c2294b4fe866\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a62d22-69fbf9644c-2svhb" Nov 03 13:27:42.229855 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:42.219972 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a432763-9702-42b0-97e4-c2294b4fe866-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-a62d22-69fbf9644c-2svhb\" (UID: \"8a432763-9702-42b0-97e4-c2294b4fe866\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a62d22-69fbf9644c-2svhb" Nov 03 13:27:42.231484 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:42.231210 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-a62d22-6746df564f-wv77r" Nov 03 13:27:42.250594 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:42.250548 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w0-9add72/auth-test-model-9add72-kserve-54f88d5d5-bm2sv" podUID="25181c16-f5c3-49bf-aec6-6c6768d87af9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 13:27:42.393352 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:42.338259 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2gkn9\" (UniqueName: \"kubernetes.io/projected/8a432763-9702-42b0-97e4-c2294b4fe866-kube-api-access-2gkn9\") pod \"payload-pre-processing-e2e-extmod-a-a62d22-69fbf9644c-2svhb\" (UID: \"8a432763-9702-42b0-97e4-c2294b4fe866\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a62d22-69fbf9644c-2svhb" Nov 03 13:27:42.425236 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:42.416664 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a432763-9702-42b0-97e4-c2294b4fe866-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-a62d22-69fbf9644c-2svhb\" (UID: \"8a432763-9702-42b0-97e4-c2294b4fe866\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a62d22-69fbf9644c-2svhb" Nov 03 13:27:42.490065 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:42.489169 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a432763-9702-42b0-97e4-c2294b4fe866-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-a62d22-69fbf9644c-2svhb\" (UID: \"8a432763-9702-42b0-97e4-c2294b4fe866\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a62d22-69fbf9644c-2svhb" Nov 03 13:27:42.720529 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:42.712073 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2gkn9\" (UniqueName: \"kubernetes.io/projected/8a432763-9702-42b0-97e4-c2294b4fe866-kube-api-access-2gkn9\") pod \"payload-pre-processing-e2e-extmod-a-a62d22-69fbf9644c-2svhb\" (UID: \"8a432763-9702-42b0-97e4-c2294b4fe866\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a62d22-69fbf9644c-2svhb" Nov 03 13:27:42.838358 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:42.814350 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a62d22-69fbf9644c-2svhb" Nov 03 13:27:43.089073 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:43.088973 5024 patch_prober.go:29] interesting pod/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-rzhn5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body= Nov 03 13:27:43.089744 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:43.089707 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-rzhn5" podUID="1727674a-caa9-4d85-8ee8-45bee8538927" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" Nov 03 13:27:43.135623 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:43.135580 5024 patch_prober.go:29] interesting pod/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-nqkls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body= Nov 03 13:27:43.135907 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:43.135877 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-nqkls" podUID="19ea0757-f048-4f73-9091-cf3a0d86dd12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" Nov 03 13:27:43.426897 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:43.426852 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w0-9add72/auth-test-model-9add72-kserve-54f88d5d5-bm2sv" podUID="25181c16-f5c3-49bf-aec6-6c6768d87af9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 13:27:43.456132 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:43.456101 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w0-9add72/auth-test-model-9add72-kserve-54f88d5d5-bm2sv" Nov 03 13:27:43.456415 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:43.456399 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-ef15d972-openshift-default-58bd498f85-wxmks"] Nov 03 13:27:43.965310 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:43.964067 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-ef15d972-openshift-default-58bd498f85-wxmks" Nov 03 13:27:43.988308 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:43.980338 5024 patch_prober.go:29] interesting pod/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-rzhn5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body= Nov 03 13:27:43.988308 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:43.980432 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-rzhn5" podUID="1727674a-caa9-4d85-8ee8-45bee8538927" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" Nov 03 13:27:44.090309 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:44.078255 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-ef15d972-openshift-default-dockercfg-9zd24\"" Nov 03 13:27:44.180307 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:44.094259 5024 patch_prober.go:29] interesting pod/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-nqkls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body= Nov 03 13:27:44.272419 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:44.269540 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-nqkls" podUID="19ea0757-f048-4f73-9091-cf3a0d86dd12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" Nov 03 13:27:44.312311 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:44.287048 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/779855bc-fa52-4f8a-8ed8-2692103f9e0d-istio-token\") pod \"e2e-derive-ef15d972-openshift-default-58bd498f85-wxmks\" (UID: \"779855bc-fa52-4f8a-8ed8-2692103f9e0d\") " pod="openshift-ingress/e2e-derive-ef15d972-openshift-default-58bd498f85-wxmks" Nov 03 13:27:44.312819 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:44.312778 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/779855bc-fa52-4f8a-8ed8-2692103f9e0d-istiod-ca-cert\") pod \"e2e-derive-ef15d972-openshift-default-58bd498f85-wxmks\" (UID: \"779855bc-fa52-4f8a-8ed8-2692103f9e0d\") " pod="openshift-ingress/e2e-derive-ef15d972-openshift-default-58bd498f85-wxmks" Nov 03 13:27:44.336309 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:44.313088 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/779855bc-fa52-4f8a-8ed8-2692103f9e0d-credential-socket\") pod \"e2e-derive-ef15d972-openshift-default-58bd498f85-wxmks\" (UID: \"779855bc-fa52-4f8a-8ed8-2692103f9e0d\") " pod="openshift-ingress/e2e-derive-ef15d972-openshift-default-58bd498f85-wxmks" Nov 03 13:27:44.481314 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:44.431520 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w0-9add72/auth-test-model-9add72-kserve-54f88d5d5-bm2sv" podUID="25181c16-f5c3-49bf-aec6-6c6768d87af9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": context deadline exceeded" Nov 03 13:27:44.867502 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:44.611657 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/779855bc-fa52-4f8a-8ed8-2692103f9e0d-istio-data\") pod \"e2e-derive-ef15d972-openshift-default-58bd498f85-wxmks\" (UID: \"779855bc-fa52-4f8a-8ed8-2692103f9e0d\") " pod="openshift-ingress/e2e-derive-ef15d972-openshift-default-58bd498f85-wxmks" Nov 03 13:27:44.867502 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:44.866060 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/779855bc-fa52-4f8a-8ed8-2692103f9e0d-workload-socket\") pod \"e2e-derive-ef15d972-openshift-default-58bd498f85-wxmks\" (UID: \"779855bc-fa52-4f8a-8ed8-2692103f9e0d\") " pod="openshift-ingress/e2e-derive-ef15d972-openshift-default-58bd498f85-wxmks" Nov 03 13:27:44.867502 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:44.866239 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ktjm\" (UniqueName: \"kubernetes.io/projected/779855bc-fa52-4f8a-8ed8-2692103f9e0d-kube-api-access-6ktjm\") pod \"e2e-derive-ef15d972-openshift-default-58bd498f85-wxmks\" (UID: \"779855bc-fa52-4f8a-8ed8-2692103f9e0d\") " pod="openshift-ingress/e2e-derive-ef15d972-openshift-default-58bd498f85-wxmks" Nov 03 13:27:44.867502 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:44.866398 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/779855bc-fa52-4f8a-8ed8-2692103f9e0d-istio-podinfo\") pod \"e2e-derive-ef15d972-openshift-default-58bd498f85-wxmks\" (UID: \"779855bc-fa52-4f8a-8ed8-2692103f9e0d\") " pod="openshift-ingress/e2e-derive-ef15d972-openshift-default-58bd498f85-wxmks" Nov 03 13:27:44.867502 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:44.866505 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/779855bc-fa52-4f8a-8ed8-2692103f9e0d-istio-envoy\") pod \"e2e-derive-ef15d972-openshift-default-58bd498f85-wxmks\" (UID: \"779855bc-fa52-4f8a-8ed8-2692103f9e0d\") " pod="openshift-ingress/e2e-derive-ef15d972-openshift-default-58bd498f85-wxmks" Nov 03 13:27:44.867502 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:44.866619 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/779855bc-fa52-4f8a-8ed8-2692103f9e0d-workload-certs\") pod \"e2e-derive-ef15d972-openshift-default-58bd498f85-wxmks\" (UID: \"779855bc-fa52-4f8a-8ed8-2692103f9e0d\") " pod="openshift-ingress/e2e-derive-ef15d972-openshift-default-58bd498f85-wxmks" Nov 03 13:27:44.925954 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:44.925877 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d3ca48/premium-w3-d3ca48-kserve-75ffc959b-zvjbc" podUID="add1e8b9-e7fd-4e4d-8d10-f97d0e2612ba" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 13:27:44.935018 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:44.927216 5024 patch_prober.go:29] interesting pod/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-rzhn5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body= Nov 03 13:27:44.935018 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:44.927264 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-rzhn5" podUID="1727674a-caa9-4d85-8ee8-45bee8538927" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" Nov 03 13:27:45.026417 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:45.026372 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-ef15d972-openshift-default-58bd498f85-wxmks"] Nov 03 13:27:45.143391 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:45.041090 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ktjm\" (UniqueName: \"kubernetes.io/projected/779855bc-fa52-4f8a-8ed8-2692103f9e0d-kube-api-access-6ktjm\") pod \"e2e-derive-ef15d972-openshift-default-58bd498f85-wxmks\" (UID: \"779855bc-fa52-4f8a-8ed8-2692103f9e0d\") " pod="openshift-ingress/e2e-derive-ef15d972-openshift-default-58bd498f85-wxmks" Nov 03 13:27:45.168499 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:45.168450 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/779855bc-fa52-4f8a-8ed8-2692103f9e0d-istio-podinfo\") pod \"e2e-derive-ef15d972-openshift-default-58bd498f85-wxmks\" (UID: \"779855bc-fa52-4f8a-8ed8-2692103f9e0d\") " pod="openshift-ingress/e2e-derive-ef15d972-openshift-default-58bd498f85-wxmks" Nov 03 13:27:45.172309 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:45.168803 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/779855bc-fa52-4f8a-8ed8-2692103f9e0d-istio-envoy\") pod \"e2e-derive-ef15d972-openshift-default-58bd498f85-wxmks\" (UID: \"779855bc-fa52-4f8a-8ed8-2692103f9e0d\") " pod="openshift-ingress/e2e-derive-ef15d972-openshift-default-58bd498f85-wxmks" Nov 03 13:27:45.184461 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:45.184423 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/779855bc-fa52-4f8a-8ed8-2692103f9e0d-workload-certs\") pod \"e2e-derive-ef15d972-openshift-default-58bd498f85-wxmks\" (UID: \"779855bc-fa52-4f8a-8ed8-2692103f9e0d\") " pod="openshift-ingress/e2e-derive-ef15d972-openshift-default-58bd498f85-wxmks" Nov 03 13:27:45.188309 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:45.184727 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/779855bc-fa52-4f8a-8ed8-2692103f9e0d-istio-token\") pod \"e2e-derive-ef15d972-openshift-default-58bd498f85-wxmks\" (UID: \"779855bc-fa52-4f8a-8ed8-2692103f9e0d\") " pod="openshift-ingress/e2e-derive-ef15d972-openshift-default-58bd498f85-wxmks" Nov 03 13:27:45.246378 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:45.187186 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/779855bc-fa52-4f8a-8ed8-2692103f9e0d-workload-certs\") pod \"e2e-derive-ef15d972-openshift-default-58bd498f85-wxmks\" (UID: \"779855bc-fa52-4f8a-8ed8-2692103f9e0d\") " pod="openshift-ingress/e2e-derive-ef15d972-openshift-default-58bd498f85-wxmks" Nov 03 13:27:45.246378 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:45.132700 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ktjm\" (UniqueName: \"kubernetes.io/projected/779855bc-fa52-4f8a-8ed8-2692103f9e0d-kube-api-access-6ktjm\") pod \"e2e-derive-ef15d972-openshift-default-58bd498f85-wxmks\" (UID: \"779855bc-fa52-4f8a-8ed8-2692103f9e0d\") " pod="openshift-ingress/e2e-derive-ef15d972-openshift-default-58bd498f85-wxmks" Nov 03 13:27:45.246378 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:45.065552 5024 patch_prober.go:29] interesting pod/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-nqkls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body= Nov 03 13:27:45.246378 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:45.236621 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-nqkls" podUID="19ea0757-f048-4f73-9091-cf3a0d86dd12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" Nov 03 13:27:45.247030 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:45.246989 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/779855bc-fa52-4f8a-8ed8-2692103f9e0d-istiod-ca-cert\") pod \"e2e-derive-ef15d972-openshift-default-58bd498f85-wxmks\" (UID: \"779855bc-fa52-4f8a-8ed8-2692103f9e0d\") " pod="openshift-ingress/e2e-derive-ef15d972-openshift-default-58bd498f85-wxmks" Nov 03 13:27:45.247260 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:45.247241 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/779855bc-fa52-4f8a-8ed8-2692103f9e0d-credential-socket\") pod \"e2e-derive-ef15d972-openshift-default-58bd498f85-wxmks\" (UID: \"779855bc-fa52-4f8a-8ed8-2692103f9e0d\") " pod="openshift-ingress/e2e-derive-ef15d972-openshift-default-58bd498f85-wxmks" Nov 03 13:27:45.247542 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:45.247509 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/779855bc-fa52-4f8a-8ed8-2692103f9e0d-istio-data\") pod \"e2e-derive-ef15d972-openshift-default-58bd498f85-wxmks\" (UID: \"779855bc-fa52-4f8a-8ed8-2692103f9e0d\") " pod="openshift-ingress/e2e-derive-ef15d972-openshift-default-58bd498f85-wxmks" Nov 03 13:27:45.247705 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:45.247687 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/779855bc-fa52-4f8a-8ed8-2692103f9e0d-workload-socket\") pod \"e2e-derive-ef15d972-openshift-default-58bd498f85-wxmks\" (UID: \"779855bc-fa52-4f8a-8ed8-2692103f9e0d\") " pod="openshift-ingress/e2e-derive-ef15d972-openshift-default-58bd498f85-wxmks" Nov 03 13:27:45.252354 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:45.252320 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/779855bc-fa52-4f8a-8ed8-2692103f9e0d-workload-socket\") pod \"e2e-derive-ef15d972-openshift-default-58bd498f85-wxmks\" (UID: \"779855bc-fa52-4f8a-8ed8-2692103f9e0d\") " pod="openshift-ingress/e2e-derive-ef15d972-openshift-default-58bd498f85-wxmks" Nov 03 13:27:45.318605 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:45.287723 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/779855bc-fa52-4f8a-8ed8-2692103f9e0d-istio-data\") pod \"e2e-derive-ef15d972-openshift-default-58bd498f85-wxmks\" (UID: \"779855bc-fa52-4f8a-8ed8-2692103f9e0d\") " pod="openshift-ingress/e2e-derive-ef15d972-openshift-default-58bd498f85-wxmks" Nov 03 13:27:45.318605 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:45.288003 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/779855bc-fa52-4f8a-8ed8-2692103f9e0d-credential-socket\") pod \"e2e-derive-ef15d972-openshift-default-58bd498f85-wxmks\" (UID: \"779855bc-fa52-4f8a-8ed8-2692103f9e0d\") " pod="openshift-ingress/e2e-derive-ef15d972-openshift-default-58bd498f85-wxmks" Nov 03 13:27:45.332922 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:45.332882 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/779855bc-fa52-4f8a-8ed8-2692103f9e0d-istio-token\") pod \"e2e-derive-ef15d972-openshift-default-58bd498f85-wxmks\" (UID: \"779855bc-fa52-4f8a-8ed8-2692103f9e0d\") " pod="openshift-ingress/e2e-derive-ef15d972-openshift-default-58bd498f85-wxmks" Nov 03 13:27:45.398362 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:45.395054 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/779855bc-fa52-4f8a-8ed8-2692103f9e0d-istiod-ca-cert\") pod \"e2e-derive-ef15d972-openshift-default-58bd498f85-wxmks\" (UID: \"779855bc-fa52-4f8a-8ed8-2692103f9e0d\") " pod="openshift-ingress/e2e-derive-ef15d972-openshift-default-58bd498f85-wxmks" Nov 03 13:27:45.521349 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:45.520959 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/779855bc-fa52-4f8a-8ed8-2692103f9e0d-istio-envoy\") pod \"e2e-derive-ef15d972-openshift-default-58bd498f85-wxmks\" (UID: \"779855bc-fa52-4f8a-8ed8-2692103f9e0d\") " pod="openshift-ingress/e2e-derive-ef15d972-openshift-default-58bd498f85-wxmks" Nov 03 13:27:45.543663 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:45.537913 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w0-9add72/auth-test-model-9add72-kserve-54f88d5d5-bm2sv" podUID="25181c16-f5c3-49bf-aec6-6c6768d87af9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": context deadline exceeded" Nov 03 13:27:45.608309 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:45.600949 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/779855bc-fa52-4f8a-8ed8-2692103f9e0d-istio-podinfo\") pod \"e2e-derive-ef15d972-openshift-default-58bd498f85-wxmks\" (UID: \"779855bc-fa52-4f8a-8ed8-2692103f9e0d\") " pod="openshift-ingress/e2e-derive-ef15d972-openshift-default-58bd498f85-wxmks" Nov 03 13:27:45.884715 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:45.883263 5024 patch_prober.go:29] interesting pod/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-nqkls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body= Nov 03 13:27:45.884715 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:45.883860 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-nqkls" podUID="19ea0757-f048-4f73-9091-cf3a0d86dd12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" Nov 03 13:27:45.944372 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:45.922693 5024 patch_prober.go:29] interesting pod/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-rzhn5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body= Nov 03 13:27:45.944372 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:45.922754 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-rzhn5" podUID="1727674a-caa9-4d85-8ee8-45bee8538927" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" Nov 03 13:27:45.944372 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:45.934761 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-ef15d972-openshift-default-58bd498f85-wxmks" Nov 03 13:27:45.963500 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:45.963462 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a62d22-69fbf9644c-2svhb" event={"ID":"8a432763-9702-42b0-97e4-c2294b4fe866","Type":"ContainerStarted","Data":"dd5e7a871b0d6b6501ca4ae7dc581eecb8a0fd5dbea8e967c5da4770f7d197b1"} Nov 03 13:27:45.966532 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:45.966501 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-a62d22-6746df564f-wv77r" event={"ID":"327a26e7-752e-4271-a3eb-1100182124a0","Type":"ContainerStarted","Data":"1f06529a64f7ea2f1850c3e5c9c4fdd24f17abfe3985bd263711c8f8ccd71ab2"} Nov 03 13:27:46.224128 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:46.224048 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w0-9add72/auth-test-model-9add72-kserve-54f88d5d5-bm2sv" Nov 03 13:27:46.224676 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:46.224655 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-a62d22-6746df564f-wv77r"] Nov 03 13:27:46.273379 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:46.273341 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-a62d22-69fbf9644c-2svhb"] Nov 03 13:27:46.775308 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:46.763267 5024 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-d3ca48/premium-w3-d3ca48-kserve-75ffc959b-zvjbc" podUID="add1e8b9-e7fd-4e4d-8d10-f97d0e2612ba" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 13:27:46.916309 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:46.897313 5024 patch_prober.go:29] interesting pod/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-nqkls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body= Nov 03 13:27:46.916309 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:46.897375 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-nqkls" podUID="19ea0757-f048-4f73-9091-cf3a0d86dd12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" Nov 03 13:27:46.916309 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:46.915163 5024 patch_prober.go:29] interesting pod/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-rzhn5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body= Nov 03 13:27:46.916309 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:46.915218 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-rzhn5" podUID="1727674a-caa9-4d85-8ee8-45bee8538927" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" Nov 03 13:27:47.117836 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:47.117392 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a62d22-69fbf9644c-2svhb" event={"ID":"8a432763-9702-42b0-97e4-c2294b4fe866","Type":"ContainerStarted","Data":"39e97a01dd324a8ca31f278514a2a0cac2b1a5985ac9a20d076b011a62c45afe"} Nov 03 13:27:47.117836 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:47.117534 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a62d22-69fbf9644c-2svhb" Nov 03 13:27:47.138540 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:47.138495 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-a62d22-6746df564f-wv77r" event={"ID":"327a26e7-752e-4271-a3eb-1100182124a0","Type":"ContainerStarted","Data":"6b3106d0694a65dc70dcf23058c45ceff9ec6af61a7d112e09571410cfc2e036"} Nov 03 13:27:47.143590 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:47.143515 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-a62d22-6746df564f-wv77r" Nov 03 13:27:47.222871 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:47.203511 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a62d22-69fbf9644c-2svhb" podStartSLOduration=6.203486157 podStartE2EDuration="6.203486157s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:27:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:27:47.153801464 +0000 UTC m=+1147.161992743" watchObservedRunningTime="2026-11-03 13:27:47.203486157 +0000 UTC m=+1147.211677435" Nov 03 13:27:47.254790 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:47.254710 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-a62d22-6746df564f-wv77r" podStartSLOduration=7.254690007 podStartE2EDuration="7.254690007s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:27:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:27:47.17964893 +0000 UTC m=+1147.187840208" watchObservedRunningTime="2026-11-03 13:27:47.254690007 +0000 UTC m=+1147.262881281" Nov 03 13:27:47.894307 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:47.893913 5024 patch_prober.go:29] interesting pod/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-nqkls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body= Nov 03 13:27:47.894307 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:47.894071 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-nqkls" podUID="19ea0757-f048-4f73-9091-cf3a0d86dd12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" Nov 03 13:27:47.952308 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:47.925861 5024 patch_prober.go:29] interesting pod/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-rzhn5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body= Nov 03 13:27:47.952308 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:47.925936 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-rzhn5" podUID="1727674a-caa9-4d85-8ee8-45bee8538927" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" Nov 03 13:27:47.967062 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:47.967017 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-ef15d972-openshift-default-58bd498f85-wxmks"] Nov 03 13:27:48.254828 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:48.254764 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:27:48.255482 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:48.255452 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:27:48.255707 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:48.255671 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:27:48.455366 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:48.452026 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-ef15d972-openshift-default-58bd498f85-wxmks" event={"ID":"779855bc-fa52-4f8a-8ed8-2692103f9e0d","Type":"ContainerStarted","Data":"8dcf10ba75075dfd553326626a09249351c007d1d4be9858a5db889c9d4f2d17"} Nov 03 13:27:48.942324 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:48.938835 5024 patch_prober.go:29] interesting pod/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-rzhn5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body= Nov 03 13:27:48.942324 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:48.938902 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-rzhn5" podUID="1727674a-caa9-4d85-8ee8-45bee8538927" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" Nov 03 13:27:49.039405 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:49.016503 5024 patch_prober.go:29] interesting pod/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-nqkls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body= Nov 03 13:27:49.039405 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:49.016596 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-nqkls" podUID="19ea0757-f048-4f73-9091-cf3a0d86dd12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" Nov 03 13:27:49.063097 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:49.063057 5024 patch_prober.go:29] interesting pod/e2e-ait-c87701ad-openshift-default-58f78ff59-b664t container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 13:27:49.063362 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:49.063333 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-c87701ad-openshift-default-58f78ff59-b664t" podUID="c8a61536-4baf-4b92-ae7f-574b72135f52" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 13:27:49.540443 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:49.540398 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-ef15d972-openshift-default-58bd498f85-wxmks" event={"ID":"779855bc-fa52-4f8a-8ed8-2692103f9e0d","Type":"ContainerStarted","Data":"f223594702089d5e56df9b58ac17e5c2ee866020fc768e2ae0514c159e196200"} Nov 03 13:27:49.638800 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:49.638731 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-ef15d972-openshift-default-58bd498f85-wxmks" podStartSLOduration=6.638707046 podStartE2EDuration="6.638707046s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:27:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:27:49.574872602 +0000 UTC m=+1149.583063880" watchObservedRunningTime="2026-11-03 13:27:49.638707046 +0000 UTC m=+1149.646898324" Nov 03 13:27:49.898313 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:49.896698 5024 patch_prober.go:29] interesting pod/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-nqkls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body= Nov 03 13:27:49.898313 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:49.896766 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-nqkls" podUID="19ea0757-f048-4f73-9091-cf3a0d86dd12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" Nov 03 13:27:49.898313 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:49.897436 5024 patch_prober.go:29] interesting pod/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-rzhn5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body= Nov 03 13:27:49.898313 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:49.897477 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-rzhn5" podUID="1727674a-caa9-4d85-8ee8-45bee8538927" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" Nov 03 13:27:49.946309 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:49.943421 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-ef15d972-openshift-default-58bd498f85-wxmks" Nov 03 13:27:50.868036 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:50.867994 5024 patch_prober.go:29] interesting pod/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-nqkls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body= Nov 03 13:27:50.868562 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:50.868064 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-nqkls" podUID="19ea0757-f048-4f73-9091-cf3a0d86dd12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" Nov 03 13:27:50.896095 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:50.896052 5024 patch_prober.go:29] interesting pod/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-rzhn5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body= Nov 03 13:27:50.896270 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:50.896115 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-rzhn5" podUID="1727674a-caa9-4d85-8ee8-45bee8538927" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" Nov 03 13:27:50.943607 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:50.943570 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-ef15d972-openshift-default-58bd498f85-wxmks" Nov 03 13:27:51.565865 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:51.565831 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-ef15d972-openshift-default-58bd498f85-wxmks" Nov 03 13:27:51.566986 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:51.566965 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-ef15d972-openshift-default-58bd498f85-wxmks" Nov 03 13:27:51.882472 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:51.882423 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-nqkls" Nov 03 13:27:51.897617 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:51.897570 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-rzhn5" Nov 03 13:27:52.665969 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:52.665931 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-nqkls" Nov 03 13:27:52.667820 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:52.667790 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-nqkls" Nov 03 13:27:52.915822 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:52.915777 5024 patch_prober.go:29] interesting pod/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-rzhn5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 13:27:52.916321 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:52.915850 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-rzhn5" podUID="1727674a-caa9-4d85-8ee8-45bee8538927" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 13:27:52.954123 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:52.954084 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-rzhn5" Nov 03 13:27:52.956326 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:52.956254 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-rzhn5" Nov 03 13:27:53.215920 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:53.215781 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-ef15d972-75849948ff-bz4sc"] Nov 03 13:27:53.222516 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:53.222490 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-ef15d972-75849948ff-bz4sc" Nov 03 13:27:53.297067 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:53.296608 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/782e08fc-dc4b-4031-bf47-b6554db484f5-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-ef15d972-75849948ff-bz4sc\" (UID: \"782e08fc-dc4b-4031-bf47-b6554db484f5\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-ef15d972-75849948ff-bz4sc" Nov 03 13:27:53.297067 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:53.296787 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-smrph\" (UniqueName: \"kubernetes.io/projected/782e08fc-dc4b-4031-bf47-b6554db484f5-kube-api-access-smrph\") pod \"payload-pre-processing-e2e-derive-ef15d972-75849948ff-bz4sc\" (UID: \"782e08fc-dc4b-4031-bf47-b6554db484f5\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-ef15d972-75849948ff-bz4sc" Nov 03 13:27:53.320311 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:53.310530 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-ef15d972\"" Nov 03 13:27:53.320311 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:53.310705 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-ef15d972-dockercfg-wfw52\"" Nov 03 13:27:53.320311 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:53.316984 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-ef15d972-75849948ff-bz4sc"] Nov 03 13:27:53.429452 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:53.426873 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/782e08fc-dc4b-4031-bf47-b6554db484f5-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-ef15d972-75849948ff-bz4sc\" (UID: \"782e08fc-dc4b-4031-bf47-b6554db484f5\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-ef15d972-75849948ff-bz4sc" Nov 03 13:27:53.429452 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:53.427112 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-smrph\" (UniqueName: \"kubernetes.io/projected/782e08fc-dc4b-4031-bf47-b6554db484f5-kube-api-access-smrph\") pod \"payload-pre-processing-e2e-derive-ef15d972-75849948ff-bz4sc\" (UID: \"782e08fc-dc4b-4031-bf47-b6554db484f5\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-ef15d972-75849948ff-bz4sc" Nov 03 13:27:53.441703 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:53.441655 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/782e08fc-dc4b-4031-bf47-b6554db484f5-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-ef15d972-75849948ff-bz4sc\" (UID: \"782e08fc-dc4b-4031-bf47-b6554db484f5\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-ef15d972-75849948ff-bz4sc" Nov 03 13:27:53.580374 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:53.578992 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-smrph\" (UniqueName: \"kubernetes.io/projected/782e08fc-dc4b-4031-bf47-b6554db484f5-kube-api-access-smrph\") pod \"payload-pre-processing-e2e-derive-ef15d972-75849948ff-bz4sc\" (UID: \"782e08fc-dc4b-4031-bf47-b6554db484f5\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-ef15d972-75849948ff-bz4sc" Nov 03 13:27:53.712197 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:53.711627 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-ef15d972-75849948ff-bz4sc" Nov 03 13:27:54.037685 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:54.037657 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-ef15d972-75849948ff-bz4sc"] Nov 03 13:27:54.041022 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:27:54.040986 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod782e08fc_dc4b_4031_bf47_b6554db484f5.slice/crio-2dcad247cbf5e142770cf7914d0a2982b8bff0c1aa76fe89cbe41ddd8db78237 WatchSource:0}: Error finding container 2dcad247cbf5e142770cf7914d0a2982b8bff0c1aa76fe89cbe41ddd8db78237: Status 404 returned error can't find the container with id 2dcad247cbf5e142770cf7914d0a2982b8bff0c1aa76fe89cbe41ddd8db78237 Nov 03 13:27:54.618357 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:54.618267 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-ef15d972-75849948ff-bz4sc" event={"ID":"782e08fc-dc4b-4031-bf47-b6554db484f5","Type":"ContainerStarted","Data":"67311f6e9516eb710765afef713dab38b041c967cfcb615815d5b4b29f4841ea"} Nov 03 13:27:54.618560 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:54.618390 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-ef15d972-75849948ff-bz4sc" event={"ID":"782e08fc-dc4b-4031-bf47-b6554db484f5","Type":"ContainerStarted","Data":"2dcad247cbf5e142770cf7914d0a2982b8bff0c1aa76fe89cbe41ddd8db78237"} Nov 03 13:27:54.618560 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:54.618435 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-ef15d972-75849948ff-bz4sc" Nov 03 13:27:54.654031 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:54.653961 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-ef15d972-75849948ff-bz4sc" podStartSLOduration=1.653942422 podStartE2EDuration="1.653942422s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:27:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:27:54.647992438 +0000 UTC m=+1154.656183715" watchObservedRunningTime="2026-11-03 13:27:54.653942422 +0000 UTC m=+1154.662133699" Nov 03 13:27:58.480322 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:58.479330 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a62d22-69fbf9644c-2svhb" Nov 03 13:27:58.513691 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:58.512616 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-a62d22-6746df564f-wv77r" Nov 03 13:27:59.467310 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:59.466261 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-ef1693/payload-processing-external-model-e2e-extmod-a-ef1693-5578jx92q"] Nov 03 13:27:59.517650 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:59.517615 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-ef1693/payload-processing-external-model-e2e-extmod-a-ef1693-5578jx92q" Nov 03 13:27:59.530558 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:59.530524 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-ef1693\"/\"agc-cred-5e1bd9e90045\"" Nov 03 13:27:59.531102 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:59.531080 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-ef1693\"/\"payload-processing-external-model-plugins-e2e-extmod-a-ef1693\"" Nov 03 13:27:59.531412 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:59.531396 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-ef1693\"/\"agc-cred-4d65870da959\"" Nov 03 13:27:59.531621 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:59.531607 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-ef1693\"/\"payload-processing-external-model-e2e--393fd9ed-dockercfg-fnc52\"" Nov 03 13:27:59.531851 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:59.531837 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-ef1693\"/\"openshift-service-ca.crt\"" Nov 03 13:27:59.540140 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:59.533593 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-ef1693\"/\"kube-root-ca.crt\"" Nov 03 13:27:59.574893 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:59.574847 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hvfzz\" (UniqueName: \"kubernetes.io/projected/5d145099-3543-4d68-a150-a5970cea2829-kube-api-access-hvfzz\") pod \"payload-processing-external-model-e2e-extmod-a-ef1693-5578jx92q\" (UID: \"5d145099-3543-4d68-a150-a5970cea2829\") " pod="ai-tenant-e2e-extmod-a-ef1693/payload-processing-external-model-e2e-extmod-a-ef1693-5578jx92q" Nov 03 13:27:59.586538 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:59.586502 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5d145099-3543-4d68-a150-a5970cea2829-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-ef1693-5578jx92q\" (UID: \"5d145099-3543-4d68-a150-a5970cea2829\") " pod="ai-tenant-e2e-extmod-a-ef1693/payload-processing-external-model-e2e-extmod-a-ef1693-5578jx92q" Nov 03 13:27:59.587142 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:59.587115 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5d145099-3543-4d68-a150-a5970cea2829-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-ef1693-5578jx92q\" (UID: \"5d145099-3543-4d68-a150-a5970cea2829\") " pod="ai-tenant-e2e-extmod-a-ef1693/payload-processing-external-model-e2e-extmod-a-ef1693-5578jx92q" Nov 03 13:27:59.588018 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:59.587995 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d145099-3543-4d68-a150-a5970cea2829-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-ef1693-5578jx92q\" (UID: \"5d145099-3543-4d68-a150-a5970cea2829\") " pod="ai-tenant-e2e-extmod-a-ef1693/payload-processing-external-model-e2e-extmod-a-ef1693-5578jx92q" Nov 03 13:27:59.640358 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:59.638173 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-ef1693\"/\"routing-overlay\"" Nov 03 13:27:59.688375 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:59.688334 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-ef1693/payload-processing-external-model-e2e-extmod-a-ef1693-5578jx92q"] Nov 03 13:27:59.691100 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:59.691054 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvfzz\" (UniqueName: \"kubernetes.io/projected/5d145099-3543-4d68-a150-a5970cea2829-kube-api-access-hvfzz\") pod \"payload-processing-external-model-e2e-extmod-a-ef1693-5578jx92q\" (UID: \"5d145099-3543-4d68-a150-a5970cea2829\") " pod="ai-tenant-e2e-extmod-a-ef1693/payload-processing-external-model-e2e-extmod-a-ef1693-5578jx92q" Nov 03 13:27:59.691903 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:59.691874 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5d145099-3543-4d68-a150-a5970cea2829-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-ef1693-5578jx92q\" (UID: \"5d145099-3543-4d68-a150-a5970cea2829\") " pod="ai-tenant-e2e-extmod-a-ef1693/payload-processing-external-model-e2e-extmod-a-ef1693-5578jx92q" Nov 03 13:27:59.705566 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:59.705531 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5d145099-3543-4d68-a150-a5970cea2829-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-ef1693-5578jx92q\" (UID: \"5d145099-3543-4d68-a150-a5970cea2829\") " pod="ai-tenant-e2e-extmod-a-ef1693/payload-processing-external-model-e2e-extmod-a-ef1693-5578jx92q" Nov 03 13:27:59.715637 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:59.706640 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5d145099-3543-4d68-a150-a5970cea2829-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-ef1693-5578jx92q\" (UID: \"5d145099-3543-4d68-a150-a5970cea2829\") " pod="ai-tenant-e2e-extmod-a-ef1693/payload-processing-external-model-e2e-extmod-a-ef1693-5578jx92q" Nov 03 13:27:59.716112 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:59.716082 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d145099-3543-4d68-a150-a5970cea2829-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-ef1693-5578jx92q\" (UID: \"5d145099-3543-4d68-a150-a5970cea2829\") " pod="ai-tenant-e2e-extmod-a-ef1693/payload-processing-external-model-e2e-extmod-a-ef1693-5578jx92q" Nov 03 13:27:59.738797 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:59.738651 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d145099-3543-4d68-a150-a5970cea2829-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-ef1693-5578jx92q\" (UID: \"5d145099-3543-4d68-a150-a5970cea2829\") " pod="ai-tenant-e2e-extmod-a-ef1693/payload-processing-external-model-e2e-extmod-a-ef1693-5578jx92q" Nov 03 13:27:59.742014 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:59.741974 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5d145099-3543-4d68-a150-a5970cea2829-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-ef1693-5578jx92q\" (UID: \"5d145099-3543-4d68-a150-a5970cea2829\") " pod="ai-tenant-e2e-extmod-a-ef1693/payload-processing-external-model-e2e-extmod-a-ef1693-5578jx92q" Nov 03 13:27:59.742864 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:59.742790 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hvfzz\" (UniqueName: \"kubernetes.io/projected/5d145099-3543-4d68-a150-a5970cea2829-kube-api-access-hvfzz\") pod \"payload-processing-external-model-e2e-extmod-a-ef1693-5578jx92q\" (UID: \"5d145099-3543-4d68-a150-a5970cea2829\") " pod="ai-tenant-e2e-extmod-a-ef1693/payload-processing-external-model-e2e-extmod-a-ef1693-5578jx92q" Nov 03 13:27:59.876627 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:27:59.876581 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-ef1693/payload-processing-external-model-e2e-extmod-a-ef1693-5578jx92q" Nov 03 13:28:00.775013 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:00.774971 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-ef1693/payload-processing-external-model-e2e-extmod-a-ef1693-5578jx92q"] Nov 03 13:28:00.966579 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:00.966544 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-c87701ad-openshift-default-58f78ff59-b664t" Nov 03 13:28:01.128400 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:01.099627 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c8a61536-4baf-4b92-ae7f-574b72135f52-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8a61536-4baf-4b92-ae7f-574b72135f52-istio-podinfo\") pod \"c8a61536-4baf-4b92-ae7f-574b72135f52\" (UID: \"c8a61536-4baf-4b92-ae7f-574b72135f52\") " Nov 03 13:28:01.128700 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:01.128670 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c8a61536-4baf-4b92-ae7f-574b72135f52-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8a61536-4baf-4b92-ae7f-574b72135f52-istiod-ca-cert\") pod \"c8a61536-4baf-4b92-ae7f-574b72135f52\" (UID: \"c8a61536-4baf-4b92-ae7f-574b72135f52\") " Nov 03 13:28:01.134462 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:01.134414 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8a61536-4baf-4b92-ae7f-574b72135f52-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8a61536-4baf-4b92-ae7f-574b72135f52-istio-envoy\") pod \"c8a61536-4baf-4b92-ae7f-574b72135f52\" (UID: \"c8a61536-4baf-4b92-ae7f-574b72135f52\") " Nov 03 13:28:01.134709 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:01.134691 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8a61536-4baf-4b92-ae7f-574b72135f52-istio-token\" (UniqueName: \"kubernetes.io/projected/c8a61536-4baf-4b92-ae7f-574b72135f52-istio-token\") pod \"c8a61536-4baf-4b92-ae7f-574b72135f52\" (UID: \"c8a61536-4baf-4b92-ae7f-574b72135f52\") " Nov 03 13:28:01.134883 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:01.134865 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8a61536-4baf-4b92-ae7f-574b72135f52-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8a61536-4baf-4b92-ae7f-574b72135f52-workload-certs\") pod \"c8a61536-4baf-4b92-ae7f-574b72135f52\" (UID: \"c8a61536-4baf-4b92-ae7f-574b72135f52\") " Nov 03 13:28:01.142315 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:01.135163 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8a61536-4baf-4b92-ae7f-574b72135f52-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8a61536-4baf-4b92-ae7f-574b72135f52-credential-socket\") pod \"c8a61536-4baf-4b92-ae7f-574b72135f52\" (UID: \"c8a61536-4baf-4b92-ae7f-574b72135f52\") " Nov 03 13:28:01.142648 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:01.142619 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8a61536-4baf-4b92-ae7f-574b72135f52-kube-api-access-vh9rq\" (UniqueName: \"kubernetes.io/projected/c8a61536-4baf-4b92-ae7f-574b72135f52-kube-api-access-vh9rq\") pod \"c8a61536-4baf-4b92-ae7f-574b72135f52\" (UID: \"c8a61536-4baf-4b92-ae7f-574b72135f52\") " Nov 03 13:28:01.142872 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:01.142852 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8a61536-4baf-4b92-ae7f-574b72135f52-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8a61536-4baf-4b92-ae7f-574b72135f52-workload-socket\") pod \"c8a61536-4baf-4b92-ae7f-574b72135f52\" (UID: \"c8a61536-4baf-4b92-ae7f-574b72135f52\") " Nov 03 13:28:01.146464 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:01.143142 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8a61536-4baf-4b92-ae7f-574b72135f52-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8a61536-4baf-4b92-ae7f-574b72135f52-istio-data\") pod \"c8a61536-4baf-4b92-ae7f-574b72135f52\" (UID: \"c8a61536-4baf-4b92-ae7f-574b72135f52\") " Nov 03 13:28:01.150619 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:01.136972 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8a61536-4baf-4b92-ae7f-574b72135f52-istiod-ca-cert" pod "c8a61536-4baf-4b92-ae7f-574b72135f52" (UID: "c8a61536-4baf-4b92-ae7f-574b72135f52"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:28:01.150619 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:01.137338 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8a61536-4baf-4b92-ae7f-574b72135f52-credential-socket" pod "c8a61536-4baf-4b92-ae7f-574b72135f52" (UID: "c8a61536-4baf-4b92-ae7f-574b72135f52"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:28:01.150619 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:01.140661 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8a61536-4baf-4b92-ae7f-574b72135f52-workload-certs" pod "c8a61536-4baf-4b92-ae7f-574b72135f52" (UID: "c8a61536-4baf-4b92-ae7f-574b72135f52"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:28:01.150619 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:01.147669 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8a61536-4baf-4b92-ae7f-574b72135f52-workload-socket" pod "c8a61536-4baf-4b92-ae7f-574b72135f52" (UID: "c8a61536-4baf-4b92-ae7f-574b72135f52"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:28:01.150619 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:01.149541 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8a61536-4baf-4b92-ae7f-574b72135f52-istio-data" pod "c8a61536-4baf-4b92-ae7f-574b72135f52" (UID: "c8a61536-4baf-4b92-ae7f-574b72135f52"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:28:01.151862 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:01.151825 5024 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8a61536-4baf-4b92-ae7f-574b72135f52-workload-socket\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:28:01.152054 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:01.152026 5024 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8a61536-4baf-4b92-ae7f-574b72135f52-istio-data\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:28:01.152214 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:01.152198 5024 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8a61536-4baf-4b92-ae7f-574b72135f52-istiod-ca-cert\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:28:01.161713 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:01.161687 5024 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8a61536-4baf-4b92-ae7f-574b72135f52-workload-certs\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:28:01.162008 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:01.161992 5024 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8a61536-4baf-4b92-ae7f-574b72135f52-credential-socket\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:28:01.171209 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:01.170992 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8a61536-4baf-4b92-ae7f-574b72135f52-istio-token" pod "c8a61536-4baf-4b92-ae7f-574b72135f52" (UID: "c8a61536-4baf-4b92-ae7f-574b72135f52"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:28:01.180492 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:01.180412 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8a61536-4baf-4b92-ae7f-574b72135f52-kube-api-access-vh9rq" pod "c8a61536-4baf-4b92-ae7f-574b72135f52" (UID: "c8a61536-4baf-4b92-ae7f-574b72135f52"). InnerVolumeSpecName "kube-api-access-vh9rq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:28:01.181372 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:01.180929 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8a61536-4baf-4b92-ae7f-574b72135f52-istio-envoy" pod "c8a61536-4baf-4b92-ae7f-574b72135f52" (UID: "c8a61536-4baf-4b92-ae7f-574b72135f52"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:28:01.187018 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:01.186990 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c8a61536-4baf-4b92-ae7f-574b72135f52-istio-podinfo" pod "c8a61536-4baf-4b92-ae7f-574b72135f52" (UID: "c8a61536-4baf-4b92-ae7f-574b72135f52"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 13:28:01.197946 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:01.197868 5024 generic.go:356] "Generic (PLEG): container finished" podID="c8a61536-4baf-4b92-ae7f-574b72135f52" containerID="8f3aed864cd2b5ea0e77670639cce8ee6a0e426732dc6796fb8e345a0b9be0ad" exitCode=0 Nov 03 13:28:01.200454 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:01.189592 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-ef1693/payload-processing-external-model-e2e-extmod-a-ef1693-5578jx92q" event={"ID":"5d145099-3543-4d68-a150-a5970cea2829","Type":"ContainerStarted","Data":"394206c46fad301c8b3c7031bec7db1237a7123f16abf866ecaa053687f4cfb9"} Nov 03 13:28:01.200700 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:01.200682 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-ef1693/payload-processing-external-model-e2e-extmod-a-ef1693-5578jx92q" Nov 03 13:28:01.200829 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:01.200813 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-ef1693/payload-processing-external-model-e2e-extmod-a-ef1693-5578jx92q" event={"ID":"5d145099-3543-4d68-a150-a5970cea2829","Type":"ContainerStarted","Data":"f132e4d18e786e4232bf4ed9d90583e1715108e19026ff090ebcb978d8a68d23"} Nov 03 13:28:01.200945 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:01.200926 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c87701ad-openshift-default-58f78ff59-b664t" event={"ID":"c8a61536-4baf-4b92-ae7f-574b72135f52","Type":"ContainerDied","Data":"8f3aed864cd2b5ea0e77670639cce8ee6a0e426732dc6796fb8e345a0b9be0ad"} Nov 03 13:28:01.201070 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:01.201054 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c87701ad-openshift-default-58f78ff59-b664t" event={"ID":"c8a61536-4baf-4b92-ae7f-574b72135f52","Type":"ContainerDied","Data":"2689a543dd1ce0e4b488895142909fa393584cc8ae8a9f4c91aa2501120d3712"} Nov 03 13:28:01.201191 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:01.201177 5024 scope.go:122] "RemoveContainer" containerID="8f3aed864cd2b5ea0e77670639cce8ee6a0e426732dc6796fb8e345a0b9be0ad" Nov 03 13:28:01.201435 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:01.198387 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-c87701ad-openshift-default-58f78ff59-b664t" Nov 03 13:28:01.252861 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:01.252834 5024 scope.go:122] "RemoveContainer" containerID="8f3aed864cd2b5ea0e77670639cce8ee6a0e426732dc6796fb8e345a0b9be0ad" Nov 03 13:28:01.253176 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:28:01.253152 5024 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f3aed864cd2b5ea0e77670639cce8ee6a0e426732dc6796fb8e345a0b9be0ad\": container with ID starting with 8f3aed864cd2b5ea0e77670639cce8ee6a0e426732dc6796fb8e345a0b9be0ad not found: ID does not exist" containerID="8f3aed864cd2b5ea0e77670639cce8ee6a0e426732dc6796fb8e345a0b9be0ad" Nov 03 13:28:01.253265 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:01.253183 5024 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f3aed864cd2b5ea0e77670639cce8ee6a0e426732dc6796fb8e345a0b9be0ad"} err="failed to get container status \"8f3aed864cd2b5ea0e77670639cce8ee6a0e426732dc6796fb8e345a0b9be0ad\": rpc error: code = NotFound desc = could not find container \"8f3aed864cd2b5ea0e77670639cce8ee6a0e426732dc6796fb8e345a0b9be0ad\": container with ID starting with 8f3aed864cd2b5ea0e77670639cce8ee6a0e426732dc6796fb8e345a0b9be0ad not found: ID does not exist" Nov 03 13:28:01.264278 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:01.264243 5024 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8a61536-4baf-4b92-ae7f-574b72135f52-istio-podinfo\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:28:01.264278 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:01.264277 5024 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8a61536-4baf-4b92-ae7f-574b72135f52-istio-envoy\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:28:01.264524 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:01.264315 5024 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8a61536-4baf-4b92-ae7f-574b72135f52-istio-token\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:28:01.264524 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:01.264369 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vh9rq\" (UniqueName: \"kubernetes.io/projected/c8a61536-4baf-4b92-ae7f-574b72135f52-kube-api-access-vh9rq\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:28:01.269433 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:01.269377 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-ef1693/payload-processing-external-model-e2e-extmod-a-ef1693-5578jx92q" podStartSLOduration=2.269362293 podStartE2EDuration="2.269362293s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:27:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:28:01.232353452 +0000 UTC m=+1161.240544725" watchObservedRunningTime="2026-11-03 13:28:01.269362293 +0000 UTC m=+1161.277553571" Nov 03 13:28:01.270088 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:01.270071 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-c87701ad-openshift-default-58f78ff59-b664t"] Nov 03 13:28:01.278732 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:01.278690 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-c87701ad-openshift-default-58f78ff59-b664t"] Nov 03 13:28:02.123404 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:02.123369 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c8a61536-4baf-4b92-ae7f-574b72135f52" path="/var/lib/kubelet/pods/c8a61536-4baf-4b92-ae7f-574b72135f52/volumes" Nov 03 13:28:03.252840 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:03.252806 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-4m57d"] Nov 03 13:28:03.253564 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:03.253547 5024 cpu_manager.go:414] "containerMap: removing container" podUID="c8a61536-4baf-4b92-ae7f-574b72135f52" containerName="istio-proxy" Nov 03 13:28:03.253564 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:03.253564 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8a61536-4baf-4b92-ae7f-574b72135f52" containerName="istio-proxy" Nov 03 13:28:03.253707 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:03.253697 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8a61536-4baf-4b92-ae7f-574b72135f52" containerName="istio-proxy" Nov 03 13:28:03.259052 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:03.259029 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-4m57d" Nov 03 13:28:03.293004 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:03.292967 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-4m57d"] Nov 03 13:28:03.350876 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:03.350830 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ca8b6af-59bf-4d09-ba65-9d79a4e2faba-istio-token\") pod \"e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-4m57d\" (UID: \"2ca8b6af-59bf-4d09-ba65-9d79a4e2faba\") " pod="openshift-ingress/e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-4m57d" Nov 03 13:28:03.350876 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:03.350890 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ca8b6af-59bf-4d09-ba65-9d79a4e2faba-istio-envoy\") pod \"e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-4m57d\" (UID: \"2ca8b6af-59bf-4d09-ba65-9d79a4e2faba\") " pod="openshift-ingress/e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-4m57d" Nov 03 13:28:03.351144 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:03.351112 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ca8b6af-59bf-4d09-ba65-9d79a4e2faba-istio-podinfo\") pod \"e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-4m57d\" (UID: \"2ca8b6af-59bf-4d09-ba65-9d79a4e2faba\") " pod="openshift-ingress/e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-4m57d" Nov 03 13:28:03.351206 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:03.351181 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ca8b6af-59bf-4d09-ba65-9d79a4e2faba-workload-certs\") pod \"e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-4m57d\" (UID: \"2ca8b6af-59bf-4d09-ba65-9d79a4e2faba\") " pod="openshift-ingress/e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-4m57d" Nov 03 13:28:03.351345 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:03.351323 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ca8b6af-59bf-4d09-ba65-9d79a4e2faba-istio-data\") pod \"e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-4m57d\" (UID: \"2ca8b6af-59bf-4d09-ba65-9d79a4e2faba\") " pod="openshift-ingress/e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-4m57d" Nov 03 13:28:03.351422 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:03.351406 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ca8b6af-59bf-4d09-ba65-9d79a4e2faba-workload-socket\") pod \"e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-4m57d\" (UID: \"2ca8b6af-59bf-4d09-ba65-9d79a4e2faba\") " pod="openshift-ingress/e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-4m57d" Nov 03 13:28:03.351468 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:03.351439 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ca8b6af-59bf-4d09-ba65-9d79a4e2faba-credential-socket\") pod \"e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-4m57d\" (UID: \"2ca8b6af-59bf-4d09-ba65-9d79a4e2faba\") " pod="openshift-ingress/e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-4m57d" Nov 03 13:28:03.351512 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:03.351480 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-np2bp\" (UniqueName: \"kubernetes.io/projected/2ca8b6af-59bf-4d09-ba65-9d79a4e2faba-kube-api-access-np2bp\") pod \"e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-4m57d\" (UID: \"2ca8b6af-59bf-4d09-ba65-9d79a4e2faba\") " pod="openshift-ingress/e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-4m57d" Nov 03 13:28:03.351572 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:03.351555 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ca8b6af-59bf-4d09-ba65-9d79a4e2faba-istiod-ca-cert\") pod \"e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-4m57d\" (UID: \"2ca8b6af-59bf-4d09-ba65-9d79a4e2faba\") " pod="openshift-ingress/e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-4m57d" Nov 03 13:28:03.453617 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:03.453575 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ca8b6af-59bf-4d09-ba65-9d79a4e2faba-istio-token\") pod \"e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-4m57d\" (UID: \"2ca8b6af-59bf-4d09-ba65-9d79a4e2faba\") " pod="openshift-ingress/e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-4m57d" Nov 03 13:28:03.453808 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:03.453638 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ca8b6af-59bf-4d09-ba65-9d79a4e2faba-istio-envoy\") pod \"e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-4m57d\" (UID: \"2ca8b6af-59bf-4d09-ba65-9d79a4e2faba\") " pod="openshift-ingress/e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-4m57d" Nov 03 13:28:03.453808 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:03.453780 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ca8b6af-59bf-4d09-ba65-9d79a4e2faba-istio-podinfo\") pod \"e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-4m57d\" (UID: \"2ca8b6af-59bf-4d09-ba65-9d79a4e2faba\") " pod="openshift-ingress/e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-4m57d" Nov 03 13:28:03.453915 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:03.453833 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ca8b6af-59bf-4d09-ba65-9d79a4e2faba-workload-certs\") pod \"e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-4m57d\" (UID: \"2ca8b6af-59bf-4d09-ba65-9d79a4e2faba\") " pod="openshift-ingress/e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-4m57d" Nov 03 13:28:03.453915 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:03.453901 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ca8b6af-59bf-4d09-ba65-9d79a4e2faba-istio-data\") pod \"e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-4m57d\" (UID: \"2ca8b6af-59bf-4d09-ba65-9d79a4e2faba\") " pod="openshift-ingress/e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-4m57d" Nov 03 13:28:03.454026 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:03.453985 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ca8b6af-59bf-4d09-ba65-9d79a4e2faba-workload-socket\") pod \"e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-4m57d\" (UID: \"2ca8b6af-59bf-4d09-ba65-9d79a4e2faba\") " pod="openshift-ingress/e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-4m57d" Nov 03 13:28:03.454026 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:03.454011 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ca8b6af-59bf-4d09-ba65-9d79a4e2faba-credential-socket\") pod \"e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-4m57d\" (UID: \"2ca8b6af-59bf-4d09-ba65-9d79a4e2faba\") " pod="openshift-ingress/e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-4m57d" Nov 03 13:28:03.454128 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:03.454052 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-np2bp\" (UniqueName: \"kubernetes.io/projected/2ca8b6af-59bf-4d09-ba65-9d79a4e2faba-kube-api-access-np2bp\") pod \"e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-4m57d\" (UID: \"2ca8b6af-59bf-4d09-ba65-9d79a4e2faba\") " pod="openshift-ingress/e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-4m57d" Nov 03 13:28:03.454128 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:03.454095 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ca8b6af-59bf-4d09-ba65-9d79a4e2faba-istiod-ca-cert\") pod \"e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-4m57d\" (UID: \"2ca8b6af-59bf-4d09-ba65-9d79a4e2faba\") " pod="openshift-ingress/e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-4m57d" Nov 03 13:28:03.454271 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:03.454234 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ca8b6af-59bf-4d09-ba65-9d79a4e2faba-workload-certs\") pod \"e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-4m57d\" (UID: \"2ca8b6af-59bf-4d09-ba65-9d79a4e2faba\") " pod="openshift-ingress/e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-4m57d" Nov 03 13:28:03.454431 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:03.454275 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ca8b6af-59bf-4d09-ba65-9d79a4e2faba-istio-data\") pod \"e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-4m57d\" (UID: \"2ca8b6af-59bf-4d09-ba65-9d79a4e2faba\") " pod="openshift-ingress/e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-4m57d" Nov 03 13:28:03.454431 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:03.454355 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ca8b6af-59bf-4d09-ba65-9d79a4e2faba-workload-socket\") pod \"e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-4m57d\" (UID: \"2ca8b6af-59bf-4d09-ba65-9d79a4e2faba\") " pod="openshift-ingress/e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-4m57d" Nov 03 13:28:03.454846 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:03.454815 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ca8b6af-59bf-4d09-ba65-9d79a4e2faba-credential-socket\") pod \"e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-4m57d\" (UID: \"2ca8b6af-59bf-4d09-ba65-9d79a4e2faba\") " pod="openshift-ingress/e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-4m57d" Nov 03 13:28:03.454959 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:03.454939 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ca8b6af-59bf-4d09-ba65-9d79a4e2faba-istiod-ca-cert\") pod \"e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-4m57d\" (UID: \"2ca8b6af-59bf-4d09-ba65-9d79a4e2faba\") " pod="openshift-ingress/e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-4m57d" Nov 03 13:28:03.456160 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:03.456142 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ca8b6af-59bf-4d09-ba65-9d79a4e2faba-istio-envoy\") pod \"e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-4m57d\" (UID: \"2ca8b6af-59bf-4d09-ba65-9d79a4e2faba\") " pod="openshift-ingress/e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-4m57d" Nov 03 13:28:03.456403 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:03.456385 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ca8b6af-59bf-4d09-ba65-9d79a4e2faba-istio-podinfo\") pod \"e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-4m57d\" (UID: \"2ca8b6af-59bf-4d09-ba65-9d79a4e2faba\") " pod="openshift-ingress/e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-4m57d" Nov 03 13:28:03.468375 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:03.468348 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ca8b6af-59bf-4d09-ba65-9d79a4e2faba-istio-token\") pod \"e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-4m57d\" (UID: \"2ca8b6af-59bf-4d09-ba65-9d79a4e2faba\") " pod="openshift-ingress/e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-4m57d" Nov 03 13:28:03.468504 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:03.468434 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-np2bp\" (UniqueName: \"kubernetes.io/projected/2ca8b6af-59bf-4d09-ba65-9d79a4e2faba-kube-api-access-np2bp\") pod \"e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-4m57d\" (UID: \"2ca8b6af-59bf-4d09-ba65-9d79a4e2faba\") " pod="openshift-ingress/e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-4m57d" Nov 03 13:28:03.574670 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:03.574570 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-4m57d" Nov 03 13:28:04.287955 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:28:04.287919 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ca8b6af_59bf_4d09_ba65_9d79a4e2faba.slice/crio-55fa9f6c43cf17c6b841292678701d5ed1c6ab7327f27d67de9530af3c2f3145 WatchSource:0}: Error finding container 55fa9f6c43cf17c6b841292678701d5ed1c6ab7327f27d67de9530af3c2f3145: Status 404 returned error can't find the container with id 55fa9f6c43cf17c6b841292678701d5ed1c6ab7327f27d67de9530af3c2f3145 Nov 03 13:28:04.288299 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:04.288180 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-4m57d"] Nov 03 13:28:04.290982 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:04.290948 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:28:04.291083 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:04.291019 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:28:04.291083 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:04.291051 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:28:05.244111 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:05.244065 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-4m57d" event={"ID":"2ca8b6af-59bf-4d09-ba65-9d79a4e2faba","Type":"ContainerStarted","Data":"f32398773d95c537adc89160133a4f7cc75a42791b497690b585e6fde7e0a3ad"} Nov 03 13:28:05.244488 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:05.244461 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-4m57d" event={"ID":"2ca8b6af-59bf-4d09-ba65-9d79a4e2faba","Type":"ContainerStarted","Data":"55fa9f6c43cf17c6b841292678701d5ed1c6ab7327f27d67de9530af3c2f3145"} Nov 03 13:28:05.309613 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:05.309553 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-4m57d" podStartSLOduration=2.30953432 podStartE2EDuration="2.30953432s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:28:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:28:05.301014098 +0000 UTC m=+1165.309205378" watchObservedRunningTime="2026-11-03 13:28:05.30953432 +0000 UTC m=+1165.317725598" Nov 03 13:28:05.588375 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:05.584359 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-4m57d" Nov 03 13:28:05.588375 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:05.585788 5024 patch_prober.go:29] interesting pod/e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-4m57d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body= Nov 03 13:28:05.588375 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:05.585863 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-4m57d" podUID="2ca8b6af-59bf-4d09-ba65-9d79a4e2faba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" Nov 03 13:28:05.667462 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:05.664420 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-ef15d972-75849948ff-bz4sc" Nov 03 13:28:06.594309 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:06.580752 5024 patch_prober.go:29] interesting pod/e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-4m57d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body= Nov 03 13:28:06.594309 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:06.580850 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-4m57d" podUID="2ca8b6af-59bf-4d09-ba65-9d79a4e2faba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" Nov 03 13:28:07.576778 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:07.576732 5024 patch_prober.go:29] interesting pod/e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-4m57d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body= Nov 03 13:28:07.577062 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:07.577031 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-4m57d" podUID="2ca8b6af-59bf-4d09-ba65-9d79a4e2faba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" Nov 03 13:28:08.576198 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:08.575679 5024 patch_prober.go:29] interesting pod/e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-4m57d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body= Nov 03 13:28:08.576198 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:08.575746 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-4m57d" podUID="2ca8b6af-59bf-4d09-ba65-9d79a4e2faba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" Nov 03 13:28:09.538353 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:28:09.538071 5024 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d7mwz" containerName="extractor" Nov 03 13:28:09.578315 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:09.576757 5024 patch_prober.go:29] interesting pod/e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-4m57d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body= Nov 03 13:28:09.578315 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:09.576834 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-4m57d" podUID="2ca8b6af-59bf-4d09-ba65-9d79a4e2faba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" Nov 03 13:28:10.319126 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:10.314393 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-1a18c8/payload-processing-external-model-e2e-extmod-b-1a18c8-8545rjtw7"] Nov 03 13:28:10.575817 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:10.575715 5024 patch_prober.go:29] interesting pod/e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-4m57d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body= Nov 03 13:28:10.575817 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:10.575777 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-4m57d" podUID="2ca8b6af-59bf-4d09-ba65-9d79a4e2faba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" Nov 03 13:28:11.631215 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:11.631180 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-4m57d" Nov 03 13:28:11.631739 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:11.631621 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-4m57d" Nov 03 13:28:11.632885 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:11.632859 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-4m57d" Nov 03 13:28:12.213131 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:12.213100 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-ef1693/payload-processing-external-model-e2e-extmod-a-ef1693-5578jx92q" Nov 03 13:28:15.343769 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:15.318368 5024 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-1a18c8/payload-processing-external-model-e2e-extmod-b-1a18c8-8545rjtw7" podUID="b9a50676-42a1-411a-a8a4-ed1ae870c843" containerName="payload-processing" containerID="cri-o://f7476c27e84e714bb39c4d4391a6d71317af1164fd26074aeaeb96559c480b22" gracePeriod=25 Nov 03 13:28:15.724523 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:15.724479 5024 generic.go:356] "Generic (PLEG): container finished" podID="b9a50676-42a1-411a-a8a4-ed1ae870c843" containerID="f7476c27e84e714bb39c4d4391a6d71317af1164fd26074aeaeb96559c480b22" exitCode=0 Nov 03 13:28:15.724704 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:15.724574 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-1a18c8/payload-processing-external-model-e2e-extmod-b-1a18c8-8545rjtw7" event={"ID":"b9a50676-42a1-411a-a8a4-ed1ae870c843","Type":"ContainerDied","Data":"f7476c27e84e714bb39c4d4391a6d71317af1164fd26074aeaeb96559c480b22"} Nov 03 13:28:15.844337 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:15.844273 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-1a18c8/payload-processing-external-model-e2e-extmod-b-1a18c8-8545rjtw7" Nov 03 13:28:15.905308 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:15.905213 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9a50676-42a1-411a-a8a4-ed1ae870c843-provider-credentials\" (UniqueName: \"kubernetes.io/projected/b9a50676-42a1-411a-a8a4-ed1ae870c843-provider-credentials\") pod \"b9a50676-42a1-411a-a8a4-ed1ae870c843\" (UID: \"b9a50676-42a1-411a-a8a4-ed1ae870c843\") " Nov 03 13:28:15.905490 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:15.905362 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9a50676-42a1-411a-a8a4-ed1ae870c843-kube-api-access-h6hqd\" (UniqueName: \"kubernetes.io/projected/b9a50676-42a1-411a-a8a4-ed1ae870c843-kube-api-access-h6hqd\") pod \"b9a50676-42a1-411a-a8a4-ed1ae870c843\" (UID: \"b9a50676-42a1-411a-a8a4-ed1ae870c843\") " Nov 03 13:28:15.905490 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:15.905479 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b9a50676-42a1-411a-a8a4-ed1ae870c843-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b9a50676-42a1-411a-a8a4-ed1ae870c843-routing-overlay-volume\") pod \"b9a50676-42a1-411a-a8a4-ed1ae870c843\" (UID: \"b9a50676-42a1-411a-a8a4-ed1ae870c843\") " Nov 03 13:28:15.905574 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:15.905561 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b9a50676-42a1-411a-a8a4-ed1ae870c843-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9a50676-42a1-411a-a8a4-ed1ae870c843-plugins-config-volume\") pod \"b9a50676-42a1-411a-a8a4-ed1ae870c843\" (UID: \"b9a50676-42a1-411a-a8a4-ed1ae870c843\") " Nov 03 13:28:15.911339 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:15.906831 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9a50676-42a1-411a-a8a4-ed1ae870c843-plugins-config-volume" pod "b9a50676-42a1-411a-a8a4-ed1ae870c843" (UID: "b9a50676-42a1-411a-a8a4-ed1ae870c843"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:28:15.911339 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:15.907884 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9a50676-42a1-411a-a8a4-ed1ae870c843-routing-overlay-volume" pod "b9a50676-42a1-411a-a8a4-ed1ae870c843" (UID: "b9a50676-42a1-411a-a8a4-ed1ae870c843"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:28:15.913075 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:15.913023 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9a50676-42a1-411a-a8a4-ed1ae870c843-kube-api-access-h6hqd" pod "b9a50676-42a1-411a-a8a4-ed1ae870c843" (UID: "b9a50676-42a1-411a-a8a4-ed1ae870c843"). InnerVolumeSpecName "kube-api-access-h6hqd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:28:15.913551 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:15.913526 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9a50676-42a1-411a-a8a4-ed1ae870c843-provider-credentials" pod "b9a50676-42a1-411a-a8a4-ed1ae870c843" (UID: "b9a50676-42a1-411a-a8a4-ed1ae870c843"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:28:16.008611 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:16.008507 5024 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b9a50676-42a1-411a-a8a4-ed1ae870c843-provider-credentials\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:28:16.008611 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:16.008550 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h6hqd\" (UniqueName: \"kubernetes.io/projected/b9a50676-42a1-411a-a8a4-ed1ae870c843-kube-api-access-h6hqd\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:28:16.008611 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:16.008565 5024 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b9a50676-42a1-411a-a8a4-ed1ae870c843-routing-overlay-volume\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:28:16.008611 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:16.008579 5024 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9a50676-42a1-411a-a8a4-ed1ae870c843-plugins-config-volume\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:28:16.389405 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:16.389355 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-cf53a4-7db4cb8dd7-ttdzp"] Nov 03 13:28:16.390597 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:16.390572 5024 cpu_manager.go:414] "containerMap: removing container" podUID="b9a50676-42a1-411a-a8a4-ed1ae870c843" containerName="payload-processing" Nov 03 13:28:16.390700 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:16.390598 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9a50676-42a1-411a-a8a4-ed1ae870c843" containerName="payload-processing" Nov 03 13:28:16.390802 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:16.390789 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9a50676-42a1-411a-a8a4-ed1ae870c843" containerName="payload-processing" Nov 03 13:28:16.395655 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:16.395630 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-cf53a4-6689d89fbf-9mxzh"] Nov 03 13:28:16.395817 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:16.395777 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-cf53a4-7db4cb8dd7-ttdzp" Nov 03 13:28:16.408783 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:16.408752 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-cf53a4-7db4cb8dd7-ttdzp"] Nov 03 13:28:16.409117 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:16.409054 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-cf53a4-6689d89fbf-9mxzh" Nov 03 13:28:16.411804 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:16.411740 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-cf53a4\"" Nov 03 13:28:16.416450 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:16.416424 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16ff3f43-3bfd-4dd9-9c9c-32ca8b9fc35e-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-cf53a4-7db4cb8dd7-ttdzp\" (UID: \"16ff3f43-3bfd-4dd9-9c9c-32ca8b9fc35e\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-cf53a4-7db4cb8dd7-ttdzp" Nov 03 13:28:16.416656 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:16.416636 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qkq6d\" (UniqueName: \"kubernetes.io/projected/16ff3f43-3bfd-4dd9-9c9c-32ca8b9fc35e-kube-api-access-qkq6d\") pod \"payload-pre-processing-e2e-extmod-b-cf53a4-7db4cb8dd7-ttdzp\" (UID: \"16ff3f43-3bfd-4dd9-9c9c-32ca8b9fc35e\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-cf53a4-7db4cb8dd7-ttdzp" Nov 03 13:28:16.419306 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:16.419264 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-cf53a4-6689d89fbf-9mxzh"] Nov 03 13:28:16.534103 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:16.530480 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hm4ms\" (UniqueName: \"kubernetes.io/projected/6b7e669a-c136-4731-bbdc-f877696d4c31-kube-api-access-hm4ms\") pod \"payload-processing-e2e-extmod-b-cf53a4-6689d89fbf-9mxzh\" (UID: \"6b7e669a-c136-4731-bbdc-f877696d4c31\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-cf53a4-6689d89fbf-9mxzh" Nov 03 13:28:16.534103 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:16.530683 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16ff3f43-3bfd-4dd9-9c9c-32ca8b9fc35e-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-cf53a4-7db4cb8dd7-ttdzp\" (UID: \"16ff3f43-3bfd-4dd9-9c9c-32ca8b9fc35e\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-cf53a4-7db4cb8dd7-ttdzp" Nov 03 13:28:16.534103 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:16.530751 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qkq6d\" (UniqueName: \"kubernetes.io/projected/16ff3f43-3bfd-4dd9-9c9c-32ca8b9fc35e-kube-api-access-qkq6d\") pod \"payload-pre-processing-e2e-extmod-b-cf53a4-7db4cb8dd7-ttdzp\" (UID: \"16ff3f43-3bfd-4dd9-9c9c-32ca8b9fc35e\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-cf53a4-7db4cb8dd7-ttdzp" Nov 03 13:28:16.534103 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:16.530832 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b7e669a-c136-4731-bbdc-f877696d4c31-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-cf53a4-6689d89fbf-9mxzh\" (UID: \"6b7e669a-c136-4731-bbdc-f877696d4c31\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-cf53a4-6689d89fbf-9mxzh" Nov 03 13:28:16.534103 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:16.531997 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16ff3f43-3bfd-4dd9-9c9c-32ca8b9fc35e-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-cf53a4-7db4cb8dd7-ttdzp\" (UID: \"16ff3f43-3bfd-4dd9-9c9c-32ca8b9fc35e\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-cf53a4-7db4cb8dd7-ttdzp" Nov 03 13:28:16.599312 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:16.587091 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qkq6d\" (UniqueName: \"kubernetes.io/projected/16ff3f43-3bfd-4dd9-9c9c-32ca8b9fc35e-kube-api-access-qkq6d\") pod \"payload-pre-processing-e2e-extmod-b-cf53a4-7db4cb8dd7-ttdzp\" (UID: \"16ff3f43-3bfd-4dd9-9c9c-32ca8b9fc35e\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-cf53a4-7db4cb8dd7-ttdzp" Nov 03 13:28:16.636541 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:16.636491 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b7e669a-c136-4731-bbdc-f877696d4c31-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-cf53a4-6689d89fbf-9mxzh\" (UID: \"6b7e669a-c136-4731-bbdc-f877696d4c31\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-cf53a4-6689d89fbf-9mxzh" Nov 03 13:28:16.648367 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:16.638870 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hm4ms\" (UniqueName: \"kubernetes.io/projected/6b7e669a-c136-4731-bbdc-f877696d4c31-kube-api-access-hm4ms\") pod \"payload-processing-e2e-extmod-b-cf53a4-6689d89fbf-9mxzh\" (UID: \"6b7e669a-c136-4731-bbdc-f877696d4c31\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-cf53a4-6689d89fbf-9mxzh" Nov 03 13:28:16.649242 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:16.638477 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b7e669a-c136-4731-bbdc-f877696d4c31-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-cf53a4-6689d89fbf-9mxzh\" (UID: \"6b7e669a-c136-4731-bbdc-f877696d4c31\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-cf53a4-6689d89fbf-9mxzh" Nov 03 13:28:16.710131 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:16.710086 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hm4ms\" (UniqueName: \"kubernetes.io/projected/6b7e669a-c136-4731-bbdc-f877696d4c31-kube-api-access-hm4ms\") pod \"payload-processing-e2e-extmod-b-cf53a4-6689d89fbf-9mxzh\" (UID: \"6b7e669a-c136-4731-bbdc-f877696d4c31\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-cf53a4-6689d89fbf-9mxzh" Nov 03 13:28:16.742308 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:16.738341 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-cf53a4-7db4cb8dd7-ttdzp" Nov 03 13:28:16.750979 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:16.750940 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-cf53a4-6689d89fbf-9mxzh" Nov 03 13:28:16.944083 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:16.943985 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-1a18c8/payload-processing-external-model-e2e-extmod-b-1a18c8-8545rjtw7" event={"ID":"b9a50676-42a1-411a-a8a4-ed1ae870c843","Type":"ContainerDied","Data":"f53482dd3d87d1be46535b54162ef64aff8d8f6fd1b9dcd6ee2458d91ffe4ed7"} Nov 03 13:28:16.944251 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:16.944140 5024 scope.go:122] "RemoveContainer" containerID="f7476c27e84e714bb39c4d4391a6d71317af1164fd26074aeaeb96559c480b22" Nov 03 13:28:16.944408 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:16.944387 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-1a18c8/payload-processing-external-model-e2e-extmod-b-1a18c8-8545rjtw7" Nov 03 13:28:17.063084 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:17.062995 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-1a18c8/payload-processing-external-model-e2e-extmod-b-1a18c8-8545rjtw7"] Nov 03 13:28:17.108310 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:17.102030 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-1a18c8/payload-processing-external-model-e2e-extmod-b-1a18c8-8545rjtw7"] Nov 03 13:28:17.436425 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:17.436394 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-cf53a4-7db4cb8dd7-ttdzp"] Nov 03 13:28:17.437096 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:28:17.437065 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16ff3f43_3bfd_4dd9_9c9c_32ca8b9fc35e.slice/crio-977baf1355b338bf06540abcfdfd84bcf69dce320cab268913457718059b3c21 WatchSource:0}: Error finding container 977baf1355b338bf06540abcfdfd84bcf69dce320cab268913457718059b3c21: Status 404 returned error can't find the container with id 977baf1355b338bf06540abcfdfd84bcf69dce320cab268913457718059b3c21 Nov 03 13:28:17.459866 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:28:17.459832 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b7e669a_c136_4731_bbdc_f877696d4c31.slice/crio-b1b641fa9bc04435f128203a9ca6a111b7dbe7f190964ca3142f898fc9928c04 WatchSource:0}: Error finding container b1b641fa9bc04435f128203a9ca6a111b7dbe7f190964ca3142f898fc9928c04: Status 404 returned error can't find the container with id b1b641fa9bc04435f128203a9ca6a111b7dbe7f190964ca3142f898fc9928c04 Nov 03 13:28:17.461201 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:17.461174 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-cf53a4-6689d89fbf-9mxzh"] Nov 03 13:28:17.954867 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:17.954822 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-cf53a4-7db4cb8dd7-ttdzp" event={"ID":"16ff3f43-3bfd-4dd9-9c9c-32ca8b9fc35e","Type":"ContainerStarted","Data":"e38f2354637a9e8afd8b4aad7308073dc18ae4e0ff7f33be660458f0926519dd"} Nov 03 13:28:17.955126 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:17.954883 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-cf53a4-7db4cb8dd7-ttdzp" event={"ID":"16ff3f43-3bfd-4dd9-9c9c-32ca8b9fc35e","Type":"ContainerStarted","Data":"977baf1355b338bf06540abcfdfd84bcf69dce320cab268913457718059b3c21"} Nov 03 13:28:17.955126 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:17.954928 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-cf53a4-7db4cb8dd7-ttdzp" Nov 03 13:28:17.956401 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:17.956378 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-cf53a4-6689d89fbf-9mxzh" event={"ID":"6b7e669a-c136-4731-bbdc-f877696d4c31","Type":"ContainerStarted","Data":"7110dcebe620eba41e23d2995f204d44fbdff5aa04b771a7d5544946480130c3"} Nov 03 13:28:17.956513 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:17.956415 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-cf53a4-6689d89fbf-9mxzh" event={"ID":"6b7e669a-c136-4731-bbdc-f877696d4c31","Type":"ContainerStarted","Data":"b1b641fa9bc04435f128203a9ca6a111b7dbe7f190964ca3142f898fc9928c04"} Nov 03 13:28:17.956513 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:17.956459 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-cf53a4-6689d89fbf-9mxzh" Nov 03 13:28:17.979350 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:17.979280 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-cf53a4-7db4cb8dd7-ttdzp" podStartSLOduration=1.979265284 podStartE2EDuration="1.979265284s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:28:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:28:17.976204795 +0000 UTC m=+1177.984396064" watchObservedRunningTime="2026-11-03 13:28:17.979265284 +0000 UTC m=+1177.987456751" Nov 03 13:28:18.004178 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:18.004117 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-cf53a4-6689d89fbf-9mxzh" podStartSLOduration=2.004099841 podStartE2EDuration="2.004099841s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:28:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:28:18.003408696 +0000 UTC m=+1178.011599978" watchObservedRunningTime="2026-11-03 13:28:18.004099841 +0000 UTC m=+1178.012291118" Nov 03 13:28:18.111731 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:18.111697 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b9a50676-42a1-411a-a8a4-ed1ae870c843" path="/var/lib/kubelet/pods/b9a50676-42a1-411a-a8a4-ed1ae870c843/volumes" Nov 03 13:28:27.405398 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:27.405344 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-cd4b9r87m"] Nov 03 13:28:28.347360 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:28.335156 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-ef15d972-75849948ff-bz4sc"] Nov 03 13:28:29.000534 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:29.000485 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-cf53a4-7db4cb8dd7-ttdzp" Nov 03 13:28:29.002891 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:29.002863 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-cf53a4-6689d89fbf-9mxzh" Nov 03 13:28:32.408751 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:32.408709 5024 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-cd4b9r87m" podUID="138f6437-e9a2-4270-8583-220e170165f4" containerName="payload-processing" containerID="cri-o://fd3e90f4a4e8be89850155ac1af02d6da28b634133713b2a7855e9ba8ef41e06" gracePeriod=25 Nov 03 13:28:32.857744 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:32.857545 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-cd4b9r87m" Nov 03 13:28:32.866455 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:32.866418 5024 generic.go:356] "Generic (PLEG): container finished" podID="138f6437-e9a2-4270-8583-220e170165f4" containerID="fd3e90f4a4e8be89850155ac1af02d6da28b634133713b2a7855e9ba8ef41e06" exitCode=0 Nov 03 13:28:32.866929 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:32.866904 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-cd4b9r87m" event={"ID":"138f6437-e9a2-4270-8583-220e170165f4","Type":"ContainerDied","Data":"fd3e90f4a4e8be89850155ac1af02d6da28b634133713b2a7855e9ba8ef41e06"} Nov 03 13:28:32.867156 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:32.867139 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-cd4b9r87m" event={"ID":"138f6437-e9a2-4270-8583-220e170165f4","Type":"ContainerDied","Data":"8dc917343b7c0495f1e756efafac70fbcf6dcc336619d841de046f79d94f1cff"} Nov 03 13:28:32.867322 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:32.867304 5024 scope.go:122] "RemoveContainer" containerID="fd3e90f4a4e8be89850155ac1af02d6da28b634133713b2a7855e9ba8ef41e06" Nov 03 13:28:32.868140 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:32.868121 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-cd4b9r87m" Nov 03 13:28:32.892266 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:32.890376 5024 scope.go:122] "RemoveContainer" containerID="fd3e90f4a4e8be89850155ac1af02d6da28b634133713b2a7855e9ba8ef41e06" Nov 03 13:28:32.892266 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:28:32.891959 5024 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd3e90f4a4e8be89850155ac1af02d6da28b634133713b2a7855e9ba8ef41e06\": container with ID starting with fd3e90f4a4e8be89850155ac1af02d6da28b634133713b2a7855e9ba8ef41e06 not found: ID does not exist" containerID="fd3e90f4a4e8be89850155ac1af02d6da28b634133713b2a7855e9ba8ef41e06" Nov 03 13:28:32.892266 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:32.892001 5024 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd3e90f4a4e8be89850155ac1af02d6da28b634133713b2a7855e9ba8ef41e06"} err="failed to get container status \"fd3e90f4a4e8be89850155ac1af02d6da28b634133713b2a7855e9ba8ef41e06\": rpc error: code = NotFound desc = could not find container \"fd3e90f4a4e8be89850155ac1af02d6da28b634133713b2a7855e9ba8ef41e06\": container with ID starting with fd3e90f4a4e8be89850155ac1af02d6da28b634133713b2a7855e9ba8ef41e06 not found: ID does not exist" Nov 03 13:28:32.976801 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:32.976694 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/138f6437-e9a2-4270-8583-220e170165f4-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/138f6437-e9a2-4270-8583-220e170165f4-routing-overlay-volume\") pod \"138f6437-e9a2-4270-8583-220e170165f4\" (UID: \"138f6437-e9a2-4270-8583-220e170165f4\") " Nov 03 13:28:32.977012 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:32.976976 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/138f6437-e9a2-4270-8583-220e170165f4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/138f6437-e9a2-4270-8583-220e170165f4-plugins-config-volume\") pod \"138f6437-e9a2-4270-8583-220e170165f4\" (UID: \"138f6437-e9a2-4270-8583-220e170165f4\") " Nov 03 13:28:32.977131 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:32.977113 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/138f6437-e9a2-4270-8583-220e170165f4-kube-api-access-dxlqz\" (UniqueName: \"kubernetes.io/projected/138f6437-e9a2-4270-8583-220e170165f4-kube-api-access-dxlqz\") pod \"138f6437-e9a2-4270-8583-220e170165f4\" (UID: \"138f6437-e9a2-4270-8583-220e170165f4\") " Nov 03 13:28:32.977206 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:32.977131 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/138f6437-e9a2-4270-8583-220e170165f4-routing-overlay-volume" pod "138f6437-e9a2-4270-8583-220e170165f4" (UID: "138f6437-e9a2-4270-8583-220e170165f4"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:28:32.977267 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:32.977219 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/138f6437-e9a2-4270-8583-220e170165f4-provider-credentials\" (UniqueName: \"kubernetes.io/projected/138f6437-e9a2-4270-8583-220e170165f4-provider-credentials\") pod \"138f6437-e9a2-4270-8583-220e170165f4\" (UID: \"138f6437-e9a2-4270-8583-220e170165f4\") " Nov 03 13:28:32.977570 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:32.977540 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/138f6437-e9a2-4270-8583-220e170165f4-plugins-config-volume" pod "138f6437-e9a2-4270-8583-220e170165f4" (UID: "138f6437-e9a2-4270-8583-220e170165f4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:28:32.978325 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:32.978266 5024 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/138f6437-e9a2-4270-8583-220e170165f4-plugins-config-volume\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:28:32.978432 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:32.978328 5024 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/138f6437-e9a2-4270-8583-220e170165f4-routing-overlay-volume\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:28:32.980457 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:32.980413 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/138f6437-e9a2-4270-8583-220e170165f4-kube-api-access-dxlqz" pod "138f6437-e9a2-4270-8583-220e170165f4" (UID: "138f6437-e9a2-4270-8583-220e170165f4"). InnerVolumeSpecName "kube-api-access-dxlqz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:28:32.980655 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:32.980632 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/138f6437-e9a2-4270-8583-220e170165f4-provider-credentials" pod "138f6437-e9a2-4270-8583-220e170165f4" (UID: "138f6437-e9a2-4270-8583-220e170165f4"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:28:33.080273 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:33.080230 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dxlqz\" (UniqueName: \"kubernetes.io/projected/138f6437-e9a2-4270-8583-220e170165f4-kube-api-access-dxlqz\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:28:33.080273 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:33.080269 5024 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/138f6437-e9a2-4270-8583-220e170165f4-provider-credentials\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:28:33.259545 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:33.259399 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-cd4b9r87m"] Nov 03 13:28:33.305530 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:33.305441 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-cd4b9r87m"] Nov 03 13:28:33.349453 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:33.349380 5024 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-ef15d972-75849948ff-bz4sc" podUID="782e08fc-dc4b-4031-bf47-b6554db484f5" containerName="payload-pre-processing" containerID="cri-o://67311f6e9516eb710765afef713dab38b041c967cfcb615815d5b4b29f4841ea" gracePeriod=25 Nov 03 13:28:33.707648 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:33.704759 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-ef15d972-75849948ff-bz4sc" Nov 03 13:28:33.804662 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:33.804576 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/782e08fc-dc4b-4031-bf47-b6554db484f5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/782e08fc-dc4b-4031-bf47-b6554db484f5-plugins-config-volume\") pod \"782e08fc-dc4b-4031-bf47-b6554db484f5\" (UID: \"782e08fc-dc4b-4031-bf47-b6554db484f5\") " Nov 03 13:28:33.804848 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:33.804710 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/782e08fc-dc4b-4031-bf47-b6554db484f5-kube-api-access-smrph\" (UniqueName: \"kubernetes.io/projected/782e08fc-dc4b-4031-bf47-b6554db484f5-kube-api-access-smrph\") pod \"782e08fc-dc4b-4031-bf47-b6554db484f5\" (UID: \"782e08fc-dc4b-4031-bf47-b6554db484f5\") " Nov 03 13:28:33.805063 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:33.805028 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/782e08fc-dc4b-4031-bf47-b6554db484f5-plugins-config-volume" pod "782e08fc-dc4b-4031-bf47-b6554db484f5" (UID: "782e08fc-dc4b-4031-bf47-b6554db484f5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:28:33.805452 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:33.805424 5024 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/782e08fc-dc4b-4031-bf47-b6554db484f5-plugins-config-volume\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:28:33.808098 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:33.808055 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/782e08fc-dc4b-4031-bf47-b6554db484f5-kube-api-access-smrph" pod "782e08fc-dc4b-4031-bf47-b6554db484f5" (UID: "782e08fc-dc4b-4031-bf47-b6554db484f5"). InnerVolumeSpecName "kube-api-access-smrph". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:28:33.897702 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:33.884274 5024 generic.go:356] "Generic (PLEG): container finished" podID="782e08fc-dc4b-4031-bf47-b6554db484f5" containerID="67311f6e9516eb710765afef713dab38b041c967cfcb615815d5b4b29f4841ea" exitCode=0 Nov 03 13:28:33.897702 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:33.884503 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-ef15d972-75849948ff-bz4sc" event={"ID":"782e08fc-dc4b-4031-bf47-b6554db484f5","Type":"ContainerDied","Data":"67311f6e9516eb710765afef713dab38b041c967cfcb615815d5b4b29f4841ea"} Nov 03 13:28:33.897702 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:33.884560 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-ef15d972-75849948ff-bz4sc" event={"ID":"782e08fc-dc4b-4031-bf47-b6554db484f5","Type":"ContainerDied","Data":"2dcad247cbf5e142770cf7914d0a2982b8bff0c1aa76fe89cbe41ddd8db78237"} Nov 03 13:28:33.897702 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:33.884594 5024 scope.go:122] "RemoveContainer" containerID="67311f6e9516eb710765afef713dab38b041c967cfcb615815d5b4b29f4841ea" Nov 03 13:28:33.897702 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:33.884724 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-ef15d972-75849948ff-bz4sc" Nov 03 13:28:33.932310 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:33.909767 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-smrph\" (UniqueName: \"kubernetes.io/projected/782e08fc-dc4b-4031-bf47-b6554db484f5-kube-api-access-smrph\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:28:33.947327 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:33.946479 5024 scope.go:122] "RemoveContainer" containerID="67311f6e9516eb710765afef713dab38b041c967cfcb615815d5b4b29f4841ea" Nov 03 13:28:33.947518 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:28:33.947451 5024 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"67311f6e9516eb710765afef713dab38b041c967cfcb615815d5b4b29f4841ea\": container with ID starting with 67311f6e9516eb710765afef713dab38b041c967cfcb615815d5b4b29f4841ea not found: ID does not exist" containerID="67311f6e9516eb710765afef713dab38b041c967cfcb615815d5b4b29f4841ea" Nov 03 13:28:33.947518 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:33.947487 5024 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67311f6e9516eb710765afef713dab38b041c967cfcb615815d5b4b29f4841ea"} err="failed to get container status \"67311f6e9516eb710765afef713dab38b041c967cfcb615815d5b4b29f4841ea\": rpc error: code = NotFound desc = could not find container \"67311f6e9516eb710765afef713dab38b041c967cfcb615815d5b4b29f4841ea\": container with ID starting with 67311f6e9516eb710765afef713dab38b041c967cfcb615815d5b4b29f4841ea not found: ID does not exist" Nov 03 13:28:34.219531 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:34.219486 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="138f6437-e9a2-4270-8583-220e170165f4" path="/var/lib/kubelet/pods/138f6437-e9a2-4270-8583-220e170165f4/volumes" Nov 03 13:28:34.232706 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:34.232657 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-ef15d972-75849948ff-bz4sc"] Nov 03 13:28:34.240400 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:34.240326 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-ef15d972-75849948ff-bz4sc"] Nov 03 13:28:34.321308 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:34.247388 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-ef15d972-openshift-default-58bd498f85-wxmks"] Nov 03 13:28:34.321308 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:34.308954 5024 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-ef15d972-openshift-default-58bd498f85-wxmks" podUID="779855bc-fa52-4f8a-8ed8-2692103f9e0d" containerName="istio-proxy" containerID="cri-o://f223594702089d5e56df9b58ac17e5c2ee866020fc768e2ae0514c159e196200" gracePeriod=30 Nov 03 13:28:36.112727 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:36.112688 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="782e08fc-dc4b-4031-bf47-b6554db484f5" path="/var/lib/kubelet/pods/782e08fc-dc4b-4031-bf47-b6554db484f5/volumes" Nov 03 13:28:37.608485 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:37.608441 5024 patch_prober.go:29] interesting pod/e2e-derive-ef15d972-openshift-default-58bd498f85-wxmks container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 13:28:37.608956 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:37.608521 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-derive-ef15d972-openshift-default-58bd498f85-wxmks" podUID="779855bc-fa52-4f8a-8ed8-2692103f9e0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 13:28:38.579510 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:38.579457 5024 patch_prober.go:29] interesting pod/e2e-extmod-a-a62d22-openshift-default-d9c57bf65-pvtj7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 13:28:38.579864 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:38.579558 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-a62d22-openshift-default-d9c57bf65-pvtj7" podUID="78398b98-5396-4f98-bce4-682198b80f4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 13:28:39.510169 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:28:39.510120 5024 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d7mwz" containerName="extractor" Nov 03 13:28:40.866928 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:40.866895 5024 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6htht_cd91de08-084e-4667-b13f-2e07b3cff624/console-operator/1.log" Nov 03 13:28:40.867438 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:40.866895 5024 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6htht_cd91de08-084e-4667-b13f-2e07b3cff624/console-operator/1.log" Nov 03 13:28:41.275369 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:41.271948 5024 generic.go:356] "Generic (PLEG): container finished" podID="78398b98-5396-4f98-bce4-682198b80f4b" containerID="8be48a1471b5009f52dfa6f88215e1419deebb1457e0e7b59bf17f1cd67a6869" exitCode=0 Nov 03 13:28:41.285304 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:41.284918 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-a62d22-openshift-default-d9c57bf65-pvtj7" event={"ID":"78398b98-5396-4f98-bce4-682198b80f4b","Type":"ContainerDied","Data":"8be48a1471b5009f52dfa6f88215e1419deebb1457e0e7b59bf17f1cd67a6869"} Nov 03 13:28:41.336250 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:41.286862 5024 scope.go:122] "RemoveContainer" containerID="8be48a1471b5009f52dfa6f88215e1419deebb1457e0e7b59bf17f1cd67a6869" Nov 03 13:28:41.336250 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:41.314967 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:28:41.336250 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:41.315070 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:28:41.336250 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:41.315119 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:28:42.346178 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:42.346129 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-a62d22-openshift-default-d9c57bf65-pvtj7" event={"ID":"78398b98-5396-4f98-bce4-682198b80f4b","Type":"ContainerStarted","Data":"fd6a5b9b94d997fd6f479495e6afbe1a3fe56e9781b6deb4d2edb54647748d71"} Nov 03 13:28:42.353547 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:42.350956 5024 generic.go:356] "Generic (PLEG): container finished" podID="779855bc-fa52-4f8a-8ed8-2692103f9e0d" containerID="f223594702089d5e56df9b58ac17e5c2ee866020fc768e2ae0514c159e196200" exitCode=0 Nov 03 13:28:42.353547 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:42.351096 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-ef15d972-openshift-default-58bd498f85-wxmks" event={"ID":"779855bc-fa52-4f8a-8ed8-2692103f9e0d","Type":"ContainerDied","Data":"f223594702089d5e56df9b58ac17e5c2ee866020fc768e2ae0514c159e196200"} Nov 03 13:28:42.478758 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:42.478726 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-ef15d972-openshift-default-58bd498f85-wxmks" Nov 03 13:28:42.531511 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:42.531477 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/779855bc-fa52-4f8a-8ed8-2692103f9e0d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/779855bc-fa52-4f8a-8ed8-2692103f9e0d-workload-certs\") pod \"779855bc-fa52-4f8a-8ed8-2692103f9e0d\" (UID: \"779855bc-fa52-4f8a-8ed8-2692103f9e0d\") " Nov 03 13:28:42.531511 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:42.531516 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/779855bc-fa52-4f8a-8ed8-2692103f9e0d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/779855bc-fa52-4f8a-8ed8-2692103f9e0d-workload-socket\") pod \"779855bc-fa52-4f8a-8ed8-2692103f9e0d\" (UID: \"779855bc-fa52-4f8a-8ed8-2692103f9e0d\") " Nov 03 13:28:42.531813 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:42.531662 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/779855bc-fa52-4f8a-8ed8-2692103f9e0d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/779855bc-fa52-4f8a-8ed8-2692103f9e0d-istio-data\") pod \"779855bc-fa52-4f8a-8ed8-2692103f9e0d\" (UID: \"779855bc-fa52-4f8a-8ed8-2692103f9e0d\") " Nov 03 13:28:42.531813 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:42.531683 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/779855bc-fa52-4f8a-8ed8-2692103f9e0d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/779855bc-fa52-4f8a-8ed8-2692103f9e0d-istiod-ca-cert\") pod \"779855bc-fa52-4f8a-8ed8-2692103f9e0d\" (UID: \"779855bc-fa52-4f8a-8ed8-2692103f9e0d\") " Nov 03 13:28:42.531813 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:42.531766 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/779855bc-fa52-4f8a-8ed8-2692103f9e0d-workload-certs" pod "779855bc-fa52-4f8a-8ed8-2692103f9e0d" (UID: "779855bc-fa52-4f8a-8ed8-2692103f9e0d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:28:42.531813 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:42.531784 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/779855bc-fa52-4f8a-8ed8-2692103f9e0d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/779855bc-fa52-4f8a-8ed8-2692103f9e0d-istio-envoy\") pod \"779855bc-fa52-4f8a-8ed8-2692103f9e0d\" (UID: \"779855bc-fa52-4f8a-8ed8-2692103f9e0d\") " Nov 03 13:28:42.532036 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:42.531833 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/779855bc-fa52-4f8a-8ed8-2692103f9e0d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/779855bc-fa52-4f8a-8ed8-2692103f9e0d-istio-podinfo\") pod \"779855bc-fa52-4f8a-8ed8-2692103f9e0d\" (UID: \"779855bc-fa52-4f8a-8ed8-2692103f9e0d\") " Nov 03 13:28:42.532036 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:42.531862 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/779855bc-fa52-4f8a-8ed8-2692103f9e0d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/779855bc-fa52-4f8a-8ed8-2692103f9e0d-credential-socket\") pod \"779855bc-fa52-4f8a-8ed8-2692103f9e0d\" (UID: \"779855bc-fa52-4f8a-8ed8-2692103f9e0d\") " Nov 03 13:28:42.532036 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:42.531878 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/779855bc-fa52-4f8a-8ed8-2692103f9e0d-workload-socket" pod "779855bc-fa52-4f8a-8ed8-2692103f9e0d" (UID: "779855bc-fa52-4f8a-8ed8-2692103f9e0d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:28:42.532036 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:42.531889 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/779855bc-fa52-4f8a-8ed8-2692103f9e0d-kube-api-access-6ktjm\" (UniqueName: \"kubernetes.io/projected/779855bc-fa52-4f8a-8ed8-2692103f9e0d-kube-api-access-6ktjm\") pod \"779855bc-fa52-4f8a-8ed8-2692103f9e0d\" (UID: \"779855bc-fa52-4f8a-8ed8-2692103f9e0d\") " Nov 03 13:28:42.532036 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:42.531913 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/779855bc-fa52-4f8a-8ed8-2692103f9e0d-istio-data" pod "779855bc-fa52-4f8a-8ed8-2692103f9e0d" (UID: "779855bc-fa52-4f8a-8ed8-2692103f9e0d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:28:42.532036 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:42.531995 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/779855bc-fa52-4f8a-8ed8-2692103f9e0d-istio-token\" (UniqueName: \"kubernetes.io/projected/779855bc-fa52-4f8a-8ed8-2692103f9e0d-istio-token\") pod \"779855bc-fa52-4f8a-8ed8-2692103f9e0d\" (UID: \"779855bc-fa52-4f8a-8ed8-2692103f9e0d\") " Nov 03 13:28:42.532422 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:42.532132 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/779855bc-fa52-4f8a-8ed8-2692103f9e0d-istiod-ca-cert" pod "779855bc-fa52-4f8a-8ed8-2692103f9e0d" (UID: "779855bc-fa52-4f8a-8ed8-2692103f9e0d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:28:42.532482 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:42.532445 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/779855bc-fa52-4f8a-8ed8-2692103f9e0d-credential-socket" pod "779855bc-fa52-4f8a-8ed8-2692103f9e0d" (UID: "779855bc-fa52-4f8a-8ed8-2692103f9e0d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:28:42.532724 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:42.532704 5024 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/779855bc-fa52-4f8a-8ed8-2692103f9e0d-istio-data\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:28:42.532796 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:42.532729 5024 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/779855bc-fa52-4f8a-8ed8-2692103f9e0d-istiod-ca-cert\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:28:42.532796 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:42.532743 5024 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/779855bc-fa52-4f8a-8ed8-2692103f9e0d-credential-socket\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:28:42.532796 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:42.532759 5024 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/779855bc-fa52-4f8a-8ed8-2692103f9e0d-workload-certs\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:28:42.532796 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:42.532775 5024 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/779855bc-fa52-4f8a-8ed8-2692103f9e0d-workload-socket\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:28:42.534058 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:42.534029 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/779855bc-fa52-4f8a-8ed8-2692103f9e0d-istio-envoy" pod "779855bc-fa52-4f8a-8ed8-2692103f9e0d" (UID: "779855bc-fa52-4f8a-8ed8-2692103f9e0d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:28:42.534509 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:42.534485 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/779855bc-fa52-4f8a-8ed8-2692103f9e0d-kube-api-access-6ktjm" pod "779855bc-fa52-4f8a-8ed8-2692103f9e0d" (UID: "779855bc-fa52-4f8a-8ed8-2692103f9e0d"). InnerVolumeSpecName "kube-api-access-6ktjm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:28:42.534603 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:42.534504 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/779855bc-fa52-4f8a-8ed8-2692103f9e0d-istio-podinfo" pod "779855bc-fa52-4f8a-8ed8-2692103f9e0d" (UID: "779855bc-fa52-4f8a-8ed8-2692103f9e0d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 13:28:42.534603 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:42.534529 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/779855bc-fa52-4f8a-8ed8-2692103f9e0d-istio-token" pod "779855bc-fa52-4f8a-8ed8-2692103f9e0d" (UID: "779855bc-fa52-4f8a-8ed8-2692103f9e0d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:28:42.634720 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:42.634681 5024 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/779855bc-fa52-4f8a-8ed8-2692103f9e0d-istio-token\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:28:42.634720 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:42.634714 5024 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/779855bc-fa52-4f8a-8ed8-2692103f9e0d-istio-envoy\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:28:42.634720 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:42.634723 5024 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/779855bc-fa52-4f8a-8ed8-2692103f9e0d-istio-podinfo\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:28:42.634978 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:42.634733 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6ktjm\" (UniqueName: \"kubernetes.io/projected/779855bc-fa52-4f8a-8ed8-2692103f9e0d-kube-api-access-6ktjm\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:28:42.762368 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:42.754740 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-a62d22-openshift-default-d9c57bf65-pvtj7" Nov 03 13:28:42.762368 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:42.754884 5024 patch_prober.go:29] interesting pod/e2e-extmod-a-a62d22-openshift-default-d9c57bf65-pvtj7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Nov 03 13:28:42.762368 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:42.754930 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-a62d22-openshift-default-d9c57bf65-pvtj7" podUID="78398b98-5396-4f98-bce4-682198b80f4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Nov 03 13:28:43.360483 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:43.360450 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-ef15d972-openshift-default-58bd498f85-wxmks" Nov 03 13:28:43.360483 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:43.360470 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-ef15d972-openshift-default-58bd498f85-wxmks" event={"ID":"779855bc-fa52-4f8a-8ed8-2692103f9e0d","Type":"ContainerDied","Data":"8dcf10ba75075dfd553326626a09249351c007d1d4be9858a5db889c9d4f2d17"} Nov 03 13:28:43.361047 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:43.360536 5024 scope.go:122] "RemoveContainer" containerID="f223594702089d5e56df9b58ac17e5c2ee866020fc768e2ae0514c159e196200" Nov 03 13:28:43.449311 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:43.442662 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-ef15d972-openshift-default-58bd498f85-wxmks"] Nov 03 13:28:43.520306 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:43.502600 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-ef15d972-openshift-default-58bd498f85-wxmks"] Nov 03 13:28:43.751795 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:43.751763 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-cf4789xvd"] Nov 03 13:28:43.752741 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:43.752713 5024 cpu_manager.go:414] "containerMap: removing container" podUID="779855bc-fa52-4f8a-8ed8-2692103f9e0d" containerName="istio-proxy" Nov 03 13:28:43.752741 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:43.752738 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="779855bc-fa52-4f8a-8ed8-2692103f9e0d" containerName="istio-proxy" Nov 03 13:28:43.752941 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:43.752792 5024 cpu_manager.go:414] "containerMap: removing container" podUID="138f6437-e9a2-4270-8583-220e170165f4" containerName="payload-processing" Nov 03 13:28:43.752941 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:43.752798 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="138f6437-e9a2-4270-8583-220e170165f4" containerName="payload-processing" Nov 03 13:28:43.752941 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:43.752815 5024 cpu_manager.go:414] "containerMap: removing container" podUID="782e08fc-dc4b-4031-bf47-b6554db484f5" containerName="payload-pre-processing" Nov 03 13:28:43.752941 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:43.752843 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="782e08fc-dc4b-4031-bf47-b6554db484f5" containerName="payload-pre-processing" Nov 03 13:28:43.753163 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:43.753024 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="779855bc-fa52-4f8a-8ed8-2692103f9e0d" containerName="istio-proxy" Nov 03 13:28:43.753163 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:43.753048 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="138f6437-e9a2-4270-8583-220e170165f4" containerName="payload-processing" Nov 03 13:28:43.753163 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:43.753059 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="782e08fc-dc4b-4031-bf47-b6554db484f5" containerName="payload-pre-processing" Nov 03 13:28:43.753331 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:43.753206 5024 patch_prober.go:29] interesting pod/e2e-extmod-a-a62d22-openshift-default-d9c57bf65-pvtj7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Nov 03 13:28:43.753331 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:43.753261 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-a62d22-openshift-default-d9c57bf65-pvtj7" podUID="78398b98-5396-4f98-bce4-682198b80f4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Nov 03 13:28:43.789910 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:43.789868 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-cf4789xvd" Nov 03 13:28:43.824057 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:43.824029 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-a62d22\"/\"kube-root-ca.crt\"" Nov 03 13:28:43.826814 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:43.825426 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-a62d22\"/\"routing-overlay\"" Nov 03 13:28:43.826814 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:43.825463 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-a62d22\"/\"payload-processing-external-model-plugins-e2e-extmod-a-a62d22\"" Nov 03 13:28:43.826814 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:43.825537 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-a62d22\"/\"openshift-service-ca.crt\"" Nov 03 13:28:43.826814 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:43.825821 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-cf4789xvd"] Nov 03 13:28:43.927374 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:43.923428 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-a62d22\"/\"payload-processing-external-model-e2e--f52a173c-dockercfg-d5wkp\"" Nov 03 13:28:43.927374 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:43.923721 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-a62d22\"/\"agc-cred-e942f4e7c82f\"" Nov 03 13:28:44.398353 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:44.368128 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b739270-1c05-44b6-bed6-de7eaadb689f-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a62d22-cf4789xvd\" (UID: \"9b739270-1c05-44b6-bed6-de7eaadb689f\") " pod="ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-cf4789xvd" Nov 03 13:28:44.398353 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:44.368187 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9b739270-1c05-44b6-bed6-de7eaadb689f-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-a62d22-cf4789xvd\" (UID: \"9b739270-1c05-44b6-bed6-de7eaadb689f\") " pod="ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-cf4789xvd" Nov 03 13:28:44.398353 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:44.368277 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9b739270-1c05-44b6-bed6-de7eaadb689f-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a62d22-cf4789xvd\" (UID: \"9b739270-1c05-44b6-bed6-de7eaadb689f\") " pod="ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-cf4789xvd" Nov 03 13:28:44.398353 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:44.368325 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wktb5\" (UniqueName: \"kubernetes.io/projected/9b739270-1c05-44b6-bed6-de7eaadb689f-kube-api-access-wktb5\") pod \"payload-processing-external-model-e2e-extmod-a-a62d22-cf4789xvd\" (UID: \"9b739270-1c05-44b6-bed6-de7eaadb689f\") " pod="ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-cf4789xvd" Nov 03 13:28:44.480334 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:44.475319 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b739270-1c05-44b6-bed6-de7eaadb689f-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a62d22-cf4789xvd\" (UID: \"9b739270-1c05-44b6-bed6-de7eaadb689f\") " pod="ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-cf4789xvd" Nov 03 13:28:44.480334 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:44.475381 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9b739270-1c05-44b6-bed6-de7eaadb689f-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-a62d22-cf4789xvd\" (UID: \"9b739270-1c05-44b6-bed6-de7eaadb689f\") " pod="ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-cf4789xvd" Nov 03 13:28:44.480334 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:44.475453 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9b739270-1c05-44b6-bed6-de7eaadb689f-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a62d22-cf4789xvd\" (UID: \"9b739270-1c05-44b6-bed6-de7eaadb689f\") " pod="ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-cf4789xvd" Nov 03 13:28:44.480334 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:44.475479 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wktb5\" (UniqueName: \"kubernetes.io/projected/9b739270-1c05-44b6-bed6-de7eaadb689f-kube-api-access-wktb5\") pod \"payload-processing-external-model-e2e-extmod-a-a62d22-cf4789xvd\" (UID: \"9b739270-1c05-44b6-bed6-de7eaadb689f\") " pod="ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-cf4789xvd" Nov 03 13:28:44.541076 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:44.541037 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="779855bc-fa52-4f8a-8ed8-2692103f9e0d" path="/var/lib/kubelet/pods/779855bc-fa52-4f8a-8ed8-2692103f9e0d/volumes" Nov 03 13:28:44.542563 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:44.542521 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9b739270-1c05-44b6-bed6-de7eaadb689f-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a62d22-cf4789xvd\" (UID: \"9b739270-1c05-44b6-bed6-de7eaadb689f\") " pod="ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-cf4789xvd" Nov 03 13:28:44.554341 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:44.544236 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wktb5\" (UniqueName: \"kubernetes.io/projected/9b739270-1c05-44b6-bed6-de7eaadb689f-kube-api-access-wktb5\") pod \"payload-processing-external-model-e2e-extmod-a-a62d22-cf4789xvd\" (UID: \"9b739270-1c05-44b6-bed6-de7eaadb689f\") " pod="ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-cf4789xvd" Nov 03 13:28:44.569008 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:44.568968 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b739270-1c05-44b6-bed6-de7eaadb689f-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a62d22-cf4789xvd\" (UID: \"9b739270-1c05-44b6-bed6-de7eaadb689f\") " pod="ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-cf4789xvd" Nov 03 13:28:44.578332 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:44.569936 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9b739270-1c05-44b6-bed6-de7eaadb689f-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-a62d22-cf4789xvd\" (UID: \"9b739270-1c05-44b6-bed6-de7eaadb689f\") " pod="ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-cf4789xvd" Nov 03 13:28:44.722251 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:44.722155 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-cf4789xvd" Nov 03 13:28:44.753992 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:44.753937 5024 patch_prober.go:29] interesting pod/e2e-extmod-a-a62d22-openshift-default-d9c57bf65-pvtj7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Nov 03 13:28:44.754195 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:44.754002 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-a62d22-openshift-default-d9c57bf65-pvtj7" podUID="78398b98-5396-4f98-bce4-682198b80f4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Nov 03 13:28:45.304829 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:45.275358 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-cf4789xvd"] Nov 03 13:28:45.666562 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:45.661445 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-cf4789xvd" event={"ID":"9b739270-1c05-44b6-bed6-de7eaadb689f","Type":"ContainerStarted","Data":"d33922f116839a0da1281c0ee59114b73542a6e08889467ad5192e20d1ae63be"} Nov 03 13:28:45.753308 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:45.752678 5024 patch_prober.go:29] interesting pod/e2e-extmod-a-a62d22-openshift-default-d9c57bf65-pvtj7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Nov 03 13:28:45.753308 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:45.752750 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-a62d22-openshift-default-d9c57bf65-pvtj7" podUID="78398b98-5396-4f98-bce4-682198b80f4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Nov 03 13:28:45.916558 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:45.916522 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-5bffndnqx"] Nov 03 13:28:45.942605 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:45.942568 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-5bffndnqx"] Nov 03 13:28:45.942950 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:45.942930 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-5bffndnqx" Nov 03 13:28:45.955405 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:45.952437 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-a62d22\"/\"e2e-cfl-e87c5e-provider-cred\"" Nov 03 13:28:46.004425 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:46.004376 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6908d762-9f04-4823-a7f2-00a4e703b601-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a62d22-5bffndnqx\" (UID: \"6908d762-9f04-4823-a7f2-00a4e703b601\") " pod="ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-5bffndnqx" Nov 03 13:28:46.004785 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:46.004761 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6908d762-9f04-4823-a7f2-00a4e703b601-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-a62d22-5bffndnqx\" (UID: \"6908d762-9f04-4823-a7f2-00a4e703b601\") " pod="ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-5bffndnqx" Nov 03 13:28:46.004970 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:46.004955 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7zw9c\" (UniqueName: \"kubernetes.io/projected/6908d762-9f04-4823-a7f2-00a4e703b601-kube-api-access-7zw9c\") pod \"payload-processing-external-model-e2e-extmod-a-a62d22-5bffndnqx\" (UID: \"6908d762-9f04-4823-a7f2-00a4e703b601\") " pod="ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-5bffndnqx" Nov 03 13:28:46.005219 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:46.005199 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6908d762-9f04-4823-a7f2-00a4e703b601-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a62d22-5bffndnqx\" (UID: \"6908d762-9f04-4823-a7f2-00a4e703b601\") " pod="ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-5bffndnqx" Nov 03 13:28:46.200403 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:46.193091 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6908d762-9f04-4823-a7f2-00a4e703b601-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a62d22-5bffndnqx\" (UID: \"6908d762-9f04-4823-a7f2-00a4e703b601\") " pod="ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-5bffndnqx" Nov 03 13:28:46.323783 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:46.323735 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6908d762-9f04-4823-a7f2-00a4e703b601-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a62d22-5bffndnqx\" (UID: \"6908d762-9f04-4823-a7f2-00a4e703b601\") " pod="ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-5bffndnqx" Nov 03 13:28:46.339788 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:46.339747 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6908d762-9f04-4823-a7f2-00a4e703b601-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-a62d22-5bffndnqx\" (UID: \"6908d762-9f04-4823-a7f2-00a4e703b601\") " pod="ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-5bffndnqx" Nov 03 13:28:46.340650 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:46.340614 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zw9c\" (UniqueName: \"kubernetes.io/projected/6908d762-9f04-4823-a7f2-00a4e703b601-kube-api-access-7zw9c\") pod \"payload-processing-external-model-e2e-extmod-a-a62d22-5bffndnqx\" (UID: \"6908d762-9f04-4823-a7f2-00a4e703b601\") " pod="ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-5bffndnqx" Nov 03 13:28:46.440346 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:46.203478 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6908d762-9f04-4823-a7f2-00a4e703b601-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a62d22-5bffndnqx\" (UID: \"6908d762-9f04-4823-a7f2-00a4e703b601\") " pod="ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-5bffndnqx" Nov 03 13:28:46.440346 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:46.325062 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6908d762-9f04-4823-a7f2-00a4e703b601-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a62d22-5bffndnqx\" (UID: \"6908d762-9f04-4823-a7f2-00a4e703b601\") " pod="ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-5bffndnqx" Nov 03 13:28:46.512354 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:46.511797 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6908d762-9f04-4823-a7f2-00a4e703b601-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-a62d22-5bffndnqx\" (UID: \"6908d762-9f04-4823-a7f2-00a4e703b601\") " pod="ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-5bffndnqx" Nov 03 13:28:46.547073 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:46.547035 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zw9c\" (UniqueName: \"kubernetes.io/projected/6908d762-9f04-4823-a7f2-00a4e703b601-kube-api-access-7zw9c\") pod \"payload-processing-external-model-e2e-extmod-a-a62d22-5bffndnqx\" (UID: \"6908d762-9f04-4823-a7f2-00a4e703b601\") " pod="ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-5bffndnqx" Nov 03 13:28:46.630775 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:46.630719 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-5bffndnqx" Nov 03 13:28:46.785569 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:46.785538 5024 patch_prober.go:29] interesting pod/e2e-extmod-a-a62d22-openshift-default-d9c57bf65-pvtj7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Nov 03 13:28:46.786069 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:46.786036 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-a62d22-openshift-default-d9c57bf65-pvtj7" podUID="78398b98-5396-4f98-bce4-682198b80f4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Nov 03 13:28:46.843324 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:46.842397 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-cf4789xvd" event={"ID":"9b739270-1c05-44b6-bed6-de7eaadb689f","Type":"ContainerStarted","Data":"d0901a47ddce81ec654fecc910a2c87fe5c16dd9d7b1d0f27056796f7e761cc7"} Nov 03 13:28:46.843324 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:46.842500 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-cf4789xvd" Nov 03 13:28:46.954314 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:46.943782 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-cf4789xvd" podStartSLOduration=3.943757433 podStartE2EDuration="3.943757433s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:28:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:28:46.903599402 +0000 UTC m=+1206.911790679" watchObservedRunningTime="2026-11-03 13:28:46.943757433 +0000 UTC m=+1206.951948739" Nov 03 13:28:47.000548 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:28:47.000503 5024 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-a62d22/agc-cred-e942f4e7c82f: secret "agc-cred-e942f4e7c82f" not found Nov 03 13:28:47.000769 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:28:47.000752 5024 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-cf4789xvd: secret "agc-cred-e942f4e7c82f" not found Nov 03 13:28:47.001304 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:28:47.000977 5024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9b739270-1c05-44b6-bed6-de7eaadb689f-provider-credentials podName:9b739270-1c05-44b6-bed6-de7eaadb689f nodeName:}" failed. No retries permitted until 2026-11-03 13:28:47.500947325 +0000 UTC m=+1207.509138599 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/9b739270-1c05-44b6-bed6-de7eaadb689f-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-a62d22-cf4789xvd" (UID: "9b739270-1c05-44b6-bed6-de7eaadb689f") : secret "agc-cred-e942f4e7c82f" not found Nov 03 13:28:47.523612 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:28:47.523529 5024 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-a62d22/agc-cred-e942f4e7c82f: secret "agc-cred-e942f4e7c82f" not found Nov 03 13:28:47.523612 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:28:47.523557 5024 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-cf4789xvd: secret "agc-cred-e942f4e7c82f" not found Nov 03 13:28:47.523612 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:28:47.523612 5024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9b739270-1c05-44b6-bed6-de7eaadb689f-provider-credentials podName:9b739270-1c05-44b6-bed6-de7eaadb689f nodeName:}" failed. No retries permitted until 2026-11-03 13:28:48.523597857 +0000 UTC m=+1208.531789112 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/9b739270-1c05-44b6-bed6-de7eaadb689f-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-a62d22-cf4789xvd" (UID: "9b739270-1c05-44b6-bed6-de7eaadb689f") : secret "agc-cred-e942f4e7c82f" not found Nov 03 13:28:47.531178 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:47.531143 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-5bffndnqx"] Nov 03 13:28:47.531597 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:28:47.531575 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6908d762_9f04_4823_a7f2_00a4e703b601.slice/crio-034c244e6819a2710b027639a0e42e6254be3a34e995d3d416e47181f94bfcc1 WatchSource:0}: Error finding container 034c244e6819a2710b027639a0e42e6254be3a34e995d3d416e47181f94bfcc1: Status 404 returned error can't find the container with id 034c244e6819a2710b027639a0e42e6254be3a34e995d3d416e47181f94bfcc1 Nov 03 13:28:47.752506 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:47.752332 5024 patch_prober.go:29] interesting pod/e2e-extmod-a-a62d22-openshift-default-d9c57bf65-pvtj7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Nov 03 13:28:47.752506 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:47.752416 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-a62d22-openshift-default-d9c57bf65-pvtj7" podUID="78398b98-5396-4f98-bce4-682198b80f4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Nov 03 13:28:47.856423 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:47.856385 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-5bffndnqx" event={"ID":"6908d762-9f04-4823-a7f2-00a4e703b601","Type":"ContainerStarted","Data":"7a22fa57cc4e724021af1499f19bda4391dbf49842f8946a064c0dee05b5ab55"} Nov 03 13:28:47.856908 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:47.856888 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-5bffndnqx" event={"ID":"6908d762-9f04-4823-a7f2-00a4e703b601","Type":"ContainerStarted","Data":"034c244e6819a2710b027639a0e42e6254be3a34e995d3d416e47181f94bfcc1"} Nov 03 13:28:47.857000 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:47.856990 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-5bffndnqx" Nov 03 13:28:48.596311 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:28:48.593908 5024 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-a62d22/agc-cred-e942f4e7c82f: secret "agc-cred-e942f4e7c82f" not found Nov 03 13:28:48.596311 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:28:48.593945 5024 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-cf4789xvd: secret "agc-cred-e942f4e7c82f" not found Nov 03 13:28:48.596311 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:28:48.594013 5024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9b739270-1c05-44b6-bed6-de7eaadb689f-provider-credentials podName:9b739270-1c05-44b6-bed6-de7eaadb689f nodeName:}" failed. No retries permitted until 2026-11-03 13:28:50.593993314 +0000 UTC m=+1210.602184589 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/9b739270-1c05-44b6-bed6-de7eaadb689f-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-a62d22-cf4789xvd" (UID: "9b739270-1c05-44b6-bed6-de7eaadb689f") : secret "agc-cred-e942f4e7c82f" not found Nov 03 13:28:48.754803 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:48.754761 5024 patch_prober.go:29] interesting pod/e2e-extmod-a-a62d22-openshift-default-d9c57bf65-pvtj7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Nov 03 13:28:48.755098 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:48.755063 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-a62d22-openshift-default-d9c57bf65-pvtj7" podUID="78398b98-5396-4f98-bce4-682198b80f4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Nov 03 13:28:49.753279 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:49.753239 5024 patch_prober.go:29] interesting pod/e2e-extmod-a-a62d22-openshift-default-d9c57bf65-pvtj7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Nov 03 13:28:49.754022 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:49.753989 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-a62d22-openshift-default-d9c57bf65-pvtj7" podUID="78398b98-5396-4f98-bce4-682198b80f4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Nov 03 13:28:50.641416 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:28:50.641361 5024 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-a62d22/agc-cred-e942f4e7c82f: secret "agc-cred-e942f4e7c82f" not found Nov 03 13:28:50.641686 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:28:50.641665 5024 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-cf4789xvd: secret "agc-cred-e942f4e7c82f" not found Nov 03 13:28:50.641889 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:28:50.641876 5024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9b739270-1c05-44b6-bed6-de7eaadb689f-provider-credentials podName:9b739270-1c05-44b6-bed6-de7eaadb689f nodeName:}" failed. No retries permitted until 2026-11-03 13:28:54.641841978 +0000 UTC m=+1214.650033247 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/9b739270-1c05-44b6-bed6-de7eaadb689f-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-a62d22-cf4789xvd" (UID: "9b739270-1c05-44b6-bed6-de7eaadb689f") : secret "agc-cred-e942f4e7c82f" not found Nov 03 13:28:50.759020 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:50.758974 5024 patch_prober.go:29] interesting pod/e2e-extmod-a-a62d22-openshift-default-d9c57bf65-pvtj7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Nov 03 13:28:50.759660 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:50.759609 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-a62d22-openshift-default-d9c57bf65-pvtj7" podUID="78398b98-5396-4f98-bce4-682198b80f4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Nov 03 13:28:51.935745 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:51.934870 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-a62d22-openshift-default-d9c57bf65-pvtj7" Nov 03 13:28:51.936245 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:51.935972 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-a62d22-openshift-default-d9c57bf65-pvtj7" Nov 03 13:28:51.936975 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:51.936944 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-a62d22-openshift-default-d9c57bf65-pvtj7" Nov 03 13:28:51.961708 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:51.961642 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-5bffndnqx" podStartSLOduration=6.961620568 podStartE2EDuration="6.961620568s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:28:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:28:47.896573365 +0000 UTC m=+1207.904764644" watchObservedRunningTime="2026-11-03 13:28:51.961620568 +0000 UTC m=+1211.969811873" Nov 03 13:28:54.688270 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:28:54.688228 5024 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-a62d22/agc-cred-e942f4e7c82f: secret "agc-cred-e942f4e7c82f" not found Nov 03 13:28:54.688824 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:28:54.688800 5024 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-cf4789xvd: secret "agc-cred-e942f4e7c82f" not found Nov 03 13:28:54.689568 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:28:54.689542 5024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9b739270-1c05-44b6-bed6-de7eaadb689f-provider-credentials podName:9b739270-1c05-44b6-bed6-de7eaadb689f nodeName:}" failed. No retries permitted until 2026-11-03 13:29:02.689022911 +0000 UTC m=+1222.697214189 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/9b739270-1c05-44b6-bed6-de7eaadb689f-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-a62d22-cf4789xvd" (UID: "9b739270-1c05-44b6-bed6-de7eaadb689f") : secret "agc-cred-e942f4e7c82f" not found Nov 03 13:28:55.868321 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:55.860339 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-bf6dd6dcc-gwprz"] Nov 03 13:28:55.889311 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:55.886616 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-bf6dd6dcc-gwprz" Nov 03 13:28:55.936306 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:55.920828 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-498b9d9ffa2b\"" Nov 03 13:28:55.936306 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:55.925613 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\"" Nov 03 13:28:55.936306 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:55.925706 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-vx42v\"" Nov 03 13:28:55.936306 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:55.925789 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\"" Nov 03 13:28:55.936306 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:55.925841 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\"" Nov 03 13:28:55.936306 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:55.925890 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\"" Nov 03 13:28:56.189212 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:56.182793 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2mrm9\" (UniqueName: \"kubernetes.io/projected/194ba584-d286-4853-bc37-d82ee939ec50-kube-api-access-2mrm9\") pod \"payload-processing-external-model-bf6dd6dcc-gwprz\" (UID: \"194ba584-d286-4853-bc37-d82ee939ec50\") " pod="models-as-a-service/payload-processing-external-model-bf6dd6dcc-gwprz" Nov 03 13:28:56.190073 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:56.190043 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/194ba584-d286-4853-bc37-d82ee939ec50-provider-credentials\") pod \"payload-processing-external-model-bf6dd6dcc-gwprz\" (UID: \"194ba584-d286-4853-bc37-d82ee939ec50\") " pod="models-as-a-service/payload-processing-external-model-bf6dd6dcc-gwprz" Nov 03 13:28:56.207308 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:56.197554 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/194ba584-d286-4853-bc37-d82ee939ec50-routing-overlay-volume\") pod \"payload-processing-external-model-bf6dd6dcc-gwprz\" (UID: \"194ba584-d286-4853-bc37-d82ee939ec50\") " pod="models-as-a-service/payload-processing-external-model-bf6dd6dcc-gwprz" Nov 03 13:28:56.334442 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:56.244135 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/194ba584-d286-4853-bc37-d82ee939ec50-plugins-config-volume\") pod \"payload-processing-external-model-bf6dd6dcc-gwprz\" (UID: \"194ba584-d286-4853-bc37-d82ee939ec50\") " pod="models-as-a-service/payload-processing-external-model-bf6dd6dcc-gwprz" Nov 03 13:28:56.390356 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:56.389866 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2mrm9\" (UniqueName: \"kubernetes.io/projected/194ba584-d286-4853-bc37-d82ee939ec50-kube-api-access-2mrm9\") pod \"payload-processing-external-model-bf6dd6dcc-gwprz\" (UID: \"194ba584-d286-4853-bc37-d82ee939ec50\") " pod="models-as-a-service/payload-processing-external-model-bf6dd6dcc-gwprz" Nov 03 13:28:56.390356 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:56.389948 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/194ba584-d286-4853-bc37-d82ee939ec50-provider-credentials\") pod \"payload-processing-external-model-bf6dd6dcc-gwprz\" (UID: \"194ba584-d286-4853-bc37-d82ee939ec50\") " pod="models-as-a-service/payload-processing-external-model-bf6dd6dcc-gwprz" Nov 03 13:28:56.390356 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:56.390156 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/194ba584-d286-4853-bc37-d82ee939ec50-routing-overlay-volume\") pod \"payload-processing-external-model-bf6dd6dcc-gwprz\" (UID: \"194ba584-d286-4853-bc37-d82ee939ec50\") " pod="models-as-a-service/payload-processing-external-model-bf6dd6dcc-gwprz" Nov 03 13:28:56.390356 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:56.390340 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/194ba584-d286-4853-bc37-d82ee939ec50-plugins-config-volume\") pod \"payload-processing-external-model-bf6dd6dcc-gwprz\" (UID: \"194ba584-d286-4853-bc37-d82ee939ec50\") " pod="models-as-a-service/payload-processing-external-model-bf6dd6dcc-gwprz" Nov 03 13:28:56.451460 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:56.451376 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/194ba584-d286-4853-bc37-d82ee939ec50-routing-overlay-volume\") pod \"payload-processing-external-model-bf6dd6dcc-gwprz\" (UID: \"194ba584-d286-4853-bc37-d82ee939ec50\") " pod="models-as-a-service/payload-processing-external-model-bf6dd6dcc-gwprz" Nov 03 13:28:56.454268 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:56.454233 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2mrm9\" (UniqueName: \"kubernetes.io/projected/194ba584-d286-4853-bc37-d82ee939ec50-kube-api-access-2mrm9\") pod \"payload-processing-external-model-bf6dd6dcc-gwprz\" (UID: \"194ba584-d286-4853-bc37-d82ee939ec50\") " pod="models-as-a-service/payload-processing-external-model-bf6dd6dcc-gwprz" Nov 03 13:28:56.455136 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:56.455102 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/194ba584-d286-4853-bc37-d82ee939ec50-provider-credentials\") pod \"payload-processing-external-model-bf6dd6dcc-gwprz\" (UID: \"194ba584-d286-4853-bc37-d82ee939ec50\") " pod="models-as-a-service/payload-processing-external-model-bf6dd6dcc-gwprz" Nov 03 13:28:56.490328 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:56.487997 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/194ba584-d286-4853-bc37-d82ee939ec50-plugins-config-volume\") pod \"payload-processing-external-model-bf6dd6dcc-gwprz\" (UID: \"194ba584-d286-4853-bc37-d82ee939ec50\") " pod="models-as-a-service/payload-processing-external-model-bf6dd6dcc-gwprz" Nov 03 13:28:56.510406 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:56.509999 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-bf6dd6dcc-gwprz" Nov 03 13:28:56.530356 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:56.517456 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-bf6dd6dcc-gwprz"] Nov 03 13:28:57.010410 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:28:57.010375 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod194ba584_d286_4853_bc37_d82ee939ec50.slice/crio-627de5b3ff068e059b2a3db9e5d63c1b51d5cc7a0f7253a8fa5a47630cafd276 WatchSource:0}: Error finding container 627de5b3ff068e059b2a3db9e5d63c1b51d5cc7a0f7253a8fa5a47630cafd276: Status 404 returned error can't find the container with id 627de5b3ff068e059b2a3db9e5d63c1b51d5cc7a0f7253a8fa5a47630cafd276 Nov 03 13:28:57.012433 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:57.012399 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-bf6dd6dcc-gwprz"] Nov 03 13:28:57.318491 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:57.316921 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-bf6dd6dcc-gwprz" event={"ID":"194ba584-d286-4853-bc37-d82ee939ec50","Type":"ContainerStarted","Data":"a248beb231ddfe0b3cc9b63f778c94ef4173c23c039f63539ba1618390d23673"} Nov 03 13:28:57.318491 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:57.316998 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-bf6dd6dcc-gwprz" event={"ID":"194ba584-d286-4853-bc37-d82ee939ec50","Type":"ContainerStarted","Data":"627de5b3ff068e059b2a3db9e5d63c1b51d5cc7a0f7253a8fa5a47630cafd276"} Nov 03 13:28:57.318491 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:57.318354 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-bf6dd6dcc-gwprz" Nov 03 13:28:57.344653 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:57.344584 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-bf6dd6dcc-gwprz" podStartSLOduration=2.344564083 podStartE2EDuration="2.344564083s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:28:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:28:57.342650537 +0000 UTC m=+1217.350841815" watchObservedRunningTime="2026-11-03 13:28:57.344564083 +0000 UTC m=+1217.352755360" Nov 03 13:28:57.863248 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:57.863219 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-cf4789xvd" Nov 03 13:28:58.881732 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:28:58.881692 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-5bffndnqx" podUID="6908d762-9f04-4823-a7f2-00a4e703b601" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 13:29:02.716851 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:29:02.716814 5024 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-a62d22/agc-cred-e942f4e7c82f: secret "agc-cred-e942f4e7c82f" not found Nov 03 13:29:02.716851 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:29:02.716847 5024 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-cf4789xvd: secret "agc-cred-e942f4e7c82f" not found Nov 03 13:29:02.717296 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:29:02.716915 5024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9b739270-1c05-44b6-bed6-de7eaadb689f-provider-credentials podName:9b739270-1c05-44b6-bed6-de7eaadb689f nodeName:}" failed. No retries permitted until 2026-11-03 13:29:18.716895625 +0000 UTC m=+1238.725086894 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/9b739270-1c05-44b6-bed6-de7eaadb689f-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-a62d22-cf4789xvd" (UID: "9b739270-1c05-44b6-bed6-de7eaadb689f") : secret "agc-cred-e942f4e7c82f" not found Nov 03 13:29:08.882308 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:29:08.882048 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-5bffndnqx" podUID="6908d762-9f04-4823-a7f2-00a4e703b601" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 13:29:09.361017 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:29:09.360977 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-bf6dd6dcc-gwprz" Nov 03 13:29:09.617327 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:29:09.616507 5024 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d7mwz" containerName="extractor" Nov 03 13:29:15.537273 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:29:15.537229 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-ef1693/payload-processing-external-model-e2e-extmod-a-ef1693-7d66w9gk2"] Nov 03 13:29:15.542542 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:29:15.542509 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-ef1693/payload-processing-external-model-e2e-extmod-a-ef1693-7d66w9gk2" Nov 03 13:29:15.549493 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:29:15.549148 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-ef1693\"/\"agc-cred-1c2ab57ceeb2\"" Nov 03 13:29:15.552958 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:29:15.552848 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-ef1693/payload-processing-external-model-e2e-extmod-a-ef1693-7d66w9gk2"] Nov 03 13:29:15.611945 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:29:15.611901 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q7fjv\" (UniqueName: \"kubernetes.io/projected/dd149ad0-f82e-4c7a-a439-ff9a7ff9a6ad-kube-api-access-q7fjv\") pod \"payload-processing-external-model-e2e-extmod-a-ef1693-7d66w9gk2\" (UID: \"dd149ad0-f82e-4c7a-a439-ff9a7ff9a6ad\") " pod="ai-tenant-e2e-extmod-a-ef1693/payload-processing-external-model-e2e-extmod-a-ef1693-7d66w9gk2" Nov 03 13:29:15.612153 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:29:15.612055 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/dd149ad0-f82e-4c7a-a439-ff9a7ff9a6ad-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-ef1693-7d66w9gk2\" (UID: \"dd149ad0-f82e-4c7a-a439-ff9a7ff9a6ad\") " pod="ai-tenant-e2e-extmod-a-ef1693/payload-processing-external-model-e2e-extmod-a-ef1693-7d66w9gk2" Nov 03 13:29:15.612153 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:29:15.612116 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd149ad0-f82e-4c7a-a439-ff9a7ff9a6ad-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-ef1693-7d66w9gk2\" (UID: \"dd149ad0-f82e-4c7a-a439-ff9a7ff9a6ad\") " pod="ai-tenant-e2e-extmod-a-ef1693/payload-processing-external-model-e2e-extmod-a-ef1693-7d66w9gk2" Nov 03 13:29:15.612333 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:29:15.612313 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/dd149ad0-f82e-4c7a-a439-ff9a7ff9a6ad-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-ef1693-7d66w9gk2\" (UID: \"dd149ad0-f82e-4c7a-a439-ff9a7ff9a6ad\") " pod="ai-tenant-e2e-extmod-a-ef1693/payload-processing-external-model-e2e-extmod-a-ef1693-7d66w9gk2" Nov 03 13:29:15.713736 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:29:15.713696 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/dd149ad0-f82e-4c7a-a439-ff9a7ff9a6ad-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-ef1693-7d66w9gk2\" (UID: \"dd149ad0-f82e-4c7a-a439-ff9a7ff9a6ad\") " pod="ai-tenant-e2e-extmod-a-ef1693/payload-processing-external-model-e2e-extmod-a-ef1693-7d66w9gk2" Nov 03 13:29:15.713736 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:29:15.713744 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd149ad0-f82e-4c7a-a439-ff9a7ff9a6ad-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-ef1693-7d66w9gk2\" (UID: \"dd149ad0-f82e-4c7a-a439-ff9a7ff9a6ad\") " pod="ai-tenant-e2e-extmod-a-ef1693/payload-processing-external-model-e2e-extmod-a-ef1693-7d66w9gk2" Nov 03 13:29:15.714018 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:29:15.713920 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/dd149ad0-f82e-4c7a-a439-ff9a7ff9a6ad-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-ef1693-7d66w9gk2\" (UID: \"dd149ad0-f82e-4c7a-a439-ff9a7ff9a6ad\") " pod="ai-tenant-e2e-extmod-a-ef1693/payload-processing-external-model-e2e-extmod-a-ef1693-7d66w9gk2" Nov 03 13:29:15.714101 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:29:15.714075 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7fjv\" (UniqueName: \"kubernetes.io/projected/dd149ad0-f82e-4c7a-a439-ff9a7ff9a6ad-kube-api-access-q7fjv\") pod \"payload-processing-external-model-e2e-extmod-a-ef1693-7d66w9gk2\" (UID: \"dd149ad0-f82e-4c7a-a439-ff9a7ff9a6ad\") " pod="ai-tenant-e2e-extmod-a-ef1693/payload-processing-external-model-e2e-extmod-a-ef1693-7d66w9gk2" Nov 03 13:29:15.715024 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:29:15.714984 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/dd149ad0-f82e-4c7a-a439-ff9a7ff9a6ad-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-ef1693-7d66w9gk2\" (UID: \"dd149ad0-f82e-4c7a-a439-ff9a7ff9a6ad\") " pod="ai-tenant-e2e-extmod-a-ef1693/payload-processing-external-model-e2e-extmod-a-ef1693-7d66w9gk2" Nov 03 13:29:15.715147 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:29:15.714999 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd149ad0-f82e-4c7a-a439-ff9a7ff9a6ad-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-ef1693-7d66w9gk2\" (UID: \"dd149ad0-f82e-4c7a-a439-ff9a7ff9a6ad\") " pod="ai-tenant-e2e-extmod-a-ef1693/payload-processing-external-model-e2e-extmod-a-ef1693-7d66w9gk2" Nov 03 13:29:15.717085 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:29:15.717059 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/dd149ad0-f82e-4c7a-a439-ff9a7ff9a6ad-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-ef1693-7d66w9gk2\" (UID: \"dd149ad0-f82e-4c7a-a439-ff9a7ff9a6ad\") " pod="ai-tenant-e2e-extmod-a-ef1693/payload-processing-external-model-e2e-extmod-a-ef1693-7d66w9gk2" Nov 03 13:29:15.724095 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:29:15.724055 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7fjv\" (UniqueName: \"kubernetes.io/projected/dd149ad0-f82e-4c7a-a439-ff9a7ff9a6ad-kube-api-access-q7fjv\") pod \"payload-processing-external-model-e2e-extmod-a-ef1693-7d66w9gk2\" (UID: \"dd149ad0-f82e-4c7a-a439-ff9a7ff9a6ad\") " pod="ai-tenant-e2e-extmod-a-ef1693/payload-processing-external-model-e2e-extmod-a-ef1693-7d66w9gk2" Nov 03 13:29:15.861189 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:29:15.861152 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-ef1693/payload-processing-external-model-e2e-extmod-a-ef1693-7d66w9gk2" Nov 03 13:29:16.506843 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:29:16.506811 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-ef1693/payload-processing-external-model-e2e-extmod-a-ef1693-7d66w9gk2"] Nov 03 13:29:16.509370 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:29:16.509337 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd149ad0_f82e_4c7a_a439_ff9a7ff9a6ad.slice/crio-e227810dde1dc3646f163ab30bfc403c8632d21b04da29957458b921fd2a6330 WatchSource:0}: Error finding container e227810dde1dc3646f163ab30bfc403c8632d21b04da29957458b921fd2a6330: Status 404 returned error can't find the container with id e227810dde1dc3646f163ab30bfc403c8632d21b04da29957458b921fd2a6330 Nov 03 13:29:17.013747 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:29:17.013708 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-ef1693/payload-processing-external-model-e2e-extmod-a-ef1693-7d66w9gk2" event={"ID":"dd149ad0-f82e-4c7a-a439-ff9a7ff9a6ad","Type":"ContainerStarted","Data":"029683318cfff514471a48d5da49f21bf5c261c1461acc9fa9778d7e1d69289b"} Nov 03 13:29:17.014244 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:29:17.013770 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-ef1693/payload-processing-external-model-e2e-extmod-a-ef1693-7d66w9gk2" event={"ID":"dd149ad0-f82e-4c7a-a439-ff9a7ff9a6ad","Type":"ContainerStarted","Data":"e227810dde1dc3646f163ab30bfc403c8632d21b04da29957458b921fd2a6330"} Nov 03 13:29:17.014244 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:29:17.013799 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-ef1693/payload-processing-external-model-e2e-extmod-a-ef1693-7d66w9gk2" Nov 03 13:29:17.049165 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:29:17.049104 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-ef1693/payload-processing-external-model-e2e-extmod-a-ef1693-7d66w9gk2" podStartSLOduration=2.049088399 podStartE2EDuration="2.049088399s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:29:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:29:17.046572854 +0000 UTC m=+1237.054764131" watchObservedRunningTime="2026-11-03 13:29:17.049088399 +0000 UTC m=+1237.057279676" Nov 03 13:29:18.723662 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:29:18.723625 5024 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-a62d22/agc-cred-e942f4e7c82f: secret "agc-cred-e942f4e7c82f" not found Nov 03 13:29:18.723662 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:29:18.723657 5024 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-cf4789xvd: secret "agc-cred-e942f4e7c82f" not found Nov 03 13:29:18.724135 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:29:18.723717 5024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9b739270-1c05-44b6-bed6-de7eaadb689f-provider-credentials podName:9b739270-1c05-44b6-bed6-de7eaadb689f nodeName:}" failed. No retries permitted until 2026-11-03 13:29:50.723701212 +0000 UTC m=+1270.731892487 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/9b739270-1c05-44b6-bed6-de7eaadb689f-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-a62d22-cf4789xvd" (UID: "9b739270-1c05-44b6-bed6-de7eaadb689f") : secret "agc-cred-e942f4e7c82f" not found Nov 03 13:29:18.895755 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:29:18.895705 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-5bffndnqx" podUID="6908d762-9f04-4823-a7f2-00a4e703b601" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 13:29:21.344345 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:29:21.344300 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-extmod-shr-user-805818/e2e-usage-sim-ea7ebc-85b488466-wxqgk"] Nov 03 13:29:21.349787 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:29:21.349760 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-extmod-shr-user-805818/e2e-usage-sim-ea7ebc-85b488466-wxqgk" Nov 03 13:29:21.355551 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:29:21.355511 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-extmod-shr-user-805818\"/\"openshift-service-ca.crt\"" Nov 03 13:29:21.355769 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:29:21.355743 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-extmod-shr-user-805818/e2e-usage-sim-ea7ebc-85b488466-wxqgk"] Nov 03 13:29:21.356965 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:29:21.356768 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-extmod-shr-user-805818\"/\"e2e-usage-sim-ea7ebc-cert\"" Nov 03 13:29:21.356965 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:29:21.356825 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-extmod-shr-user-805818\"/\"kube-root-ca.crt\"" Nov 03 13:29:21.356965 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:29:21.356851 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-extmod-shr-user-805818\"/\"default-dockercfg-fn94n\"" Nov 03 13:29:21.481937 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:29:21.481895 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/b0aac0fd-ce21-463d-b6f8-4830f81582e2-tls\") pod \"e2e-usage-sim-ea7ebc-85b488466-wxqgk\" (UID: \"b0aac0fd-ce21-463d-b6f8-4830f81582e2\") " pod="e2e-extmod-shr-user-805818/e2e-usage-sim-ea7ebc-85b488466-wxqgk" Nov 03 13:29:21.482155 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:29:21.482043 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dj5v6\" (UniqueName: \"kubernetes.io/projected/b0aac0fd-ce21-463d-b6f8-4830f81582e2-kube-api-access-dj5v6\") pod \"e2e-usage-sim-ea7ebc-85b488466-wxqgk\" (UID: \"b0aac0fd-ce21-463d-b6f8-4830f81582e2\") " pod="e2e-extmod-shr-user-805818/e2e-usage-sim-ea7ebc-85b488466-wxqgk" Nov 03 13:29:21.583659 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:29:21.583621 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dj5v6\" (UniqueName: \"kubernetes.io/projected/b0aac0fd-ce21-463d-b6f8-4830f81582e2-kube-api-access-dj5v6\") pod \"e2e-usage-sim-ea7ebc-85b488466-wxqgk\" (UID: \"b0aac0fd-ce21-463d-b6f8-4830f81582e2\") " pod="e2e-extmod-shr-user-805818/e2e-usage-sim-ea7ebc-85b488466-wxqgk" Nov 03 13:29:21.583888 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:29:21.583873 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/b0aac0fd-ce21-463d-b6f8-4830f81582e2-tls\") pod \"e2e-usage-sim-ea7ebc-85b488466-wxqgk\" (UID: \"b0aac0fd-ce21-463d-b6f8-4830f81582e2\") " pod="e2e-extmod-shr-user-805818/e2e-usage-sim-ea7ebc-85b488466-wxqgk" Nov 03 13:29:21.586480 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:29:21.586454 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/b0aac0fd-ce21-463d-b6f8-4830f81582e2-tls\") pod \"e2e-usage-sim-ea7ebc-85b488466-wxqgk\" (UID: \"b0aac0fd-ce21-463d-b6f8-4830f81582e2\") " pod="e2e-extmod-shr-user-805818/e2e-usage-sim-ea7ebc-85b488466-wxqgk" Nov 03 13:29:21.593125 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:29:21.593091 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dj5v6\" (UniqueName: \"kubernetes.io/projected/b0aac0fd-ce21-463d-b6f8-4830f81582e2-kube-api-access-dj5v6\") pod \"e2e-usage-sim-ea7ebc-85b488466-wxqgk\" (UID: \"b0aac0fd-ce21-463d-b6f8-4830f81582e2\") " pod="e2e-extmod-shr-user-805818/e2e-usage-sim-ea7ebc-85b488466-wxqgk" Nov 03 13:29:21.676467 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:29:21.676430 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-extmod-shr-user-805818/e2e-usage-sim-ea7ebc-85b488466-wxqgk" Nov 03 13:29:21.988379 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:29:21.985301 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-extmod-shr-user-805818/e2e-usage-sim-ea7ebc-85b488466-wxqgk"] Nov 03 13:29:22.125374 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:29:22.125340 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-extmod-shr-user-805818/e2e-usage-sim-ea7ebc-85b488466-wxqgk" event={"ID":"b0aac0fd-ce21-463d-b6f8-4830f81582e2","Type":"ContainerStarted","Data":"9d51d381262f2148ea0b5c4bf3a3502de0c81216b393509280f89040340db28a"} Nov 03 13:29:23.150342 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:29:23.142043 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-extmod-shr-user-805818/e2e-usage-sim-ea7ebc-85b488466-wxqgk" event={"ID":"b0aac0fd-ce21-463d-b6f8-4830f81582e2","Type":"ContainerStarted","Data":"3681e931878f9d104470811577ceaab09b1b6d7103af06ea4b59f54d8d3744ec"} Nov 03 13:29:23.150342 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:29:23.143611 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-extmod-shr-user-805818/e2e-usage-sim-ea7ebc-85b488466-wxqgk" Nov 03 13:29:23.190306 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:29:23.181267 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-extmod-shr-user-805818/e2e-usage-sim-ea7ebc-85b488466-wxqgk" podStartSLOduration=2.181248768 podStartE2EDuration="2.181248768s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:29:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:29:23.178804122 +0000 UTC m=+1243.186995401" watchObservedRunningTime="2026-11-03 13:29:23.181248768 +0000 UTC m=+1243.189440045" Nov 03 13:29:23.190306 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:29:23.186791 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-extmod-shr-user-805818/e2e-usage-sim-ea7ebc-85b488466-wxqgk" Nov 03 13:29:25.761817 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:29:25.761770 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-84587756dc-nr5hn"] Nov 03 13:29:25.767984 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:29:25.767947 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-nr5hn" Nov 03 13:29:25.798196 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:29:25.798165 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-84587756dc-nr5hn"] Nov 03 13:29:25.829545 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:29:25.829505 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3cd0d23e-9823-4988-ac79-0820dcc4c028-istio-envoy\") pod \"maas-default-gateway-openshift-default-84587756dc-nr5hn\" (UID: \"3cd0d23e-9823-4988-ac79-0820dcc4c028\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-nr5hn" Nov 03 13:29:25.829735 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:29:25.829579 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3cd0d23e-9823-4988-ac79-0820dcc4c028-istio-token\") pod \"maas-default-gateway-openshift-default-84587756dc-nr5hn\" (UID: \"3cd0d23e-9823-4988-ac79-0820dcc4c028\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-nr5hn" Nov 03 13:29:25.829735 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:29:25.829688 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rhbfk\" (UniqueName: \"kubernetes.io/projected/3cd0d23e-9823-4988-ac79-0820dcc4c028-kube-api-access-rhbfk\") pod \"maas-default-gateway-openshift-default-84587756dc-nr5hn\" (UID: \"3cd0d23e-9823-4988-ac79-0820dcc4c028\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-nr5hn" Nov 03 13:29:25.829859 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:29:25.829798 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3cd0d23e-9823-4988-ac79-0820dcc4c028-istio-data\") pod \"maas-default-gateway-openshift-default-84587756dc-nr5hn\" (UID: \"3cd0d23e-9823-4988-ac79-0820dcc4c028\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-nr5hn" Nov 03 13:29:25.829859 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:29:25.829823 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cd0d23e-9823-4988-ac79-0820dcc4c028-workload-socket\") pod \"maas-default-gateway-openshift-default-84587756dc-nr5hn\" (UID: \"3cd0d23e-9823-4988-ac79-0820dcc4c028\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-nr5hn" Nov 03 13:29:25.830013 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:29:25.829993 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3cd0d23e-9823-4988-ac79-0820dcc4c028-workload-certs\") pod \"maas-default-gateway-openshift-default-84587756dc-nr5hn\" (UID: \"3cd0d23e-9823-4988-ac79-0820dcc4c028\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-nr5hn" Nov 03 13:29:25.830072 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:29:25.830020 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3cd0d23e-9823-4988-ac79-0820dcc4c028-istio-podinfo\") pod \"maas-default-gateway-openshift-default-84587756dc-nr5hn\" (UID: \"3cd0d23e-9823-4988-ac79-0820dcc4c028\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-nr5hn" Nov 03 13:29:25.830072 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:29:25.830038 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3cd0d23e-9823-4988-ac79-0820dcc4c028-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-84587756dc-nr5hn\" (UID: \"3cd0d23e-9823-4988-ac79-0820dcc4c028\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-nr5hn" Nov 03 13:29:25.830203 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:29:25.830142 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cd0d23e-9823-4988-ac79-0820dcc4c028-credential-socket\") pod \"maas-default-gateway-openshift-default-84587756dc-nr5hn\" (UID: \"3cd0d23e-9823-4988-ac79-0820dcc4c028\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-nr5hn" Nov 03 13:29:25.931802 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:29:25.931752 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhbfk\" (UniqueName: \"kubernetes.io/projected/3cd0d23e-9823-4988-ac79-0820dcc4c028-kube-api-access-rhbfk\") pod \"maas-default-gateway-openshift-default-84587756dc-nr5hn\" (UID: \"3cd0d23e-9823-4988-ac79-0820dcc4c028\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-nr5hn" Nov 03 13:29:25.931982 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:29:25.931852 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3cd0d23e-9823-4988-ac79-0820dcc4c028-istio-data\") pod \"maas-default-gateway-openshift-default-84587756dc-nr5hn\" (UID: \"3cd0d23e-9823-4988-ac79-0820dcc4c028\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-nr5hn" Nov 03 13:29:25.931982 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:29:25.931872 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cd0d23e-9823-4988-ac79-0820dcc4c028-workload-socket\") pod \"maas-default-gateway-openshift-default-84587756dc-nr5hn\" (UID: \"3cd0d23e-9823-4988-ac79-0820dcc4c028\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-nr5hn" Nov 03 13:29:25.931982 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:29:25.931974 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3cd0d23e-9823-4988-ac79-0820dcc4c028-workload-certs\") pod \"maas-default-gateway-openshift-default-84587756dc-nr5hn\" (UID: \"3cd0d23e-9823-4988-ac79-0820dcc4c028\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-nr5hn" Nov 03 13:29:25.932173 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:29:25.931991 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3cd0d23e-9823-4988-ac79-0820dcc4c028-istio-podinfo\") pod \"maas-default-gateway-openshift-default-84587756dc-nr5hn\" (UID: \"3cd0d23e-9823-4988-ac79-0820dcc4c028\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-nr5hn" Nov 03 13:29:25.932173 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:29:25.932015 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3cd0d23e-9823-4988-ac79-0820dcc4c028-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-84587756dc-nr5hn\" (UID: \"3cd0d23e-9823-4988-ac79-0820dcc4c028\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-nr5hn" Nov 03 13:29:25.932173 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:29:25.932074 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cd0d23e-9823-4988-ac79-0820dcc4c028-credential-socket\") pod \"maas-default-gateway-openshift-default-84587756dc-nr5hn\" (UID: \"3cd0d23e-9823-4988-ac79-0820dcc4c028\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-nr5hn" Nov 03 13:29:25.932173 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:29:25.932137 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3cd0d23e-9823-4988-ac79-0820dcc4c028-istio-envoy\") pod \"maas-default-gateway-openshift-default-84587756dc-nr5hn\" (UID: \"3cd0d23e-9823-4988-ac79-0820dcc4c028\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-nr5hn" Nov 03 13:29:25.932439 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:29:25.932217 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3cd0d23e-9823-4988-ac79-0820dcc4c028-istio-token\") pod \"maas-default-gateway-openshift-default-84587756dc-nr5hn\" (UID: \"3cd0d23e-9823-4988-ac79-0820dcc4c028\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-nr5hn" Nov 03 13:29:25.932439 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:29:25.932404 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cd0d23e-9823-4988-ac79-0820dcc4c028-workload-socket\") pod \"maas-default-gateway-openshift-default-84587756dc-nr5hn\" (UID: \"3cd0d23e-9823-4988-ac79-0820dcc4c028\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-nr5hn" Nov 03 13:29:25.932439 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:29:25.932406 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3cd0d23e-9823-4988-ac79-0820dcc4c028-istio-data\") pod \"maas-default-gateway-openshift-default-84587756dc-nr5hn\" (UID: \"3cd0d23e-9823-4988-ac79-0820dcc4c028\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-nr5hn" Nov 03 13:29:25.932587 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:29:25.932435 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3cd0d23e-9823-4988-ac79-0820dcc4c028-workload-certs\") pod \"maas-default-gateway-openshift-default-84587756dc-nr5hn\" (UID: \"3cd0d23e-9823-4988-ac79-0820dcc4c028\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-nr5hn" Nov 03 13:29:25.932689 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:29:25.932664 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cd0d23e-9823-4988-ac79-0820dcc4c028-credential-socket\") pod \"maas-default-gateway-openshift-default-84587756dc-nr5hn\" (UID: \"3cd0d23e-9823-4988-ac79-0820dcc4c028\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-nr5hn" Nov 03 13:29:25.933384 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:29:25.933363 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3cd0d23e-9823-4988-ac79-0820dcc4c028-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-84587756dc-nr5hn\" (UID: \"3cd0d23e-9823-4988-ac79-0820dcc4c028\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-nr5hn" Nov 03 13:29:25.934668 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:29:25.934644 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3cd0d23e-9823-4988-ac79-0820dcc4c028-istio-envoy\") pod \"maas-default-gateway-openshift-default-84587756dc-nr5hn\" (UID: \"3cd0d23e-9823-4988-ac79-0820dcc4c028\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-nr5hn" Nov 03 13:29:25.934668 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:29:25.934669 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3cd0d23e-9823-4988-ac79-0820dcc4c028-istio-podinfo\") pod \"maas-default-gateway-openshift-default-84587756dc-nr5hn\" (UID: \"3cd0d23e-9823-4988-ac79-0820dcc4c028\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-nr5hn" Nov 03 13:29:25.942344 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:29:25.942270 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3cd0d23e-9823-4988-ac79-0820dcc4c028-istio-token\") pod \"maas-default-gateway-openshift-default-84587756dc-nr5hn\" (UID: \"3cd0d23e-9823-4988-ac79-0820dcc4c028\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-nr5hn" Nov 03 13:29:25.942488 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:29:25.942443 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rhbfk\" (UniqueName: \"kubernetes.io/projected/3cd0d23e-9823-4988-ac79-0820dcc4c028-kube-api-access-rhbfk\") pod \"maas-default-gateway-openshift-default-84587756dc-nr5hn\" (UID: \"3cd0d23e-9823-4988-ac79-0820dcc4c028\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-nr5hn" Nov 03 13:29:26.005439 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:29:26.005403 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w0-9add72/auth-test-model-9add72-kserve-54f88d5d5-bm2sv"] Nov 03 13:29:26.150039 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:29:26.144987 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-nr5hn" Nov 03 13:29:26.611862 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:29:26.611833 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3cd0d23e_9823_4988_ac79_0820dcc4c028.slice/crio-7d41b7cc92ccd825236e58107052a0126aee2168b939255dcb973cdcdcda12e2 WatchSource:0}: Error finding container 7d41b7cc92ccd825236e58107052a0126aee2168b939255dcb973cdcdcda12e2: Status 404 returned error can't find the container with id 7d41b7cc92ccd825236e58107052a0126aee2168b939255dcb973cdcdcda12e2 Nov 03 13:29:26.612451 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:29:26.612428 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-84587756dc-nr5hn"] Nov 03 13:29:26.614773 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:29:26.614741 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:29:26.614857 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:29:26.614808 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:29:26.614857 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:29:26.614842 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:29:27.244836 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:29:27.244792 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-nr5hn" event={"ID":"3cd0d23e-9823-4988-ac79-0820dcc4c028","Type":"ContainerStarted","Data":"8cef8680dcbf34f6e439e6c1d96440fe7a53d66759de93261895d1190c9769ed"} Nov 03 13:29:27.245364 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:29:27.244861 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-nr5hn" event={"ID":"3cd0d23e-9823-4988-ac79-0820dcc4c028","Type":"ContainerStarted","Data":"7d41b7cc92ccd825236e58107052a0126aee2168b939255dcb973cdcdcda12e2"} Nov 03 13:29:27.283326 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:29:27.283246 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-nr5hn" podStartSLOduration=2.283226236 podStartE2EDuration="2.283226236s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:29:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:29:27.280101184 +0000 UTC m=+1247.288292459" watchObservedRunningTime="2026-11-03 13:29:27.283226236 +0000 UTC m=+1247.291417515" Nov 03 13:29:28.062383 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:29:28.053829 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-ef1693/payload-processing-external-model-e2e-extmod-a-ef1693-7d66w9gk2" Nov 03 13:29:28.217307 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:29:28.208462 5024 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-nr5hn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body= Nov 03 13:29:28.217307 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:29:28.208530 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-nr5hn" podUID="3cd0d23e-9823-4988-ac79-0820dcc4c028" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" Nov 03 13:29:28.243851 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:29:28.243813 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-nr5hn" Nov 03 13:29:28.268700 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:29:28.268670 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-ef1693/payload-processing-external-model-e2e-extmod-a-ef1693-5578jx92q"] Nov 03 13:29:28.891311 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:29:28.890742 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-5bffndnqx" podUID="6908d762-9f04-4823-a7f2-00a4e703b601" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 13:29:29.146641 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:29:29.146557 5024 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-nr5hn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body= Nov 03 13:29:29.146874 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:29:29.146846 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-nr5hn" podUID="3cd0d23e-9823-4988-ac79-0820dcc4c028" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" Nov 03 13:29:30.172309 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:29:30.170191 5024 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-nr5hn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body= Nov 03 13:29:30.172309 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:29:30.170270 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-nr5hn" podUID="3cd0d23e-9823-4988-ac79-0820dcc4c028" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" Nov 03 13:29:31.146167 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:29:31.146135 5024 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-nr5hn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body= Nov 03 13:29:31.146468 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:29:31.146434 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-nr5hn" podUID="3cd0d23e-9823-4988-ac79-0820dcc4c028" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" Nov 03 13:29:32.146412 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:29:32.146373 5024 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-nr5hn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body= Nov 03 13:29:32.146845 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:29:32.146434 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-nr5hn" podUID="3cd0d23e-9823-4988-ac79-0820dcc4c028" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" Nov 03 13:29:33.146416 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:29:33.146379 5024 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-nr5hn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body= Nov 03 13:29:33.146903 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:29:33.146440 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-nr5hn" podUID="3cd0d23e-9823-4988-ac79-0820dcc4c028" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" Nov 03 13:29:33.269702 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:29:33.269659 5024 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-ef1693/payload-processing-external-model-e2e-extmod-a-ef1693-5578jx92q" podUID="5d145099-3543-4d68-a150-a5970cea2829" containerName="payload-processing" containerID="cri-o://394206c46fad301c8b3c7031bec7db1237a7123f16abf866ecaa053687f4cfb9" gracePeriod=25 Nov 03 13:29:33.482971 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:29:33.482933 5024 generic.go:356] "Generic (PLEG): container finished" podID="5d145099-3543-4d68-a150-a5970cea2829" containerID="394206c46fad301c8b3c7031bec7db1237a7123f16abf866ecaa053687f4cfb9" exitCode=0 Nov 03 13:29:33.483239 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:29:33.483217 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-ef1693/payload-processing-external-model-e2e-extmod-a-ef1693-5578jx92q" event={"ID":"5d145099-3543-4d68-a150-a5970cea2829","Type":"ContainerDied","Data":"394206c46fad301c8b3c7031bec7db1237a7123f16abf866ecaa053687f4cfb9"} Nov 03 13:29:33.746156 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:29:33.746126 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-ef1693/payload-processing-external-model-e2e-extmod-a-ef1693-5578jx92q" Nov 03 13:29:33.875134 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:29:33.875095 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d145099-3543-4d68-a150-a5970cea2829-kube-api-access-hvfzz\" (UniqueName: \"kubernetes.io/projected/5d145099-3543-4d68-a150-a5970cea2829-kube-api-access-hvfzz\") pod \"5d145099-3543-4d68-a150-a5970cea2829\" (UID: \"5d145099-3543-4d68-a150-a5970cea2829\") " Nov 03 13:29:33.875370 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:29:33.875250 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5d145099-3543-4d68-a150-a5970cea2829-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d145099-3543-4d68-a150-a5970cea2829-plugins-config-volume\") pod \"5d145099-3543-4d68-a150-a5970cea2829\" (UID: \"5d145099-3543-4d68-a150-a5970cea2829\") " Nov 03 13:29:33.875370 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:29:33.875302 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d145099-3543-4d68-a150-a5970cea2829-provider-credentials\" (UniqueName: \"kubernetes.io/projected/5d145099-3543-4d68-a150-a5970cea2829-provider-credentials\") pod \"5d145099-3543-4d68-a150-a5970cea2829\" (UID: \"5d145099-3543-4d68-a150-a5970cea2829\") " Nov 03 13:29:33.875494 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:29:33.875407 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5d145099-3543-4d68-a150-a5970cea2829-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5d145099-3543-4d68-a150-a5970cea2829-routing-overlay-volume\") pod \"5d145099-3543-4d68-a150-a5970cea2829\" (UID: \"5d145099-3543-4d68-a150-a5970cea2829\") " Nov 03 13:29:33.875998 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:29:33.875948 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d145099-3543-4d68-a150-a5970cea2829-routing-overlay-volume" pod "5d145099-3543-4d68-a150-a5970cea2829" (UID: "5d145099-3543-4d68-a150-a5970cea2829"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:29:33.876121 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:29:33.876014 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d145099-3543-4d68-a150-a5970cea2829-plugins-config-volume" pod "5d145099-3543-4d68-a150-a5970cea2829" (UID: "5d145099-3543-4d68-a150-a5970cea2829"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:29:33.876199 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:29:33.876177 5024 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d145099-3543-4d68-a150-a5970cea2829-plugins-config-volume\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:29:33.876257 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:29:33.876201 5024 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5d145099-3543-4d68-a150-a5970cea2829-routing-overlay-volume\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:29:33.878190 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:29:33.878161 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d145099-3543-4d68-a150-a5970cea2829-kube-api-access-hvfzz" pod "5d145099-3543-4d68-a150-a5970cea2829" (UID: "5d145099-3543-4d68-a150-a5970cea2829"). InnerVolumeSpecName "kube-api-access-hvfzz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:29:33.878312 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:29:33.878249 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d145099-3543-4d68-a150-a5970cea2829-provider-credentials" pod "5d145099-3543-4d68-a150-a5970cea2829" (UID: "5d145099-3543-4d68-a150-a5970cea2829"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:29:34.001979 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:29:34.001876 5024 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5d145099-3543-4d68-a150-a5970cea2829-provider-credentials\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:29:34.002229 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:29:34.002206 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hvfzz\" (UniqueName: \"kubernetes.io/projected/5d145099-3543-4d68-a150-a5970cea2829-kube-api-access-hvfzz\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:29:34.146814 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:29:34.146769 5024 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-nr5hn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body= Nov 03 13:29:34.147250 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:29:34.146841 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-nr5hn" podUID="3cd0d23e-9823-4988-ac79-0820dcc4c028" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" Nov 03 13:29:34.494956 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:29:34.494918 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-ef1693/payload-processing-external-model-e2e-extmod-a-ef1693-5578jx92q" event={"ID":"5d145099-3543-4d68-a150-a5970cea2829","Type":"ContainerDied","Data":"f132e4d18e786e4232bf4ed9d90583e1715108e19026ff090ebcb978d8a68d23"} Nov 03 13:29:34.494956 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:29:34.494961 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-ef1693/payload-processing-external-model-e2e-extmod-a-ef1693-5578jx92q" Nov 03 13:29:34.495234 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:29:34.494985 5024 scope.go:122] "RemoveContainer" containerID="394206c46fad301c8b3c7031bec7db1237a7123f16abf866ecaa053687f4cfb9" Nov 03 13:29:34.523257 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:29:34.523215 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-ef1693/payload-processing-external-model-e2e-extmod-a-ef1693-5578jx92q"] Nov 03 13:29:34.530289 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:29:34.530244 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-ef1693/payload-processing-external-model-e2e-extmod-a-ef1693-5578jx92q"] Nov 03 13:29:35.146665 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:29:35.146622 5024 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-nr5hn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body= Nov 03 13:29:35.146863 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:29:35.146712 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-nr5hn" podUID="3cd0d23e-9823-4988-ac79-0820dcc4c028" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" Nov 03 13:29:36.252323 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:29:36.243900 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d145099-3543-4d68-a150-a5970cea2829" path="/var/lib/kubelet/pods/5d145099-3543-4d68-a150-a5970cea2829/volumes" Nov 03 13:29:37.213310 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:29:37.211395 5024 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-nr5hn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 13:29:37.213310 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:29:37.211464 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-nr5hn" podUID="3cd0d23e-9823-4988-ac79-0820dcc4c028" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 13:29:37.912372 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:29:37.912082 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-nr5hn" Nov 03 13:29:37.913147 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:29:37.913037 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-nr5hn" Nov 03 13:29:37.917219 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:29:37.917192 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-nr5hn" Nov 03 13:29:38.880111 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:29:38.880068 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-5bffndnqx" podUID="6908d762-9f04-4823-a7f2-00a4e703b601" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 13:29:39.559379 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:29:39.557492 5024 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d7mwz" containerName="extractor" Nov 03 13:29:41.036157 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:29:41.036099 5024 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w0-9add72/auth-test-model-9add72-kserve-54f88d5d5-bm2sv" podUID="25181c16-f5c3-49bf-aec6-6c6768d87af9" containerName="main" containerID="cri-o://5aca63baf039a960ae4a1d7a0d74b07c12c101d35216cc0176fe47afc31db294" gracePeriod=45 Nov 03 13:29:41.438208 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:29:41.432482 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w0-9add72/auth-test-model-9add72-kserve-54f88d5d5-bm2sv" podUID="25181c16-f5c3-49bf-aec6-6c6768d87af9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": dial tcp 10.132.0.100:8000: connect: connection refused" Nov 03 13:29:41.707972 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:29:41.707938 5024 generic.go:356] "Generic (PLEG): container finished" podID="25181c16-f5c3-49bf-aec6-6c6768d87af9" containerID="5aca63baf039a960ae4a1d7a0d74b07c12c101d35216cc0176fe47afc31db294" exitCode=0 Nov 03 13:29:41.708176 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:29:41.708008 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-9add72/auth-test-model-9add72-kserve-54f88d5d5-bm2sv" event={"ID":"25181c16-f5c3-49bf-aec6-6c6768d87af9","Type":"ContainerDied","Data":"5aca63baf039a960ae4a1d7a0d74b07c12c101d35216cc0176fe47afc31db294"} Nov 03 13:29:41.988192 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:29:41.988109 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-9add72/auth-test-model-9add72-kserve-54f88d5d5-bm2sv" Nov 03 13:29:42.096359 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:29:42.095965 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/25181c16-f5c3-49bf-aec6-6c6768d87af9-tls-certs\" (UniqueName: \"kubernetes.io/secret/25181c16-f5c3-49bf-aec6-6c6768d87af9-tls-certs\") pod \"25181c16-f5c3-49bf-aec6-6c6768d87af9\" (UID: \"25181c16-f5c3-49bf-aec6-6c6768d87af9\") " Nov 03 13:29:42.096881 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:29:42.096587 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25181c16-f5c3-49bf-aec6-6c6768d87af9-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/25181c16-f5c3-49bf-aec6-6c6768d87af9-model-cache\") pod \"25181c16-f5c3-49bf-aec6-6c6768d87af9\" (UID: \"25181c16-f5c3-49bf-aec6-6c6768d87af9\") " Nov 03 13:29:42.096881 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:29:42.096640 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25181c16-f5c3-49bf-aec6-6c6768d87af9-kube-api-access-ln8ln\" (UniqueName: \"kubernetes.io/projected/25181c16-f5c3-49bf-aec6-6c6768d87af9-kube-api-access-ln8ln\") pod \"25181c16-f5c3-49bf-aec6-6c6768d87af9\" (UID: \"25181c16-f5c3-49bf-aec6-6c6768d87af9\") " Nov 03 13:29:42.096881 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:29:42.096762 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25181c16-f5c3-49bf-aec6-6c6768d87af9-dshm\" (UniqueName: \"kubernetes.io/empty-dir/25181c16-f5c3-49bf-aec6-6c6768d87af9-dshm\") pod \"25181c16-f5c3-49bf-aec6-6c6768d87af9\" (UID: \"25181c16-f5c3-49bf-aec6-6c6768d87af9\") " Nov 03 13:29:42.125862 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:29:42.118632 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25181c16-f5c3-49bf-aec6-6c6768d87af9-model-cache" pod "25181c16-f5c3-49bf-aec6-6c6768d87af9" (UID: "25181c16-f5c3-49bf-aec6-6c6768d87af9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:29:42.125862 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:29:42.120688 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25181c16-f5c3-49bf-aec6-6c6768d87af9-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/25181c16-f5c3-49bf-aec6-6c6768d87af9-tmp-dir\") pod \"25181c16-f5c3-49bf-aec6-6c6768d87af9\" (UID: \"25181c16-f5c3-49bf-aec6-6c6768d87af9\") " Nov 03 13:29:42.125862 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:29:42.120755 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25181c16-f5c3-49bf-aec6-6c6768d87af9-home\" (UniqueName: \"kubernetes.io/empty-dir/25181c16-f5c3-49bf-aec6-6c6768d87af9-home\") pod \"25181c16-f5c3-49bf-aec6-6c6768d87af9\" (UID: \"25181c16-f5c3-49bf-aec6-6c6768d87af9\") " Nov 03 13:29:42.125862 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:29:42.121869 5024 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/25181c16-f5c3-49bf-aec6-6c6768d87af9-model-cache\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:29:42.157016 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:29:42.148492 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25181c16-f5c3-49bf-aec6-6c6768d87af9-tmp-dir" pod "25181c16-f5c3-49bf-aec6-6c6768d87af9" (UID: "25181c16-f5c3-49bf-aec6-6c6768d87af9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:29:42.157016 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:29:42.150758 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25181c16-f5c3-49bf-aec6-6c6768d87af9-kube-api-access-ln8ln" pod "25181c16-f5c3-49bf-aec6-6c6768d87af9" (UID: "25181c16-f5c3-49bf-aec6-6c6768d87af9"). InnerVolumeSpecName "kube-api-access-ln8ln". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:29:42.161308 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:29:42.158421 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25181c16-f5c3-49bf-aec6-6c6768d87af9-home" pod "25181c16-f5c3-49bf-aec6-6c6768d87af9" (UID: "25181c16-f5c3-49bf-aec6-6c6768d87af9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:29:42.166032 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:29:42.165446 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/25181c16-f5c3-49bf-aec6-6c6768d87af9-tls-certs" pod "25181c16-f5c3-49bf-aec6-6c6768d87af9" (UID: "25181c16-f5c3-49bf-aec6-6c6768d87af9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 13:29:42.166032 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:29:42.165920 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25181c16-f5c3-49bf-aec6-6c6768d87af9-dshm" pod "25181c16-f5c3-49bf-aec6-6c6768d87af9" (UID: "25181c16-f5c3-49bf-aec6-6c6768d87af9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:29:42.200692 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:29:42.200654 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-ef1693/payload-processing-external-model-e2e-extmod-a-ef1693-7d66w9gk2"] Nov 03 13:29:42.222999 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:29:42.222968 5024 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/25181c16-f5c3-49bf-aec6-6c6768d87af9-tmp-dir\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:29:42.222999 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:29:42.222995 5024 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/25181c16-f5c3-49bf-aec6-6c6768d87af9-home\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:29:42.222999 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:29:42.223007 5024 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/25181c16-f5c3-49bf-aec6-6c6768d87af9-tls-certs\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:29:42.223269 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:29:42.223018 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ln8ln\" (UniqueName: \"kubernetes.io/projected/25181c16-f5c3-49bf-aec6-6c6768d87af9-kube-api-access-ln8ln\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:29:42.223269 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:29:42.223031 5024 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/25181c16-f5c3-49bf-aec6-6c6768d87af9-dshm\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:29:42.721103 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:29:42.721064 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-9add72/auth-test-model-9add72-kserve-54f88d5d5-bm2sv" Nov 03 13:29:42.721103 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:29:42.721086 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-9add72/auth-test-model-9add72-kserve-54f88d5d5-bm2sv" event={"ID":"25181c16-f5c3-49bf-aec6-6c6768d87af9","Type":"ContainerDied","Data":"a63bec35070320a9d12649f898dedf2701b5e25060c3fb1960efbd1ca290f111"} Nov 03 13:29:42.721386 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:29:42.721150 5024 scope.go:122] "RemoveContainer" containerID="5aca63baf039a960ae4a1d7a0d74b07c12c101d35216cc0176fe47afc31db294" Nov 03 13:29:42.744734 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:29:42.744702 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w0-9add72/auth-test-model-9add72-kserve-54f88d5d5-bm2sv"] Nov 03 13:29:42.750266 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:29:42.750236 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w0-9add72/auth-test-model-9add72-kserve-54f88d5d5-bm2sv"] Nov 03 13:29:44.112872 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:29:44.112824 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="25181c16-f5c3-49bf-aec6-6c6768d87af9" path="/var/lib/kubelet/pods/25181c16-f5c3-49bf-aec6-6c6768d87af9/volumes" Nov 03 13:29:47.201872 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:29:47.201774 5024 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-ef1693/payload-processing-external-model-e2e-extmod-a-ef1693-7d66w9gk2" podUID="dd149ad0-f82e-4c7a-a439-ff9a7ff9a6ad" containerName="payload-processing" containerID="cri-o://029683318cfff514471a48d5da49f21bf5c261c1461acc9fa9778d7e1d69289b" gracePeriod=25 Nov 03 13:29:47.513267 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:29:47.513231 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-ef1693/payload-processing-external-model-e2e-extmod-a-ef1693-7d66w9gk2" Nov 03 13:29:47.683308 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:29:47.662942 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd149ad0-f82e-4c7a-a439-ff9a7ff9a6ad-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/dd149ad0-f82e-4c7a-a439-ff9a7ff9a6ad-routing-overlay-volume\") pod \"dd149ad0-f82e-4c7a-a439-ff9a7ff9a6ad\" (UID: \"dd149ad0-f82e-4c7a-a439-ff9a7ff9a6ad\") " Nov 03 13:29:47.683308 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:29:47.663387 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd149ad0-f82e-4c7a-a439-ff9a7ff9a6ad-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd149ad0-f82e-4c7a-a439-ff9a7ff9a6ad-plugins-config-volume\") pod \"dd149ad0-f82e-4c7a-a439-ff9a7ff9a6ad\" (UID: \"dd149ad0-f82e-4c7a-a439-ff9a7ff9a6ad\") " Nov 03 13:29:47.683308 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:29:47.663639 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd149ad0-f82e-4c7a-a439-ff9a7ff9a6ad-provider-credentials\" (UniqueName: \"kubernetes.io/projected/dd149ad0-f82e-4c7a-a439-ff9a7ff9a6ad-provider-credentials\") pod \"dd149ad0-f82e-4c7a-a439-ff9a7ff9a6ad\" (UID: \"dd149ad0-f82e-4c7a-a439-ff9a7ff9a6ad\") " Nov 03 13:29:47.683308 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:29:47.663723 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd149ad0-f82e-4c7a-a439-ff9a7ff9a6ad-kube-api-access-q7fjv\" (UniqueName: \"kubernetes.io/projected/dd149ad0-f82e-4c7a-a439-ff9a7ff9a6ad-kube-api-access-q7fjv\") pod \"dd149ad0-f82e-4c7a-a439-ff9a7ff9a6ad\" (UID: \"dd149ad0-f82e-4c7a-a439-ff9a7ff9a6ad\") " Nov 03 13:29:47.683308 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:29:47.667764 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd149ad0-f82e-4c7a-a439-ff9a7ff9a6ad-plugins-config-volume" pod "dd149ad0-f82e-4c7a-a439-ff9a7ff9a6ad" (UID: "dd149ad0-f82e-4c7a-a439-ff9a7ff9a6ad"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:29:47.683308 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:29:47.670968 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd149ad0-f82e-4c7a-a439-ff9a7ff9a6ad-routing-overlay-volume" pod "dd149ad0-f82e-4c7a-a439-ff9a7ff9a6ad" (UID: "dd149ad0-f82e-4c7a-a439-ff9a7ff9a6ad"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:29:47.710328 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:29:47.698842 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd149ad0-f82e-4c7a-a439-ff9a7ff9a6ad-provider-credentials" pod "dd149ad0-f82e-4c7a-a439-ff9a7ff9a6ad" (UID: "dd149ad0-f82e-4c7a-a439-ff9a7ff9a6ad"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:29:47.715826 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:29:47.715655 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd149ad0-f82e-4c7a-a439-ff9a7ff9a6ad-kube-api-access-q7fjv" pod "dd149ad0-f82e-4c7a-a439-ff9a7ff9a6ad" (UID: "dd149ad0-f82e-4c7a-a439-ff9a7ff9a6ad"). InnerVolumeSpecName "kube-api-access-q7fjv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:29:47.784608 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:29:47.784318 5024 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd149ad0-f82e-4c7a-a439-ff9a7ff9a6ad-plugins-config-volume\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:29:47.784608 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:29:47.784353 5024 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/dd149ad0-f82e-4c7a-a439-ff9a7ff9a6ad-provider-credentials\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:29:47.784608 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:29:47.784384 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q7fjv\" (UniqueName: \"kubernetes.io/projected/dd149ad0-f82e-4c7a-a439-ff9a7ff9a6ad-kube-api-access-q7fjv\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:29:47.784608 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:29:47.784400 5024 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/dd149ad0-f82e-4c7a-a439-ff9a7ff9a6ad-routing-overlay-volume\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:29:47.843378 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:29:47.839060 5024 generic.go:356] "Generic (PLEG): container finished" podID="dd149ad0-f82e-4c7a-a439-ff9a7ff9a6ad" containerID="029683318cfff514471a48d5da49f21bf5c261c1461acc9fa9778d7e1d69289b" exitCode=0 Nov 03 13:29:47.843378 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:29:47.839150 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-ef1693/payload-processing-external-model-e2e-extmod-a-ef1693-7d66w9gk2" event={"ID":"dd149ad0-f82e-4c7a-a439-ff9a7ff9a6ad","Type":"ContainerDied","Data":"029683318cfff514471a48d5da49f21bf5c261c1461acc9fa9778d7e1d69289b"} Nov 03 13:29:47.843378 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:29:47.839194 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-ef1693/payload-processing-external-model-e2e-extmod-a-ef1693-7d66w9gk2" event={"ID":"dd149ad0-f82e-4c7a-a439-ff9a7ff9a6ad","Type":"ContainerDied","Data":"e227810dde1dc3646f163ab30bfc403c8632d21b04da29957458b921fd2a6330"} Nov 03 13:29:47.843378 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:29:47.839216 5024 scope.go:122] "RemoveContainer" containerID="029683318cfff514471a48d5da49f21bf5c261c1461acc9fa9778d7e1d69289b" Nov 03 13:29:47.843378 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:29:47.839410 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-ef1693/payload-processing-external-model-e2e-extmod-a-ef1693-7d66w9gk2" Nov 03 13:29:47.873567 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:29:47.873538 5024 scope.go:122] "RemoveContainer" containerID="029683318cfff514471a48d5da49f21bf5c261c1461acc9fa9778d7e1d69289b" Nov 03 13:29:47.873977 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:29:47.873947 5024 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"029683318cfff514471a48d5da49f21bf5c261c1461acc9fa9778d7e1d69289b\": container with ID starting with 029683318cfff514471a48d5da49f21bf5c261c1461acc9fa9778d7e1d69289b not found: ID does not exist" containerID="029683318cfff514471a48d5da49f21bf5c261c1461acc9fa9778d7e1d69289b" Nov 03 13:29:47.874062 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:29:47.873994 5024 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"029683318cfff514471a48d5da49f21bf5c261c1461acc9fa9778d7e1d69289b"} err="failed to get container status \"029683318cfff514471a48d5da49f21bf5c261c1461acc9fa9778d7e1d69289b\": rpc error: code = NotFound desc = could not find container \"029683318cfff514471a48d5da49f21bf5c261c1461acc9fa9778d7e1d69289b\": container with ID starting with 029683318cfff514471a48d5da49f21bf5c261c1461acc9fa9778d7e1d69289b not found: ID does not exist" Nov 03 13:29:47.883626 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:29:47.883588 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-ef1693/payload-processing-external-model-e2e-extmod-a-ef1693-7d66w9gk2"] Nov 03 13:29:47.892683 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:29:47.892648 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-ef1693/payload-processing-external-model-e2e-extmod-a-ef1693-7d66w9gk2"] Nov 03 13:29:48.118269 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:29:48.118231 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd149ad0-f82e-4c7a-a439-ff9a7ff9a6ad" path="/var/lib/kubelet/pods/dd149ad0-f82e-4c7a-a439-ff9a7ff9a6ad/volumes" Nov 03 13:29:48.879228 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:29:48.879190 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-5bffndnqx" podUID="6908d762-9f04-4823-a7f2-00a4e703b601" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 13:29:50.761742 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:29:50.761697 5024 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-a62d22/agc-cred-e942f4e7c82f: secret "agc-cred-e942f4e7c82f" not found Nov 03 13:29:50.761742 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:29:50.761737 5024 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-cf4789xvd: secret "agc-cred-e942f4e7c82f" not found Nov 03 13:29:50.762189 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:29:50.761807 5024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9b739270-1c05-44b6-bed6-de7eaadb689f-provider-credentials podName:9b739270-1c05-44b6-bed6-de7eaadb689f nodeName:}" failed. No retries permitted until 2026-11-03 13:30:54.761791689 +0000 UTC m=+1334.769982958 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/9b739270-1c05-44b6-bed6-de7eaadb689f-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-a62d22-cf4789xvd" (UID: "9b739270-1c05-44b6-bed6-de7eaadb689f") : secret "agc-cred-e942f4e7c82f" not found Nov 03 13:29:55.347630 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:29:55.347588 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-bf6dd6dcc-gwprz"] Nov 03 13:29:58.932317 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:29:58.918440 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-5bffndnqx" podUID="6908d762-9f04-4823-a7f2-00a4e703b601" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 13:30:00.369324 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:00.366374 5024 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-bf6dd6dcc-gwprz" podUID="194ba584-d286-4853-bc37-d82ee939ec50" containerName="payload-processing" containerID="cri-o://a248beb231ddfe0b3cc9b63f778c94ef4173c23c039f63539ba1618390d23673" gracePeriod=25 Nov 03 13:30:01.300728 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:01.295873 5024 generic.go:356] "Generic (PLEG): container finished" podID="194ba584-d286-4853-bc37-d82ee939ec50" containerID="a248beb231ddfe0b3cc9b63f778c94ef4173c23c039f63539ba1618390d23673" exitCode=0 Nov 03 13:30:01.300728 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:01.296028 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-bf6dd6dcc-gwprz" event={"ID":"194ba584-d286-4853-bc37-d82ee939ec50","Type":"ContainerDied","Data":"a248beb231ddfe0b3cc9b63f778c94ef4173c23c039f63539ba1618390d23673"} Nov 03 13:30:01.496137 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:01.496096 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-bf6dd6dcc-gwprz" Nov 03 13:30:01.657271 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:01.657230 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/194ba584-d286-4853-bc37-d82ee939ec50-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/194ba584-d286-4853-bc37-d82ee939ec50-plugins-config-volume\") pod \"194ba584-d286-4853-bc37-d82ee939ec50\" (UID: \"194ba584-d286-4853-bc37-d82ee939ec50\") " Nov 03 13:30:01.657735 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:01.657701 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/194ba584-d286-4853-bc37-d82ee939ec50-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/194ba584-d286-4853-bc37-d82ee939ec50-routing-overlay-volume\") pod \"194ba584-d286-4853-bc37-d82ee939ec50\" (UID: \"194ba584-d286-4853-bc37-d82ee939ec50\") " Nov 03 13:30:01.657916 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:01.657893 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/194ba584-d286-4853-bc37-d82ee939ec50-kube-api-access-2mrm9\" (UniqueName: \"kubernetes.io/projected/194ba584-d286-4853-bc37-d82ee939ec50-kube-api-access-2mrm9\") pod \"194ba584-d286-4853-bc37-d82ee939ec50\" (UID: \"194ba584-d286-4853-bc37-d82ee939ec50\") " Nov 03 13:30:01.658030 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:01.657969 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/194ba584-d286-4853-bc37-d82ee939ec50-provider-credentials\" (UniqueName: \"kubernetes.io/projected/194ba584-d286-4853-bc37-d82ee939ec50-provider-credentials\") pod \"194ba584-d286-4853-bc37-d82ee939ec50\" (UID: \"194ba584-d286-4853-bc37-d82ee939ec50\") " Nov 03 13:30:01.658365 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:01.658333 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/194ba584-d286-4853-bc37-d82ee939ec50-plugins-config-volume" pod "194ba584-d286-4853-bc37-d82ee939ec50" (UID: "194ba584-d286-4853-bc37-d82ee939ec50"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:30:01.659669 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:01.659627 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/194ba584-d286-4853-bc37-d82ee939ec50-routing-overlay-volume" pod "194ba584-d286-4853-bc37-d82ee939ec50" (UID: "194ba584-d286-4853-bc37-d82ee939ec50"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:30:01.660797 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:01.660758 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/194ba584-d286-4853-bc37-d82ee939ec50-kube-api-access-2mrm9" pod "194ba584-d286-4853-bc37-d82ee939ec50" (UID: "194ba584-d286-4853-bc37-d82ee939ec50"). InnerVolumeSpecName "kube-api-access-2mrm9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:30:01.662304 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:01.662257 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/194ba584-d286-4853-bc37-d82ee939ec50-provider-credentials" pod "194ba584-d286-4853-bc37-d82ee939ec50" (UID: "194ba584-d286-4853-bc37-d82ee939ec50"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:30:01.760800 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:01.760758 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2mrm9\" (UniqueName: \"kubernetes.io/projected/194ba584-d286-4853-bc37-d82ee939ec50-kube-api-access-2mrm9\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:30:01.760800 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:01.760800 5024 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/194ba584-d286-4853-bc37-d82ee939ec50-provider-credentials\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:30:01.761001 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:01.760813 5024 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/194ba584-d286-4853-bc37-d82ee939ec50-plugins-config-volume\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:30:01.761001 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:01.760822 5024 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/194ba584-d286-4853-bc37-d82ee939ec50-routing-overlay-volume\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:30:02.309455 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:02.309416 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-bf6dd6dcc-gwprz" event={"ID":"194ba584-d286-4853-bc37-d82ee939ec50","Type":"ContainerDied","Data":"627de5b3ff068e059b2a3db9e5d63c1b51d5cc7a0f7253a8fa5a47630cafd276"} Nov 03 13:30:02.309642 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:02.309458 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-bf6dd6dcc-gwprz" Nov 03 13:30:02.309642 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:02.309508 5024 scope.go:122] "RemoveContainer" containerID="a248beb231ddfe0b3cc9b63f778c94ef4173c23c039f63539ba1618390d23673" Nov 03 13:30:02.340316 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:02.340261 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-bf6dd6dcc-gwprz"] Nov 03 13:30:02.348101 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:02.348065 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-bf6dd6dcc-gwprz"] Nov 03 13:30:02.931928 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:02.931894 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-86e0bec8-g62lg"] Nov 03 13:30:02.933436 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:02.933409 5024 cpu_manager.go:414] "containerMap: removing container" podUID="25181c16-f5c3-49bf-aec6-6c6768d87af9" containerName="main" Nov 03 13:30:02.935611 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:02.935587 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25181c16-f5c3-49bf-aec6-6c6768d87af9" containerName="main" Nov 03 13:30:02.941166 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:02.941133 5024 cpu_manager.go:414] "containerMap: removing container" podUID="5d145099-3543-4d68-a150-a5970cea2829" containerName="payload-processing" Nov 03 13:30:02.941377 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:02.941358 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d145099-3543-4d68-a150-a5970cea2829" containerName="payload-processing" Nov 03 13:30:02.941503 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:02.941492 5024 cpu_manager.go:414] "containerMap: removing container" podUID="194ba584-d286-4853-bc37-d82ee939ec50" containerName="payload-processing" Nov 03 13:30:02.941625 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:02.941611 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="194ba584-d286-4853-bc37-d82ee939ec50" containerName="payload-processing" Nov 03 13:30:02.941782 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:02.941771 5024 cpu_manager.go:414] "containerMap: removing container" podUID="dd149ad0-f82e-4c7a-a439-ff9a7ff9a6ad" containerName="payload-processing" Nov 03 13:30:02.941862 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:02.941852 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd149ad0-f82e-4c7a-a439-ff9a7ff9a6ad" containerName="payload-processing" Nov 03 13:30:02.942156 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:02.942141 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="194ba584-d286-4853-bc37-d82ee939ec50" containerName="payload-processing" Nov 03 13:30:02.942267 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:02.942257 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd149ad0-f82e-4c7a-a439-ff9a7ff9a6ad" containerName="payload-processing" Nov 03 13:30:02.942410 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:02.942395 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="25181c16-f5c3-49bf-aec6-6c6768d87af9" containerName="main" Nov 03 13:30:02.942760 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:02.942744 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d145099-3543-4d68-a150-a5970cea2829" containerName="payload-processing" Nov 03 13:30:02.963317 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:02.950539 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-86e0bec8-g62lg"] Nov 03 13:30:02.963317 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:02.950668 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86e0bec8-g62lg" Nov 03 13:30:02.963317 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:02.955803 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 13:30:02.971762 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:02.970436 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-phqfq\"" Nov 03 13:30:03.111485 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:03.111448 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cf47s\" (UniqueName: \"kubernetes.io/projected/928b4c26-66a3-4996-9b33-b9542a24d479-kube-api-access-cf47s\") pod \"maas-api-revoke-sub-86e0bec8-g62lg\" (UID: \"928b4c26-66a3-4996-9b33-b9542a24d479\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86e0bec8-g62lg" Nov 03 13:30:03.111485 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:03.111492 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/928b4c26-66a3-4996-9b33-b9542a24d479-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-86e0bec8-g62lg\" (UID: \"928b4c26-66a3-4996-9b33-b9542a24d479\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86e0bec8-g62lg" Nov 03 13:30:03.111744 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:03.111517 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/928b4c26-66a3-4996-9b33-b9542a24d479-maas-api-xks-ca\") pod \"maas-api-revoke-sub-86e0bec8-g62lg\" (UID: \"928b4c26-66a3-4996-9b33-b9542a24d479\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86e0bec8-g62lg" Nov 03 13:30:03.213421 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:03.213313 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cf47s\" (UniqueName: \"kubernetes.io/projected/928b4c26-66a3-4996-9b33-b9542a24d479-kube-api-access-cf47s\") pod \"maas-api-revoke-sub-86e0bec8-g62lg\" (UID: \"928b4c26-66a3-4996-9b33-b9542a24d479\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86e0bec8-g62lg" Nov 03 13:30:03.213421 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:03.213370 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/928b4c26-66a3-4996-9b33-b9542a24d479-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-86e0bec8-g62lg\" (UID: \"928b4c26-66a3-4996-9b33-b9542a24d479\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86e0bec8-g62lg" Nov 03 13:30:03.213421 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:03.213399 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/928b4c26-66a3-4996-9b33-b9542a24d479-maas-api-xks-ca\") pod \"maas-api-revoke-sub-86e0bec8-g62lg\" (UID: \"928b4c26-66a3-4996-9b33-b9542a24d479\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86e0bec8-g62lg" Nov 03 13:30:03.214921 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:03.214886 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/928b4c26-66a3-4996-9b33-b9542a24d479-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-86e0bec8-g62lg\" (UID: \"928b4c26-66a3-4996-9b33-b9542a24d479\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86e0bec8-g62lg" Nov 03 13:30:03.216652 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:03.216543 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/928b4c26-66a3-4996-9b33-b9542a24d479-maas-api-xks-ca\") pod \"maas-api-revoke-sub-86e0bec8-g62lg\" (UID: \"928b4c26-66a3-4996-9b33-b9542a24d479\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86e0bec8-g62lg" Nov 03 13:30:03.224847 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:03.224811 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cf47s\" (UniqueName: \"kubernetes.io/projected/928b4c26-66a3-4996-9b33-b9542a24d479-kube-api-access-cf47s\") pod \"maas-api-revoke-sub-86e0bec8-g62lg\" (UID: \"928b4c26-66a3-4996-9b33-b9542a24d479\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86e0bec8-g62lg" Nov 03 13:30:03.292572 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:03.292528 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86e0bec8-g62lg" Nov 03 13:30:03.595579 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:30:03.595529 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod928b4c26_66a3_4996_9b33_b9542a24d479.slice/crio-b2a9a686ab69c1467e8fdb4f0fa29d65bd61940037b8cdb49103688192dd2165 WatchSource:0}: Error finding container b2a9a686ab69c1467e8fdb4f0fa29d65bd61940037b8cdb49103688192dd2165: Status 404 returned error can't find the container with id b2a9a686ab69c1467e8fdb4f0fa29d65bd61940037b8cdb49103688192dd2165 Nov 03 13:30:03.596881 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:03.596848 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-86e0bec8-g62lg"] Nov 03 13:30:04.113951 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:04.113905 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="194ba584-d286-4853-bc37-d82ee939ec50" path="/var/lib/kubelet/pods/194ba584-d286-4853-bc37-d82ee939ec50/volumes" Nov 03 13:30:04.331771 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:04.331720 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86e0bec8-g62lg" event={"ID":"928b4c26-66a3-4996-9b33-b9542a24d479","Type":"ContainerStarted","Data":"b2a9a686ab69c1467e8fdb4f0fa29d65bd61940037b8cdb49103688192dd2165"} Nov 03 13:30:08.980014 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:08.979972 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-5bffndnqx" podUID="6908d762-9f04-4823-a7f2-00a4e703b601" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 13:30:09.400413 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:09.400371 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86e0bec8-g62lg" event={"ID":"928b4c26-66a3-4996-9b33-b9542a24d479","Type":"ContainerStarted","Data":"3e5de967862f794dcb98270059cb343b2490d3b5a8dd35d16aeedeb789be3873"} Nov 03 13:30:09.426501 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:09.426431 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86e0bec8-g62lg" podStartSLOduration=2.322033363 podStartE2EDuration="7.426416932s" totalImagesPullingTime="5.104383569s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:30:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:30:09.421867805 +0000 UTC m=+1289.430059096" watchObservedRunningTime="2026-11-03 13:30:09.426416932 +0000 UTC m=+1289.434608209" Nov 03 13:30:09.517947 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:30:09.517906 5024 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d7mwz" containerName="extractor" Nov 03 13:30:10.412260 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:10.412221 5024 generic.go:356] "Generic (PLEG): container finished" podID="928b4c26-66a3-4996-9b33-b9542a24d479" containerID="3e5de967862f794dcb98270059cb343b2490d3b5a8dd35d16aeedeb789be3873" exitCode=0 Nov 03 13:30:10.412766 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:10.412486 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86e0bec8-g62lg" event={"ID":"928b4c26-66a3-4996-9b33-b9542a24d479","Type":"ContainerDied","Data":"3e5de967862f794dcb98270059cb343b2490d3b5a8dd35d16aeedeb789be3873"} Nov 03 13:30:11.102723 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:11.102684 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-5bffndnqx" podUID="6908d762-9f04-4823-a7f2-00a4e703b601" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 13:30:11.712855 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:11.712825 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86e0bec8-g62lg" Nov 03 13:30:11.788231 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:11.788194 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/928b4c26-66a3-4996-9b33-b9542a24d479-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/928b4c26-66a3-4996-9b33-b9542a24d479-maas-api-openshift-ca\") pod \"928b4c26-66a3-4996-9b33-b9542a24d479\" (UID: \"928b4c26-66a3-4996-9b33-b9542a24d479\") " Nov 03 13:30:11.788560 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:11.788307 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/928b4c26-66a3-4996-9b33-b9542a24d479-kube-api-access-cf47s\" (UniqueName: \"kubernetes.io/projected/928b4c26-66a3-4996-9b33-b9542a24d479-kube-api-access-cf47s\") pod \"928b4c26-66a3-4996-9b33-b9542a24d479\" (UID: \"928b4c26-66a3-4996-9b33-b9542a24d479\") " Nov 03 13:30:11.788560 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:11.788512 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/928b4c26-66a3-4996-9b33-b9542a24d479-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/928b4c26-66a3-4996-9b33-b9542a24d479-maas-api-xks-ca\") pod \"928b4c26-66a3-4996-9b33-b9542a24d479\" (UID: \"928b4c26-66a3-4996-9b33-b9542a24d479\") " Nov 03 13:30:11.788721 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:11.788693 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/928b4c26-66a3-4996-9b33-b9542a24d479-maas-api-openshift-ca" pod "928b4c26-66a3-4996-9b33-b9542a24d479" (UID: "928b4c26-66a3-4996-9b33-b9542a24d479"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:30:11.789167 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:11.789145 5024 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/928b4c26-66a3-4996-9b33-b9542a24d479-maas-api-openshift-ca\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:30:11.790662 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:11.790632 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/928b4c26-66a3-4996-9b33-b9542a24d479-maas-api-xks-ca" pod "928b4c26-66a3-4996-9b33-b9542a24d479" (UID: "928b4c26-66a3-4996-9b33-b9542a24d479"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 13:30:11.790756 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:11.790696 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/928b4c26-66a3-4996-9b33-b9542a24d479-kube-api-access-cf47s" pod "928b4c26-66a3-4996-9b33-b9542a24d479" (UID: "928b4c26-66a3-4996-9b33-b9542a24d479"). InnerVolumeSpecName "kube-api-access-cf47s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:30:11.890715 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:11.890680 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cf47s\" (UniqueName: \"kubernetes.io/projected/928b4c26-66a3-4996-9b33-b9542a24d479-kube-api-access-cf47s\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:30:11.890715 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:11.890712 5024 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/928b4c26-66a3-4996-9b33-b9542a24d479-maas-api-xks-ca\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:30:12.431362 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:12.431234 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86e0bec8-g62lg" Nov 03 13:30:12.431362 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:12.431266 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86e0bec8-g62lg" event={"ID":"928b4c26-66a3-4996-9b33-b9542a24d479","Type":"ContainerDied","Data":"b2a9a686ab69c1467e8fdb4f0fa29d65bd61940037b8cdb49103688192dd2165"} Nov 03 13:30:12.431549 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:12.431361 5024 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b2a9a686ab69c1467e8fdb4f0fa29d65bd61940037b8cdb49103688192dd2165" Nov 03 13:30:13.965459 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:13.965233 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-6958bdd4ff-m66rf"] Nov 03 13:30:13.967308 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:13.967250 5024 cpu_manager.go:414] "containerMap: removing container" podUID="928b4c26-66a3-4996-9b33-b9542a24d479" containerName="revoke-keys" Nov 03 13:30:13.967494 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:13.967478 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="928b4c26-66a3-4996-9b33-b9542a24d479" containerName="revoke-keys" Nov 03 13:30:13.967809 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:13.967790 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="928b4c26-66a3-4996-9b33-b9542a24d479" containerName="revoke-keys" Nov 03 13:30:13.972886 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:13.972854 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6958bdd4ff-m66rf" Nov 03 13:30:13.985605 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:13.983338 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\"" Nov 03 13:30:13.985605 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:13.983462 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-vx42v\"" Nov 03 13:30:13.985605 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:13.984101 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-1d111babc549\"" Nov 03 13:30:13.985605 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:13.984182 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-3de2312914d9\"" Nov 03 13:30:13.985605 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:13.984263 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\"" Nov 03 13:30:13.986364 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:13.986151 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\"" Nov 03 13:30:13.986364 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:13.986261 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\"" Nov 03 13:30:14.011719 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:14.011677 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-6958bdd4ff-m66rf"] Nov 03 13:30:14.078213 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:14.077953 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rdjnx\" (UniqueName: \"kubernetes.io/projected/90ddc3f3-a4b4-4ddd-9bfe-175af740f60e-kube-api-access-rdjnx\") pod \"payload-processing-external-model-6958bdd4ff-m66rf\" (UID: \"90ddc3f3-a4b4-4ddd-9bfe-175af740f60e\") " pod="models-as-a-service/payload-processing-external-model-6958bdd4ff-m66rf" Nov 03 13:30:14.078591 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:14.078522 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/90ddc3f3-a4b4-4ddd-9bfe-175af740f60e-provider-credentials\") pod \"payload-processing-external-model-6958bdd4ff-m66rf\" (UID: \"90ddc3f3-a4b4-4ddd-9bfe-175af740f60e\") " pod="models-as-a-service/payload-processing-external-model-6958bdd4ff-m66rf" Nov 03 13:30:14.078952 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:14.078933 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90ddc3f3-a4b4-4ddd-9bfe-175af740f60e-plugins-config-volume\") pod \"payload-processing-external-model-6958bdd4ff-m66rf\" (UID: \"90ddc3f3-a4b4-4ddd-9bfe-175af740f60e\") " pod="models-as-a-service/payload-processing-external-model-6958bdd4ff-m66rf" Nov 03 13:30:14.079097 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:14.079081 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/90ddc3f3-a4b4-4ddd-9bfe-175af740f60e-routing-overlay-volume\") pod \"payload-processing-external-model-6958bdd4ff-m66rf\" (UID: \"90ddc3f3-a4b4-4ddd-9bfe-175af740f60e\") " pod="models-as-a-service/payload-processing-external-model-6958bdd4ff-m66rf" Nov 03 13:30:14.180847 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:14.180782 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdjnx\" (UniqueName: \"kubernetes.io/projected/90ddc3f3-a4b4-4ddd-9bfe-175af740f60e-kube-api-access-rdjnx\") pod \"payload-processing-external-model-6958bdd4ff-m66rf\" (UID: \"90ddc3f3-a4b4-4ddd-9bfe-175af740f60e\") " pod="models-as-a-service/payload-processing-external-model-6958bdd4ff-m66rf" Nov 03 13:30:14.189888 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:14.189853 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/90ddc3f3-a4b4-4ddd-9bfe-175af740f60e-provider-credentials\") pod \"payload-processing-external-model-6958bdd4ff-m66rf\" (UID: \"90ddc3f3-a4b4-4ddd-9bfe-175af740f60e\") " pod="models-as-a-service/payload-processing-external-model-6958bdd4ff-m66rf" Nov 03 13:30:14.190943 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:14.190909 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90ddc3f3-a4b4-4ddd-9bfe-175af740f60e-plugins-config-volume\") pod \"payload-processing-external-model-6958bdd4ff-m66rf\" (UID: \"90ddc3f3-a4b4-4ddd-9bfe-175af740f60e\") " pod="models-as-a-service/payload-processing-external-model-6958bdd4ff-m66rf" Nov 03 13:30:14.191191 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:14.191174 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/90ddc3f3-a4b4-4ddd-9bfe-175af740f60e-routing-overlay-volume\") pod \"payload-processing-external-model-6958bdd4ff-m66rf\" (UID: \"90ddc3f3-a4b4-4ddd-9bfe-175af740f60e\") " pod="models-as-a-service/payload-processing-external-model-6958bdd4ff-m66rf" Nov 03 13:30:14.192523 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:14.192367 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/90ddc3f3-a4b4-4ddd-9bfe-175af740f60e-routing-overlay-volume\") pod \"payload-processing-external-model-6958bdd4ff-m66rf\" (UID: \"90ddc3f3-a4b4-4ddd-9bfe-175af740f60e\") " pod="models-as-a-service/payload-processing-external-model-6958bdd4ff-m66rf" Nov 03 13:30:14.193345 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:14.193314 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90ddc3f3-a4b4-4ddd-9bfe-175af740f60e-plugins-config-volume\") pod \"payload-processing-external-model-6958bdd4ff-m66rf\" (UID: \"90ddc3f3-a4b4-4ddd-9bfe-175af740f60e\") " pod="models-as-a-service/payload-processing-external-model-6958bdd4ff-m66rf" Nov 03 13:30:14.198464 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:14.198411 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rdjnx\" (UniqueName: \"kubernetes.io/projected/90ddc3f3-a4b4-4ddd-9bfe-175af740f60e-kube-api-access-rdjnx\") pod \"payload-processing-external-model-6958bdd4ff-m66rf\" (UID: \"90ddc3f3-a4b4-4ddd-9bfe-175af740f60e\") " pod="models-as-a-service/payload-processing-external-model-6958bdd4ff-m66rf" Nov 03 13:30:14.199815 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:14.199777 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/90ddc3f3-a4b4-4ddd-9bfe-175af740f60e-provider-credentials\") pod \"payload-processing-external-model-6958bdd4ff-m66rf\" (UID: \"90ddc3f3-a4b4-4ddd-9bfe-175af740f60e\") " pod="models-as-a-service/payload-processing-external-model-6958bdd4ff-m66rf" Nov 03 13:30:14.309839 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:14.309736 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6958bdd4ff-m66rf" Nov 03 13:30:15.056152 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:15.055742 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-6958bdd4ff-m66rf"] Nov 03 13:30:15.846306 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:15.845371 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6958bdd4ff-m66rf" event={"ID":"90ddc3f3-a4b4-4ddd-9bfe-175af740f60e","Type":"ContainerStarted","Data":"852e603f206278ea5a4ce7e9eaa53dc44ee340d8a1507c2283e73a1e1e7ebbc4"} Nov 03 13:30:15.846306 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:15.845444 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6958bdd4ff-m66rf" event={"ID":"90ddc3f3-a4b4-4ddd-9bfe-175af740f60e","Type":"ContainerStarted","Data":"7883579da2ba68d5540b28a32d07155fb1b1b02e775ca781b03b3573a7bfbc46"} Nov 03 13:30:15.847945 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:15.847052 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-6958bdd4ff-m66rf" Nov 03 13:30:15.888858 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:15.888784 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-6958bdd4ff-m66rf" podStartSLOduration=2.8887614360000002 podStartE2EDuration="2.888761436s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:30:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:30:15.865746545 +0000 UTC m=+1295.873937821" watchObservedRunningTime="2026-11-03 13:30:15.888761436 +0000 UTC m=+1295.896952715" Nov 03 13:30:17.003444 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:17.003404 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-6958bdd4ff-m66rf"] Nov 03 13:30:17.098245 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:17.098204 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-55dcdb647d-8hjmz"] Nov 03 13:30:17.103160 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:17.103124 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-55dcdb647d-8hjmz" Nov 03 13:30:17.118954 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:17.118920 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-64a3f5248b8f\"" Nov 03 13:30:17.129766 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:17.129724 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lc8gx\" (UniqueName: \"kubernetes.io/projected/48073b49-8f68-47ec-bdcb-a82aace8f8c8-kube-api-access-lc8gx\") pod \"payload-processing-external-model-55dcdb647d-8hjmz\" (UID: \"48073b49-8f68-47ec-bdcb-a82aace8f8c8\") " pod="models-as-a-service/payload-processing-external-model-55dcdb647d-8hjmz" Nov 03 13:30:17.129954 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:17.129848 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/48073b49-8f68-47ec-bdcb-a82aace8f8c8-plugins-config-volume\") pod \"payload-processing-external-model-55dcdb647d-8hjmz\" (UID: \"48073b49-8f68-47ec-bdcb-a82aace8f8c8\") " pod="models-as-a-service/payload-processing-external-model-55dcdb647d-8hjmz" Nov 03 13:30:17.130059 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:17.130038 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/48073b49-8f68-47ec-bdcb-a82aace8f8c8-routing-overlay-volume\") pod \"payload-processing-external-model-55dcdb647d-8hjmz\" (UID: \"48073b49-8f68-47ec-bdcb-a82aace8f8c8\") " pod="models-as-a-service/payload-processing-external-model-55dcdb647d-8hjmz" Nov 03 13:30:17.130201 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:17.130181 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/48073b49-8f68-47ec-bdcb-a82aace8f8c8-provider-credentials\") pod \"payload-processing-external-model-55dcdb647d-8hjmz\" (UID: \"48073b49-8f68-47ec-bdcb-a82aace8f8c8\") " pod="models-as-a-service/payload-processing-external-model-55dcdb647d-8hjmz" Nov 03 13:30:17.145698 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:17.145662 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-55dcdb647d-8hjmz"] Nov 03 13:30:17.232230 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:17.232176 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/48073b49-8f68-47ec-bdcb-a82aace8f8c8-routing-overlay-volume\") pod \"payload-processing-external-model-55dcdb647d-8hjmz\" (UID: \"48073b49-8f68-47ec-bdcb-a82aace8f8c8\") " pod="models-as-a-service/payload-processing-external-model-55dcdb647d-8hjmz" Nov 03 13:30:17.232479 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:17.232423 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/48073b49-8f68-47ec-bdcb-a82aace8f8c8-provider-credentials\") pod \"payload-processing-external-model-55dcdb647d-8hjmz\" (UID: \"48073b49-8f68-47ec-bdcb-a82aace8f8c8\") " pod="models-as-a-service/payload-processing-external-model-55dcdb647d-8hjmz" Nov 03 13:30:17.233019 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:17.232945 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lc8gx\" (UniqueName: \"kubernetes.io/projected/48073b49-8f68-47ec-bdcb-a82aace8f8c8-kube-api-access-lc8gx\") pod \"payload-processing-external-model-55dcdb647d-8hjmz\" (UID: \"48073b49-8f68-47ec-bdcb-a82aace8f8c8\") " pod="models-as-a-service/payload-processing-external-model-55dcdb647d-8hjmz" Nov 03 13:30:17.233147 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:17.233021 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/48073b49-8f68-47ec-bdcb-a82aace8f8c8-plugins-config-volume\") pod \"payload-processing-external-model-55dcdb647d-8hjmz\" (UID: \"48073b49-8f68-47ec-bdcb-a82aace8f8c8\") " pod="models-as-a-service/payload-processing-external-model-55dcdb647d-8hjmz" Nov 03 13:30:17.233147 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:17.233026 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/48073b49-8f68-47ec-bdcb-a82aace8f8c8-routing-overlay-volume\") pod \"payload-processing-external-model-55dcdb647d-8hjmz\" (UID: \"48073b49-8f68-47ec-bdcb-a82aace8f8c8\") " pod="models-as-a-service/payload-processing-external-model-55dcdb647d-8hjmz" Nov 03 13:30:17.233299 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:30:17.233266 5024 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-64a3f5248b8f: secret "agc-cred-64a3f5248b8f" not found Nov 03 13:30:17.233371 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:30:17.233302 5024 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-55dcdb647d-8hjmz: secret "agc-cred-64a3f5248b8f" not found Nov 03 13:30:17.233371 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:30:17.233360 5024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/48073b49-8f68-47ec-bdcb-a82aace8f8c8-provider-credentials podName:48073b49-8f68-47ec-bdcb-a82aace8f8c8 nodeName:}" failed. No retries permitted until 2026-11-03 13:30:17.733342558 +0000 UTC m=+1297.741533830 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/48073b49-8f68-47ec-bdcb-a82aace8f8c8-provider-credentials") pod "payload-processing-external-model-55dcdb647d-8hjmz" (UID: "48073b49-8f68-47ec-bdcb-a82aace8f8c8") : secret "agc-cred-64a3f5248b8f" not found Nov 03 13:30:17.233844 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:17.233820 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/48073b49-8f68-47ec-bdcb-a82aace8f8c8-plugins-config-volume\") pod \"payload-processing-external-model-55dcdb647d-8hjmz\" (UID: \"48073b49-8f68-47ec-bdcb-a82aace8f8c8\") " pod="models-as-a-service/payload-processing-external-model-55dcdb647d-8hjmz" Nov 03 13:30:17.254384 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:17.247556 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lc8gx\" (UniqueName: \"kubernetes.io/projected/48073b49-8f68-47ec-bdcb-a82aace8f8c8-kube-api-access-lc8gx\") pod \"payload-processing-external-model-55dcdb647d-8hjmz\" (UID: \"48073b49-8f68-47ec-bdcb-a82aace8f8c8\") " pod="models-as-a-service/payload-processing-external-model-55dcdb647d-8hjmz" Nov 03 13:30:17.668757 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:17.668704 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-55dcdb647d-8hjmz"] Nov 03 13:30:17.672020 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:30:17.670754 5024 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[provider-credentials], unattached volumes=[], failed to process volumes=[]: context canceled" pod="models-as-a-service/payload-processing-external-model-55dcdb647d-8hjmz" podUID="48073b49-8f68-47ec-bdcb-a82aace8f8c8" Nov 03 13:30:17.751631 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:17.751561 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/48073b49-8f68-47ec-bdcb-a82aace8f8c8-provider-credentials\") pod \"payload-processing-external-model-55dcdb647d-8hjmz\" (UID: \"48073b49-8f68-47ec-bdcb-a82aace8f8c8\") " pod="models-as-a-service/payload-processing-external-model-55dcdb647d-8hjmz" Nov 03 13:30:17.784342 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:30:17.767395 5024 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-64a3f5248b8f: secret "agc-cred-64a3f5248b8f" not found Nov 03 13:30:17.784342 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:30:17.767457 5024 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-55dcdb647d-8hjmz: secret "agc-cred-64a3f5248b8f" not found Nov 03 13:30:17.784342 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:30:17.767621 5024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/48073b49-8f68-47ec-bdcb-a82aace8f8c8-provider-credentials podName:48073b49-8f68-47ec-bdcb-a82aace8f8c8 nodeName:}" failed. No retries permitted until 2026-11-03 13:30:18.767583096 +0000 UTC m=+1298.775774368 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/48073b49-8f68-47ec-bdcb-a82aace8f8c8-provider-credentials") pod "payload-processing-external-model-55dcdb647d-8hjmz" (UID: "48073b49-8f68-47ec-bdcb-a82aace8f8c8") : secret "agc-cred-64a3f5248b8f" not found Nov 03 13:30:17.784342 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:17.768089 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-6958bdd4ff-zdwmx"] Nov 03 13:30:17.814645 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:17.814608 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6958bdd4ff-zdwmx" Nov 03 13:30:17.830444 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:17.830400 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-6958bdd4ff-zdwmx"] Nov 03 13:30:17.864077 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:17.864040 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-55dcdb647d-8hjmz" Nov 03 13:30:17.873911 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:17.873877 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-55dcdb647d-8hjmz" Nov 03 13:30:17.956216 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:17.956122 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/48073b49-8f68-47ec-bdcb-a82aace8f8c8-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/48073b49-8f68-47ec-bdcb-a82aace8f8c8-routing-overlay-volume\") pod \"48073b49-8f68-47ec-bdcb-a82aace8f8c8\" (UID: \"48073b49-8f68-47ec-bdcb-a82aace8f8c8\") " Nov 03 13:30:17.956526 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:17.956499 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/69adcc7f-84e5-4bb0-a312-9e0918f9b888-routing-overlay-volume\") pod \"payload-processing-external-model-6958bdd4ff-zdwmx\" (UID: \"69adcc7f-84e5-4bb0-a312-9e0918f9b888\") " pod="models-as-a-service/payload-processing-external-model-6958bdd4ff-zdwmx" Nov 03 13:30:17.956663 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:17.956641 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48073b49-8f68-47ec-bdcb-a82aace8f8c8-routing-overlay-volume" pod "48073b49-8f68-47ec-bdcb-a82aace8f8c8" (UID: "48073b49-8f68-47ec-bdcb-a82aace8f8c8"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:30:17.956721 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:17.956703 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69adcc7f-84e5-4bb0-a312-9e0918f9b888-plugins-config-volume\") pod \"payload-processing-external-model-6958bdd4ff-zdwmx\" (UID: \"69adcc7f-84e5-4bb0-a312-9e0918f9b888\") " pod="models-as-a-service/payload-processing-external-model-6958bdd4ff-zdwmx" Nov 03 13:30:17.956789 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:17.956778 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mrj2w\" (UniqueName: \"kubernetes.io/projected/69adcc7f-84e5-4bb0-a312-9e0918f9b888-kube-api-access-mrj2w\") pod \"payload-processing-external-model-6958bdd4ff-zdwmx\" (UID: \"69adcc7f-84e5-4bb0-a312-9e0918f9b888\") " pod="models-as-a-service/payload-processing-external-model-6958bdd4ff-zdwmx" Nov 03 13:30:17.956913 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:17.956886 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/69adcc7f-84e5-4bb0-a312-9e0918f9b888-provider-credentials\") pod \"payload-processing-external-model-6958bdd4ff-zdwmx\" (UID: \"69adcc7f-84e5-4bb0-a312-9e0918f9b888\") " pod="models-as-a-service/payload-processing-external-model-6958bdd4ff-zdwmx" Nov 03 13:30:17.957235 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:17.957207 5024 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/48073b49-8f68-47ec-bdcb-a82aace8f8c8-routing-overlay-volume\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:30:18.058533 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:18.058490 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/48073b49-8f68-47ec-bdcb-a82aace8f8c8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/48073b49-8f68-47ec-bdcb-a82aace8f8c8-plugins-config-volume\") pod \"48073b49-8f68-47ec-bdcb-a82aace8f8c8\" (UID: \"48073b49-8f68-47ec-bdcb-a82aace8f8c8\") " Nov 03 13:30:18.059057 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:18.058577 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48073b49-8f68-47ec-bdcb-a82aace8f8c8-kube-api-access-lc8gx\" (UniqueName: \"kubernetes.io/projected/48073b49-8f68-47ec-bdcb-a82aace8f8c8-kube-api-access-lc8gx\") pod \"48073b49-8f68-47ec-bdcb-a82aace8f8c8\" (UID: \"48073b49-8f68-47ec-bdcb-a82aace8f8c8\") " Nov 03 13:30:18.059057 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:18.058995 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48073b49-8f68-47ec-bdcb-a82aace8f8c8-plugins-config-volume" pod "48073b49-8f68-47ec-bdcb-a82aace8f8c8" (UID: "48073b49-8f68-47ec-bdcb-a82aace8f8c8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:30:18.059184 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:18.059092 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/69adcc7f-84e5-4bb0-a312-9e0918f9b888-routing-overlay-volume\") pod \"payload-processing-external-model-6958bdd4ff-zdwmx\" (UID: \"69adcc7f-84e5-4bb0-a312-9e0918f9b888\") " pod="models-as-a-service/payload-processing-external-model-6958bdd4ff-zdwmx" Nov 03 13:30:18.059256 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:18.059235 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69adcc7f-84e5-4bb0-a312-9e0918f9b888-plugins-config-volume\") pod \"payload-processing-external-model-6958bdd4ff-zdwmx\" (UID: \"69adcc7f-84e5-4bb0-a312-9e0918f9b888\") " pod="models-as-a-service/payload-processing-external-model-6958bdd4ff-zdwmx" Nov 03 13:30:18.059344 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:18.059327 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mrj2w\" (UniqueName: \"kubernetes.io/projected/69adcc7f-84e5-4bb0-a312-9e0918f9b888-kube-api-access-mrj2w\") pod \"payload-processing-external-model-6958bdd4ff-zdwmx\" (UID: \"69adcc7f-84e5-4bb0-a312-9e0918f9b888\") " pod="models-as-a-service/payload-processing-external-model-6958bdd4ff-zdwmx" Nov 03 13:30:18.059443 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:18.059429 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/69adcc7f-84e5-4bb0-a312-9e0918f9b888-provider-credentials\") pod \"payload-processing-external-model-6958bdd4ff-zdwmx\" (UID: \"69adcc7f-84e5-4bb0-a312-9e0918f9b888\") " pod="models-as-a-service/payload-processing-external-model-6958bdd4ff-zdwmx" Nov 03 13:30:18.059543 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:18.059531 5024 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/48073b49-8f68-47ec-bdcb-a82aace8f8c8-plugins-config-volume\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:30:18.059905 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:18.059881 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/69adcc7f-84e5-4bb0-a312-9e0918f9b888-routing-overlay-volume\") pod \"payload-processing-external-model-6958bdd4ff-zdwmx\" (UID: \"69adcc7f-84e5-4bb0-a312-9e0918f9b888\") " pod="models-as-a-service/payload-processing-external-model-6958bdd4ff-zdwmx" Nov 03 13:30:18.060040 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:18.060019 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69adcc7f-84e5-4bb0-a312-9e0918f9b888-plugins-config-volume\") pod \"payload-processing-external-model-6958bdd4ff-zdwmx\" (UID: \"69adcc7f-84e5-4bb0-a312-9e0918f9b888\") " pod="models-as-a-service/payload-processing-external-model-6958bdd4ff-zdwmx" Nov 03 13:30:18.060888 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:18.060858 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48073b49-8f68-47ec-bdcb-a82aace8f8c8-kube-api-access-lc8gx" pod "48073b49-8f68-47ec-bdcb-a82aace8f8c8" (UID: "48073b49-8f68-47ec-bdcb-a82aace8f8c8"). InnerVolumeSpecName "kube-api-access-lc8gx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:30:18.062073 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:18.062048 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/69adcc7f-84e5-4bb0-a312-9e0918f9b888-provider-credentials\") pod \"payload-processing-external-model-6958bdd4ff-zdwmx\" (UID: \"69adcc7f-84e5-4bb0-a312-9e0918f9b888\") " pod="models-as-a-service/payload-processing-external-model-6958bdd4ff-zdwmx" Nov 03 13:30:18.068873 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:18.068842 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mrj2w\" (UniqueName: \"kubernetes.io/projected/69adcc7f-84e5-4bb0-a312-9e0918f9b888-kube-api-access-mrj2w\") pod \"payload-processing-external-model-6958bdd4ff-zdwmx\" (UID: \"69adcc7f-84e5-4bb0-a312-9e0918f9b888\") " pod="models-as-a-service/payload-processing-external-model-6958bdd4ff-zdwmx" Nov 03 13:30:18.149104 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:18.149064 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6958bdd4ff-zdwmx" Nov 03 13:30:18.161565 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:18.161532 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lc8gx\" (UniqueName: \"kubernetes.io/projected/48073b49-8f68-47ec-bdcb-a82aace8f8c8-kube-api-access-lc8gx\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:30:18.506691 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:18.506468 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-6958bdd4ff-zdwmx"] Nov 03 13:30:18.828623 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:18.828527 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/48073b49-8f68-47ec-bdcb-a82aace8f8c8-provider-credentials\") pod \"payload-processing-external-model-55dcdb647d-8hjmz\" (UID: \"48073b49-8f68-47ec-bdcb-a82aace8f8c8\") " pod="models-as-a-service/payload-processing-external-model-55dcdb647d-8hjmz" Nov 03 13:30:18.828793 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:30:18.828692 5024 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-64a3f5248b8f: secret "agc-cred-64a3f5248b8f" not found Nov 03 13:30:18.828793 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:30:18.828714 5024 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-55dcdb647d-8hjmz: secret "agc-cred-64a3f5248b8f" not found Nov 03 13:30:18.828793 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:30:18.828771 5024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/48073b49-8f68-47ec-bdcb-a82aace8f8c8-provider-credentials podName:48073b49-8f68-47ec-bdcb-a82aace8f8c8 nodeName:}" failed. No retries permitted until 2026-11-03 13:30:20.828756119 +0000 UTC m=+1300.836947398 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/48073b49-8f68-47ec-bdcb-a82aace8f8c8-provider-credentials") pod "payload-processing-external-model-55dcdb647d-8hjmz" (UID: "48073b49-8f68-47ec-bdcb-a82aace8f8c8") : secret "agc-cred-64a3f5248b8f" not found Nov 03 13:30:18.873372 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:18.873326 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6958bdd4ff-zdwmx" event={"ID":"69adcc7f-84e5-4bb0-a312-9e0918f9b888","Type":"ContainerStarted","Data":"45bc9af59fe8602de35b8461c0af5c35a180c7607587ecdd3b0994f3b4ad4ca0"} Nov 03 13:30:18.873574 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:18.873402 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6958bdd4ff-zdwmx" event={"ID":"69adcc7f-84e5-4bb0-a312-9e0918f9b888","Type":"ContainerStarted","Data":"d33faacf69fbfc73593046335561345ef19d5a93393540428fba1d3560288cf3"} Nov 03 13:30:18.873574 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:18.873446 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-6958bdd4ff-zdwmx" Nov 03 13:30:18.873574 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:18.873530 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-55dcdb647d-8hjmz" Nov 03 13:30:18.904032 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:18.903927 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-6958bdd4ff-zdwmx" podStartSLOduration=1.903906318 podStartE2EDuration="1.903906318s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:30:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:30:18.899676069 +0000 UTC m=+1298.907867346" watchObservedRunningTime="2026-11-03 13:30:18.903906318 +0000 UTC m=+1298.912097595" Nov 03 13:30:18.954176 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:18.954135 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-55dcdb647d-8hjmz"] Nov 03 13:30:18.961277 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:18.961243 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-55dcdb647d-8hjmz"] Nov 03 13:30:19.033220 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:19.033180 5024 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/48073b49-8f68-47ec-bdcb-a82aace8f8c8-provider-credentials\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:30:20.112096 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:20.112056 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="48073b49-8f68-47ec-bdcb-a82aace8f8c8" path="/var/lib/kubelet/pods/48073b49-8f68-47ec-bdcb-a82aace8f8c8/volumes" Nov 03 13:30:21.104168 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:21.104126 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-5bffndnqx" podUID="6908d762-9f04-4823-a7f2-00a4e703b601" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 13:30:23.884311 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:23.882194 5024 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-6958bdd4ff-m66rf" podUID="90ddc3f3-a4b4-4ddd-9bfe-175af740f60e" containerName="payload-processing" containerID="cri-o://852e603f206278ea5a4ce7e9eaa53dc44ee340d8a1507c2283e73a1e1e7ebbc4" gracePeriod=25 Nov 03 13:30:24.279132 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:24.279103 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6958bdd4ff-m66rf" Nov 03 13:30:24.307533 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:24.307496 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90ddc3f3-a4b4-4ddd-9bfe-175af740f60e-provider-credentials\" (UniqueName: \"kubernetes.io/projected/90ddc3f3-a4b4-4ddd-9bfe-175af740f60e-provider-credentials\") pod \"90ddc3f3-a4b4-4ddd-9bfe-175af740f60e\" (UID: \"90ddc3f3-a4b4-4ddd-9bfe-175af740f60e\") " Nov 03 13:30:24.307684 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:24.307639 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/90ddc3f3-a4b4-4ddd-9bfe-175af740f60e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90ddc3f3-a4b4-4ddd-9bfe-175af740f60e-plugins-config-volume\") pod \"90ddc3f3-a4b4-4ddd-9bfe-175af740f60e\" (UID: \"90ddc3f3-a4b4-4ddd-9bfe-175af740f60e\") " Nov 03 13:30:24.307818 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:24.307797 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/90ddc3f3-a4b4-4ddd-9bfe-175af740f60e-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/90ddc3f3-a4b4-4ddd-9bfe-175af740f60e-routing-overlay-volume\") pod \"90ddc3f3-a4b4-4ddd-9bfe-175af740f60e\" (UID: \"90ddc3f3-a4b4-4ddd-9bfe-175af740f60e\") " Nov 03 13:30:24.307954 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:24.307874 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90ddc3f3-a4b4-4ddd-9bfe-175af740f60e-kube-api-access-rdjnx\" (UniqueName: \"kubernetes.io/projected/90ddc3f3-a4b4-4ddd-9bfe-175af740f60e-kube-api-access-rdjnx\") pod \"90ddc3f3-a4b4-4ddd-9bfe-175af740f60e\" (UID: \"90ddc3f3-a4b4-4ddd-9bfe-175af740f60e\") " Nov 03 13:30:24.308134 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:24.308102 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90ddc3f3-a4b4-4ddd-9bfe-175af740f60e-routing-overlay-volume" pod "90ddc3f3-a4b4-4ddd-9bfe-175af740f60e" (UID: "90ddc3f3-a4b4-4ddd-9bfe-175af740f60e"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:30:24.308210 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:24.308111 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90ddc3f3-a4b4-4ddd-9bfe-175af740f60e-plugins-config-volume" pod "90ddc3f3-a4b4-4ddd-9bfe-175af740f60e" (UID: "90ddc3f3-a4b4-4ddd-9bfe-175af740f60e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:30:24.308541 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:24.308523 5024 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90ddc3f3-a4b4-4ddd-9bfe-175af740f60e-plugins-config-volume\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:30:24.308587 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:24.308543 5024 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/90ddc3f3-a4b4-4ddd-9bfe-175af740f60e-routing-overlay-volume\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:30:24.309862 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:24.309829 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90ddc3f3-a4b4-4ddd-9bfe-175af740f60e-provider-credentials" pod "90ddc3f3-a4b4-4ddd-9bfe-175af740f60e" (UID: "90ddc3f3-a4b4-4ddd-9bfe-175af740f60e"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:30:24.310095 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:24.310076 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90ddc3f3-a4b4-4ddd-9bfe-175af740f60e-kube-api-access-rdjnx" pod "90ddc3f3-a4b4-4ddd-9bfe-175af740f60e" (UID: "90ddc3f3-a4b4-4ddd-9bfe-175af740f60e"). InnerVolumeSpecName "kube-api-access-rdjnx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:30:24.409850 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:24.409749 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rdjnx\" (UniqueName: \"kubernetes.io/projected/90ddc3f3-a4b4-4ddd-9bfe-175af740f60e-kube-api-access-rdjnx\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:30:24.409850 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:24.409786 5024 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/90ddc3f3-a4b4-4ddd-9bfe-175af740f60e-provider-credentials\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:30:24.507619 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:24.506598 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-6958bdd4ff-zdwmx"] Nov 03 13:30:24.530380 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:24.530069 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-6958bdd4ff-zdwmx" Nov 03 13:30:24.581040 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:24.578945 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-55dcdb647d-l4wrh"] Nov 03 13:30:24.581040 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:24.580423 5024 cpu_manager.go:414] "containerMap: removing container" podUID="90ddc3f3-a4b4-4ddd-9bfe-175af740f60e" containerName="payload-processing" Nov 03 13:30:24.581040 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:24.580456 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90ddc3f3-a4b4-4ddd-9bfe-175af740f60e" containerName="payload-processing" Nov 03 13:30:24.581040 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:24.580656 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="90ddc3f3-a4b4-4ddd-9bfe-175af740f60e" containerName="payload-processing" Nov 03 13:30:24.585023 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:24.584848 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-55dcdb647d-l4wrh" Nov 03 13:30:24.588236 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:24.588204 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-55dcdb647d-l4wrh"] Nov 03 13:30:24.590825 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:24.590729 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-64a3f5248b8f\"" Nov 03 13:30:24.623963 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:24.623920 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/040b6e8c-e334-4b79-ad7c-34bd40bb7d2b-routing-overlay-volume\") pod \"payload-processing-external-model-55dcdb647d-l4wrh\" (UID: \"040b6e8c-e334-4b79-ad7c-34bd40bb7d2b\") " pod="models-as-a-service/payload-processing-external-model-55dcdb647d-l4wrh" Nov 03 13:30:24.624344 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:24.624317 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bgh9p\" (UniqueName: \"kubernetes.io/projected/040b6e8c-e334-4b79-ad7c-34bd40bb7d2b-kube-api-access-bgh9p\") pod \"payload-processing-external-model-55dcdb647d-l4wrh\" (UID: \"040b6e8c-e334-4b79-ad7c-34bd40bb7d2b\") " pod="models-as-a-service/payload-processing-external-model-55dcdb647d-l4wrh" Nov 03 13:30:24.624560 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:24.624542 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/040b6e8c-e334-4b79-ad7c-34bd40bb7d2b-provider-credentials\") pod \"payload-processing-external-model-55dcdb647d-l4wrh\" (UID: \"040b6e8c-e334-4b79-ad7c-34bd40bb7d2b\") " pod="models-as-a-service/payload-processing-external-model-55dcdb647d-l4wrh" Nov 03 13:30:24.624714 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:24.624699 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/040b6e8c-e334-4b79-ad7c-34bd40bb7d2b-plugins-config-volume\") pod \"payload-processing-external-model-55dcdb647d-l4wrh\" (UID: \"040b6e8c-e334-4b79-ad7c-34bd40bb7d2b\") " pod="models-as-a-service/payload-processing-external-model-55dcdb647d-l4wrh" Nov 03 13:30:24.726796 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:24.726691 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/040b6e8c-e334-4b79-ad7c-34bd40bb7d2b-plugins-config-volume\") pod \"payload-processing-external-model-55dcdb647d-l4wrh\" (UID: \"040b6e8c-e334-4b79-ad7c-34bd40bb7d2b\") " pod="models-as-a-service/payload-processing-external-model-55dcdb647d-l4wrh" Nov 03 13:30:24.726995 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:24.726974 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/040b6e8c-e334-4b79-ad7c-34bd40bb7d2b-routing-overlay-volume\") pod \"payload-processing-external-model-55dcdb647d-l4wrh\" (UID: \"040b6e8c-e334-4b79-ad7c-34bd40bb7d2b\") " pod="models-as-a-service/payload-processing-external-model-55dcdb647d-l4wrh" Nov 03 13:30:24.727101 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:24.727087 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgh9p\" (UniqueName: \"kubernetes.io/projected/040b6e8c-e334-4b79-ad7c-34bd40bb7d2b-kube-api-access-bgh9p\") pod \"payload-processing-external-model-55dcdb647d-l4wrh\" (UID: \"040b6e8c-e334-4b79-ad7c-34bd40bb7d2b\") " pod="models-as-a-service/payload-processing-external-model-55dcdb647d-l4wrh" Nov 03 13:30:24.727197 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:24.727183 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/040b6e8c-e334-4b79-ad7c-34bd40bb7d2b-provider-credentials\") pod \"payload-processing-external-model-55dcdb647d-l4wrh\" (UID: \"040b6e8c-e334-4b79-ad7c-34bd40bb7d2b\") " pod="models-as-a-service/payload-processing-external-model-55dcdb647d-l4wrh" Nov 03 13:30:24.727618 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:24.727591 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/040b6e8c-e334-4b79-ad7c-34bd40bb7d2b-plugins-config-volume\") pod \"payload-processing-external-model-55dcdb647d-l4wrh\" (UID: \"040b6e8c-e334-4b79-ad7c-34bd40bb7d2b\") " pod="models-as-a-service/payload-processing-external-model-55dcdb647d-l4wrh" Nov 03 13:30:24.727686 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:24.727641 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/040b6e8c-e334-4b79-ad7c-34bd40bb7d2b-routing-overlay-volume\") pod \"payload-processing-external-model-55dcdb647d-l4wrh\" (UID: \"040b6e8c-e334-4b79-ad7c-34bd40bb7d2b\") " pod="models-as-a-service/payload-processing-external-model-55dcdb647d-l4wrh" Nov 03 13:30:24.729974 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:24.729951 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/040b6e8c-e334-4b79-ad7c-34bd40bb7d2b-provider-credentials\") pod \"payload-processing-external-model-55dcdb647d-l4wrh\" (UID: \"040b6e8c-e334-4b79-ad7c-34bd40bb7d2b\") " pod="models-as-a-service/payload-processing-external-model-55dcdb647d-l4wrh" Nov 03 13:30:24.736875 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:24.736843 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bgh9p\" (UniqueName: \"kubernetes.io/projected/040b6e8c-e334-4b79-ad7c-34bd40bb7d2b-kube-api-access-bgh9p\") pod \"payload-processing-external-model-55dcdb647d-l4wrh\" (UID: \"040b6e8c-e334-4b79-ad7c-34bd40bb7d2b\") " pod="models-as-a-service/payload-processing-external-model-55dcdb647d-l4wrh" Nov 03 13:30:24.927921 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:24.927877 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-55dcdb647d-l4wrh" Nov 03 13:30:24.996859 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:24.986710 5024 generic.go:356] "Generic (PLEG): container finished" podID="90ddc3f3-a4b4-4ddd-9bfe-175af740f60e" containerID="852e603f206278ea5a4ce7e9eaa53dc44ee340d8a1507c2283e73a1e1e7ebbc4" exitCode=0 Nov 03 13:30:24.996859 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:24.986758 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6958bdd4ff-m66rf" event={"ID":"90ddc3f3-a4b4-4ddd-9bfe-175af740f60e","Type":"ContainerDied","Data":"852e603f206278ea5a4ce7e9eaa53dc44ee340d8a1507c2283e73a1e1e7ebbc4"} Nov 03 13:30:24.996859 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:24.986834 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6958bdd4ff-m66rf" event={"ID":"90ddc3f3-a4b4-4ddd-9bfe-175af740f60e","Type":"ContainerDied","Data":"7883579da2ba68d5540b28a32d07155fb1b1b02e775ca781b03b3573a7bfbc46"} Nov 03 13:30:24.996859 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:24.986856 5024 scope.go:122] "RemoveContainer" containerID="852e603f206278ea5a4ce7e9eaa53dc44ee340d8a1507c2283e73a1e1e7ebbc4" Nov 03 13:30:24.996859 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:24.987009 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6958bdd4ff-m66rf" Nov 03 13:30:25.005810 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:25.005770 5024 scope.go:122] "RemoveContainer" containerID="852e603f206278ea5a4ce7e9eaa53dc44ee340d8a1507c2283e73a1e1e7ebbc4" Nov 03 13:30:25.006463 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:30:25.006385 5024 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"852e603f206278ea5a4ce7e9eaa53dc44ee340d8a1507c2283e73a1e1e7ebbc4\": container with ID starting with 852e603f206278ea5a4ce7e9eaa53dc44ee340d8a1507c2283e73a1e1e7ebbc4 not found: ID does not exist" containerID="852e603f206278ea5a4ce7e9eaa53dc44ee340d8a1507c2283e73a1e1e7ebbc4" Nov 03 13:30:25.006463 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:25.006425 5024 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"852e603f206278ea5a4ce7e9eaa53dc44ee340d8a1507c2283e73a1e1e7ebbc4"} err="failed to get container status \"852e603f206278ea5a4ce7e9eaa53dc44ee340d8a1507c2283e73a1e1e7ebbc4\": rpc error: code = NotFound desc = could not find container \"852e603f206278ea5a4ce7e9eaa53dc44ee340d8a1507c2283e73a1e1e7ebbc4\": container with ID starting with 852e603f206278ea5a4ce7e9eaa53dc44ee340d8a1507c2283e73a1e1e7ebbc4 not found: ID does not exist" Nov 03 13:30:25.037381 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:25.037323 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-6958bdd4ff-m66rf"] Nov 03 13:30:25.045122 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:25.044132 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-6958bdd4ff-m66rf"] Nov 03 13:30:25.263732 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:25.253867 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-55dcdb647d-l4wrh"] Nov 03 13:30:25.287463 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:30:25.279634 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod040b6e8c_e334_4b79_ad7c_34bd40bb7d2b.slice/crio-fe4d4a7f08a406fab8068acd634a9d3e33a1c00ac5abcc9834d9f46907a1bc20 WatchSource:0}: Error finding container fe4d4a7f08a406fab8068acd634a9d3e33a1c00ac5abcc9834d9f46907a1bc20: Status 404 returned error can't find the container with id fe4d4a7f08a406fab8068acd634a9d3e33a1c00ac5abcc9834d9f46907a1bc20 Nov 03 13:30:26.051745 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:26.051701 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-55dcdb647d-l4wrh" event={"ID":"040b6e8c-e334-4b79-ad7c-34bd40bb7d2b","Type":"ContainerStarted","Data":"c23967006a2051bf6687518beb0dcbeab62ba4a3b73be8679b3336654d67ebd0"} Nov 03 13:30:26.052484 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:26.052450 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-55dcdb647d-l4wrh" event={"ID":"040b6e8c-e334-4b79-ad7c-34bd40bb7d2b","Type":"ContainerStarted","Data":"fe4d4a7f08a406fab8068acd634a9d3e33a1c00ac5abcc9834d9f46907a1bc20"} Nov 03 13:30:26.060029 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:26.059992 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-55dcdb647d-l4wrh" Nov 03 13:30:26.112371 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:26.112305 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-55dcdb647d-l4wrh" podStartSLOduration=2.112270297 podStartE2EDuration="2.112270297s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:30:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:30:26.110673128 +0000 UTC m=+1306.118864422" watchObservedRunningTime="2026-11-03 13:30:26.112270297 +0000 UTC m=+1306.120461575" Nov 03 13:30:26.134308 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:26.132056 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="90ddc3f3-a4b4-4ddd-9bfe-175af740f60e" path="/var/lib/kubelet/pods/90ddc3f3-a4b4-4ddd-9bfe-175af740f60e/volumes" Nov 03 13:30:29.507051 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:29.506981 5024 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-6958bdd4ff-zdwmx" podUID="69adcc7f-84e5-4bb0-a312-9e0918f9b888" containerName="payload-processing" containerID="cri-o://45bc9af59fe8602de35b8461c0af5c35a180c7607587ecdd3b0994f3b4ad4ca0" gracePeriod=25 Nov 03 13:30:29.947058 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:29.947021 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6958bdd4ff-zdwmx" Nov 03 13:30:30.088672 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:30.088639 5024 generic.go:356] "Generic (PLEG): container finished" podID="69adcc7f-84e5-4bb0-a312-9e0918f9b888" containerID="45bc9af59fe8602de35b8461c0af5c35a180c7607587ecdd3b0994f3b4ad4ca0" exitCode=0 Nov 03 13:30:30.088865 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:30.088701 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6958bdd4ff-zdwmx" Nov 03 13:30:30.088865 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:30.088739 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6958bdd4ff-zdwmx" event={"ID":"69adcc7f-84e5-4bb0-a312-9e0918f9b888","Type":"ContainerDied","Data":"45bc9af59fe8602de35b8461c0af5c35a180c7607587ecdd3b0994f3b4ad4ca0"} Nov 03 13:30:30.088865 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:30.088797 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6958bdd4ff-zdwmx" event={"ID":"69adcc7f-84e5-4bb0-a312-9e0918f9b888","Type":"ContainerDied","Data":"d33faacf69fbfc73593046335561345ef19d5a93393540428fba1d3560288cf3"} Nov 03 13:30:30.088865 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:30.088826 5024 scope.go:122] "RemoveContainer" containerID="45bc9af59fe8602de35b8461c0af5c35a180c7607587ecdd3b0994f3b4ad4ca0" Nov 03 13:30:30.104381 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:30.102063 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/69adcc7f-84e5-4bb0-a312-9e0918f9b888-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69adcc7f-84e5-4bb0-a312-9e0918f9b888-plugins-config-volume\") pod \"69adcc7f-84e5-4bb0-a312-9e0918f9b888\" (UID: \"69adcc7f-84e5-4bb0-a312-9e0918f9b888\") " Nov 03 13:30:30.104381 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:30.102135 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69adcc7f-84e5-4bb0-a312-9e0918f9b888-kube-api-access-mrj2w\" (UniqueName: \"kubernetes.io/projected/69adcc7f-84e5-4bb0-a312-9e0918f9b888-kube-api-access-mrj2w\") pod \"69adcc7f-84e5-4bb0-a312-9e0918f9b888\" (UID: \"69adcc7f-84e5-4bb0-a312-9e0918f9b888\") " Nov 03 13:30:30.104381 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:30.102183 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69adcc7f-84e5-4bb0-a312-9e0918f9b888-provider-credentials\" (UniqueName: \"kubernetes.io/projected/69adcc7f-84e5-4bb0-a312-9e0918f9b888-provider-credentials\") pod \"69adcc7f-84e5-4bb0-a312-9e0918f9b888\" (UID: \"69adcc7f-84e5-4bb0-a312-9e0918f9b888\") " Nov 03 13:30:30.104381 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:30.102542 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/69adcc7f-84e5-4bb0-a312-9e0918f9b888-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/69adcc7f-84e5-4bb0-a312-9e0918f9b888-routing-overlay-volume\") pod \"69adcc7f-84e5-4bb0-a312-9e0918f9b888\" (UID: \"69adcc7f-84e5-4bb0-a312-9e0918f9b888\") " Nov 03 13:30:30.104381 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:30.103173 5024 scope.go:122] "RemoveContainer" containerID="45bc9af59fe8602de35b8461c0af5c35a180c7607587ecdd3b0994f3b4ad4ca0" Nov 03 13:30:30.104381 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:30:30.104148 5024 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"45bc9af59fe8602de35b8461c0af5c35a180c7607587ecdd3b0994f3b4ad4ca0\": container with ID starting with 45bc9af59fe8602de35b8461c0af5c35a180c7607587ecdd3b0994f3b4ad4ca0 not found: ID does not exist" containerID="45bc9af59fe8602de35b8461c0af5c35a180c7607587ecdd3b0994f3b4ad4ca0" Nov 03 13:30:30.104381 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:30.104183 5024 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45bc9af59fe8602de35b8461c0af5c35a180c7607587ecdd3b0994f3b4ad4ca0"} err="failed to get container status \"45bc9af59fe8602de35b8461c0af5c35a180c7607587ecdd3b0994f3b4ad4ca0\": rpc error: code = NotFound desc = could not find container \"45bc9af59fe8602de35b8461c0af5c35a180c7607587ecdd3b0994f3b4ad4ca0\": container with ID starting with 45bc9af59fe8602de35b8461c0af5c35a180c7607587ecdd3b0994f3b4ad4ca0 not found: ID does not exist" Nov 03 13:30:30.105885 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:30.104491 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69adcc7f-84e5-4bb0-a312-9e0918f9b888-plugins-config-volume" pod "69adcc7f-84e5-4bb0-a312-9e0918f9b888" (UID: "69adcc7f-84e5-4bb0-a312-9e0918f9b888"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:30:30.105885 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:30.104837 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69adcc7f-84e5-4bb0-a312-9e0918f9b888-routing-overlay-volume" pod "69adcc7f-84e5-4bb0-a312-9e0918f9b888" (UID: "69adcc7f-84e5-4bb0-a312-9e0918f9b888"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:30:30.106013 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:30.105980 5024 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/69adcc7f-84e5-4bb0-a312-9e0918f9b888-routing-overlay-volume\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:30:30.106013 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:30.106016 5024 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69adcc7f-84e5-4bb0-a312-9e0918f9b888-plugins-config-volume\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:30:30.112434 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:30.112391 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69adcc7f-84e5-4bb0-a312-9e0918f9b888-provider-credentials" pod "69adcc7f-84e5-4bb0-a312-9e0918f9b888" (UID: "69adcc7f-84e5-4bb0-a312-9e0918f9b888"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:30:30.113071 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:30.113043 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69adcc7f-84e5-4bb0-a312-9e0918f9b888-kube-api-access-mrj2w" pod "69adcc7f-84e5-4bb0-a312-9e0918f9b888" (UID: "69adcc7f-84e5-4bb0-a312-9e0918f9b888"). InnerVolumeSpecName "kube-api-access-mrj2w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:30:30.228320 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:30.226330 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mrj2w\" (UniqueName: \"kubernetes.io/projected/69adcc7f-84e5-4bb0-a312-9e0918f9b888-kube-api-access-mrj2w\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:30:30.228320 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:30.226371 5024 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/69adcc7f-84e5-4bb0-a312-9e0918f9b888-provider-credentials\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:30:30.307309 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:30.300190 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-55dcdb647d-l4wrh"] Nov 03 13:30:30.341835 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:30.341757 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-55dcdb647d-l4wrh" Nov 03 13:30:30.370327 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:30.366342 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-6958bdd4ff-mjcfh"] Nov 03 13:30:30.370327 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:30.367657 5024 cpu_manager.go:414] "containerMap: removing container" podUID="69adcc7f-84e5-4bb0-a312-9e0918f9b888" containerName="payload-processing" Nov 03 13:30:30.370327 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:30.367685 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69adcc7f-84e5-4bb0-a312-9e0918f9b888" containerName="payload-processing" Nov 03 13:30:30.370327 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:30.367874 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="69adcc7f-84e5-4bb0-a312-9e0918f9b888" containerName="payload-processing" Nov 03 13:30:30.393781 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:30.393743 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-6958bdd4ff-mjcfh"] Nov 03 13:30:30.402306 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:30.394091 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6958bdd4ff-mjcfh" Nov 03 13:30:30.456455 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:30.440101 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0561186c-eec8-4b5f-aac2-7aedf3f060cd-routing-overlay-volume\") pod \"payload-processing-external-model-6958bdd4ff-mjcfh\" (UID: \"0561186c-eec8-4b5f-aac2-7aedf3f060cd\") " pod="models-as-a-service/payload-processing-external-model-6958bdd4ff-mjcfh" Nov 03 13:30:30.456455 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:30.449609 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6mckk\" (UniqueName: \"kubernetes.io/projected/0561186c-eec8-4b5f-aac2-7aedf3f060cd-kube-api-access-6mckk\") pod \"payload-processing-external-model-6958bdd4ff-mjcfh\" (UID: \"0561186c-eec8-4b5f-aac2-7aedf3f060cd\") " pod="models-as-a-service/payload-processing-external-model-6958bdd4ff-mjcfh" Nov 03 13:30:30.456455 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:30.449782 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0561186c-eec8-4b5f-aac2-7aedf3f060cd-provider-credentials\") pod \"payload-processing-external-model-6958bdd4ff-mjcfh\" (UID: \"0561186c-eec8-4b5f-aac2-7aedf3f060cd\") " pod="models-as-a-service/payload-processing-external-model-6958bdd4ff-mjcfh" Nov 03 13:30:30.456455 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:30.450076 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0561186c-eec8-4b5f-aac2-7aedf3f060cd-plugins-config-volume\") pod \"payload-processing-external-model-6958bdd4ff-mjcfh\" (UID: \"0561186c-eec8-4b5f-aac2-7aedf3f060cd\") " pod="models-as-a-service/payload-processing-external-model-6958bdd4ff-mjcfh" Nov 03 13:30:30.519531 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:30.519494 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-6958bdd4ff-zdwmx"] Nov 03 13:30:30.529769 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:30.529734 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-6958bdd4ff-zdwmx"] Nov 03 13:30:30.560919 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:30.560877 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0561186c-eec8-4b5f-aac2-7aedf3f060cd-provider-credentials\") pod \"payload-processing-external-model-6958bdd4ff-mjcfh\" (UID: \"0561186c-eec8-4b5f-aac2-7aedf3f060cd\") " pod="models-as-a-service/payload-processing-external-model-6958bdd4ff-mjcfh" Nov 03 13:30:30.561106 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:30.561055 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0561186c-eec8-4b5f-aac2-7aedf3f060cd-plugins-config-volume\") pod \"payload-processing-external-model-6958bdd4ff-mjcfh\" (UID: \"0561186c-eec8-4b5f-aac2-7aedf3f060cd\") " pod="models-as-a-service/payload-processing-external-model-6958bdd4ff-mjcfh" Nov 03 13:30:30.561200 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:30.561185 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0561186c-eec8-4b5f-aac2-7aedf3f060cd-routing-overlay-volume\") pod \"payload-processing-external-model-6958bdd4ff-mjcfh\" (UID: \"0561186c-eec8-4b5f-aac2-7aedf3f060cd\") " pod="models-as-a-service/payload-processing-external-model-6958bdd4ff-mjcfh" Nov 03 13:30:30.561239 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:30.561211 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6mckk\" (UniqueName: \"kubernetes.io/projected/0561186c-eec8-4b5f-aac2-7aedf3f060cd-kube-api-access-6mckk\") pod \"payload-processing-external-model-6958bdd4ff-mjcfh\" (UID: \"0561186c-eec8-4b5f-aac2-7aedf3f060cd\") " pod="models-as-a-service/payload-processing-external-model-6958bdd4ff-mjcfh" Nov 03 13:30:30.561885 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:30.561847 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0561186c-eec8-4b5f-aac2-7aedf3f060cd-plugins-config-volume\") pod \"payload-processing-external-model-6958bdd4ff-mjcfh\" (UID: \"0561186c-eec8-4b5f-aac2-7aedf3f060cd\") " pod="models-as-a-service/payload-processing-external-model-6958bdd4ff-mjcfh" Nov 03 13:30:30.562026 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:30.561908 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0561186c-eec8-4b5f-aac2-7aedf3f060cd-routing-overlay-volume\") pod \"payload-processing-external-model-6958bdd4ff-mjcfh\" (UID: \"0561186c-eec8-4b5f-aac2-7aedf3f060cd\") " pod="models-as-a-service/payload-processing-external-model-6958bdd4ff-mjcfh" Nov 03 13:30:30.563627 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:30.563605 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0561186c-eec8-4b5f-aac2-7aedf3f060cd-provider-credentials\") pod \"payload-processing-external-model-6958bdd4ff-mjcfh\" (UID: \"0561186c-eec8-4b5f-aac2-7aedf3f060cd\") " pod="models-as-a-service/payload-processing-external-model-6958bdd4ff-mjcfh" Nov 03 13:30:30.571830 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:30.571799 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6mckk\" (UniqueName: \"kubernetes.io/projected/0561186c-eec8-4b5f-aac2-7aedf3f060cd-kube-api-access-6mckk\") pod \"payload-processing-external-model-6958bdd4ff-mjcfh\" (UID: \"0561186c-eec8-4b5f-aac2-7aedf3f060cd\") " pod="models-as-a-service/payload-processing-external-model-6958bdd4ff-mjcfh" Nov 03 13:30:30.809360 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:30.808623 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6958bdd4ff-mjcfh" Nov 03 13:30:31.106250 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:31.105930 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-5bffndnqx" podUID="6908d762-9f04-4823-a7f2-00a4e703b601" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 13:30:31.243402 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:31.243374 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-6958bdd4ff-mjcfh"] Nov 03 13:30:31.243756 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:30:31.243733 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0561186c_eec8_4b5f_aac2_7aedf3f060cd.slice/crio-9911afbe427087b1b806421a27db075c3d33751c81679c4366b6f14d473c2ae4 WatchSource:0}: Error finding container 9911afbe427087b1b806421a27db075c3d33751c81679c4366b6f14d473c2ae4: Status 404 returned error can't find the container with id 9911afbe427087b1b806421a27db075c3d33751c81679c4366b6f14d473c2ae4 Nov 03 13:30:32.234008 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:32.233963 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="69adcc7f-84e5-4bb0-a312-9e0918f9b888" path="/var/lib/kubelet/pods/69adcc7f-84e5-4bb0-a312-9e0918f9b888/volumes" Nov 03 13:30:32.278775 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:32.278737 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6958bdd4ff-mjcfh" event={"ID":"0561186c-eec8-4b5f-aac2-7aedf3f060cd","Type":"ContainerStarted","Data":"431b2852c65533851b4988d4bb5e16f30814a6104809be4f26940e5317443ad0"} Nov 03 13:30:32.279062 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:32.279025 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-6958bdd4ff-mjcfh" Nov 03 13:30:32.279237 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:32.279216 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6958bdd4ff-mjcfh" event={"ID":"0561186c-eec8-4b5f-aac2-7aedf3f060cd","Type":"ContainerStarted","Data":"9911afbe427087b1b806421a27db075c3d33751c81679c4366b6f14d473c2ae4"} Nov 03 13:30:32.909362 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:32.902756 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-6958bdd4ff-mjcfh" podStartSLOduration=2.902736581 podStartE2EDuration="2.902736581s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:30:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:30:32.312450116 +0000 UTC m=+1312.320641395" watchObservedRunningTime="2026-11-03 13:30:32.902736581 +0000 UTC m=+1312.910927859" Nov 03 13:30:32.921525 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:32.921482 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-6958bdd4ff-mjcfh"] Nov 03 13:30:33.014349 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:33.007306 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-5766f45f4c-4c24z"] Nov 03 13:30:33.057373 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:33.031501 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5766f45f4c-4c24z" Nov 03 13:30:33.057373 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:33.056270 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-36a6915adee5\"" Nov 03 13:30:33.163440 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:33.163273 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kb8rt\" (UniqueName: \"kubernetes.io/projected/29507c6d-dd21-4763-849f-a94e4de511ca-kube-api-access-kb8rt\") pod \"payload-processing-external-model-5766f45f4c-4c24z\" (UID: \"29507c6d-dd21-4763-849f-a94e4de511ca\") " pod="models-as-a-service/payload-processing-external-model-5766f45f4c-4c24z" Nov 03 13:30:33.163440 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:33.163374 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/29507c6d-dd21-4763-849f-a94e4de511ca-provider-credentials\") pod \"payload-processing-external-model-5766f45f4c-4c24z\" (UID: \"29507c6d-dd21-4763-849f-a94e4de511ca\") " pod="models-as-a-service/payload-processing-external-model-5766f45f4c-4c24z" Nov 03 13:30:33.163701 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:33.163536 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29507c6d-dd21-4763-849f-a94e4de511ca-plugins-config-volume\") pod \"payload-processing-external-model-5766f45f4c-4c24z\" (UID: \"29507c6d-dd21-4763-849f-a94e4de511ca\") " pod="models-as-a-service/payload-processing-external-model-5766f45f4c-4c24z" Nov 03 13:30:33.163701 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:33.163605 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/29507c6d-dd21-4763-849f-a94e4de511ca-routing-overlay-volume\") pod \"payload-processing-external-model-5766f45f4c-4c24z\" (UID: \"29507c6d-dd21-4763-849f-a94e4de511ca\") " pod="models-as-a-service/payload-processing-external-model-5766f45f4c-4c24z" Nov 03 13:30:33.163867 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:33.163848 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-rddcl"] Nov 03 13:30:33.164100 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:33.164075 5024 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-rddcl" podUID="77fc5206-8873-4bcc-89b1-fb637422962b" containerName="istio-proxy" containerID="cri-o://aebdfbf8569969e34481bf8e94bc420143339f71a7e38b7baa320243eb1d2097" gracePeriod=30 Nov 03 13:30:33.177724 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:33.171497 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-kjbs5"] Nov 03 13:30:33.177724 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:33.171683 5024 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-kjbs5" podUID="4eeaf0ad-8a27-44c2-8641-88d4d64e7969" containerName="istio-proxy" containerID="cri-o://fbdcddf96d6c74e3a56a93df3868b856b4423d9bf7ad5fa43d10d6b5cd79be48" gracePeriod=30 Nov 03 13:30:33.177724 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:33.177397 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5766f45f4c-4c24z"] Nov 03 13:30:33.265227 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:33.265184 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kb8rt\" (UniqueName: \"kubernetes.io/projected/29507c6d-dd21-4763-849f-a94e4de511ca-kube-api-access-kb8rt\") pod \"payload-processing-external-model-5766f45f4c-4c24z\" (UID: \"29507c6d-dd21-4763-849f-a94e4de511ca\") " pod="models-as-a-service/payload-processing-external-model-5766f45f4c-4c24z" Nov 03 13:30:33.265811 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:33.265303 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/29507c6d-dd21-4763-849f-a94e4de511ca-provider-credentials\") pod \"payload-processing-external-model-5766f45f4c-4c24z\" (UID: \"29507c6d-dd21-4763-849f-a94e4de511ca\") " pod="models-as-a-service/payload-processing-external-model-5766f45f4c-4c24z" Nov 03 13:30:33.265811 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:30:33.265471 5024 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-36a6915adee5: secret "agc-cred-36a6915adee5" not found Nov 03 13:30:33.265811 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:30:33.265490 5024 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-5766f45f4c-4c24z: secret "agc-cred-36a6915adee5" not found Nov 03 13:30:33.265811 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:30:33.265547 5024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/29507c6d-dd21-4763-849f-a94e4de511ca-provider-credentials podName:29507c6d-dd21-4763-849f-a94e4de511ca nodeName:}" failed. No retries permitted until 2026-11-03 13:30:33.765527887 +0000 UTC m=+1313.773719157 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/29507c6d-dd21-4763-849f-a94e4de511ca-provider-credentials") pod "payload-processing-external-model-5766f45f4c-4c24z" (UID: "29507c6d-dd21-4763-849f-a94e4de511ca") : secret "agc-cred-36a6915adee5" not found Nov 03 13:30:33.266188 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:33.265470 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29507c6d-dd21-4763-849f-a94e4de511ca-plugins-config-volume\") pod \"payload-processing-external-model-5766f45f4c-4c24z\" (UID: \"29507c6d-dd21-4763-849f-a94e4de511ca\") " pod="models-as-a-service/payload-processing-external-model-5766f45f4c-4c24z" Nov 03 13:30:33.266358 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:33.266314 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/29507c6d-dd21-4763-849f-a94e4de511ca-routing-overlay-volume\") pod \"payload-processing-external-model-5766f45f4c-4c24z\" (UID: \"29507c6d-dd21-4763-849f-a94e4de511ca\") " pod="models-as-a-service/payload-processing-external-model-5766f45f4c-4c24z" Nov 03 13:30:33.266460 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:33.266329 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29507c6d-dd21-4763-849f-a94e4de511ca-plugins-config-volume\") pod \"payload-processing-external-model-5766f45f4c-4c24z\" (UID: \"29507c6d-dd21-4763-849f-a94e4de511ca\") " pod="models-as-a-service/payload-processing-external-model-5766f45f4c-4c24z" Nov 03 13:30:33.266787 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:33.266752 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/29507c6d-dd21-4763-849f-a94e4de511ca-routing-overlay-volume\") pod \"payload-processing-external-model-5766f45f4c-4c24z\" (UID: \"29507c6d-dd21-4763-849f-a94e4de511ca\") " pod="models-as-a-service/payload-processing-external-model-5766f45f4c-4c24z" Nov 03 13:30:33.301846 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:33.301808 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kb8rt\" (UniqueName: \"kubernetes.io/projected/29507c6d-dd21-4763-849f-a94e4de511ca-kube-api-access-kb8rt\") pod \"payload-processing-external-model-5766f45f4c-4c24z\" (UID: \"29507c6d-dd21-4763-849f-a94e4de511ca\") " pod="models-as-a-service/payload-processing-external-model-5766f45f4c-4c24z" Nov 03 13:30:33.780896 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:33.780855 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5766f45f4c-4c24z"] Nov 03 13:30:33.781436 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:30:33.781411 5024 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[provider-credentials], unattached volumes=[], failed to process volumes=[]: context canceled" pod="models-as-a-service/payload-processing-external-model-5766f45f4c-4c24z" podUID="29507c6d-dd21-4763-849f-a94e4de511ca" Nov 03 13:30:33.785385 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:33.785322 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/29507c6d-dd21-4763-849f-a94e4de511ca-provider-credentials\") pod \"payload-processing-external-model-5766f45f4c-4c24z\" (UID: \"29507c6d-dd21-4763-849f-a94e4de511ca\") " pod="models-as-a-service/payload-processing-external-model-5766f45f4c-4c24z" Nov 03 13:30:33.785533 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:30:33.785487 5024 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-36a6915adee5: secret "agc-cred-36a6915adee5" not found Nov 03 13:30:33.785533 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:30:33.785513 5024 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-5766f45f4c-4c24z: secret "agc-cred-36a6915adee5" not found Nov 03 13:30:33.785624 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:30:33.785569 5024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/29507c6d-dd21-4763-849f-a94e4de511ca-provider-credentials podName:29507c6d-dd21-4763-849f-a94e4de511ca nodeName:}" failed. No retries permitted until 2026-11-03 13:30:34.785556294 +0000 UTC m=+1314.793747549 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/29507c6d-dd21-4763-849f-a94e4de511ca-provider-credentials") pod "payload-processing-external-model-5766f45f4c-4c24z" (UID: "29507c6d-dd21-4763-849f-a94e4de511ca") : secret "agc-cred-36a6915adee5" not found Nov 03 13:30:34.302665 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:34.302626 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5766f45f4c-4c24z" Nov 03 13:30:34.313671 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:34.313637 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5766f45f4c-4c24z" Nov 03 13:30:34.427898 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:34.427865 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29507c6d-dd21-4763-849f-a94e4de511ca-kube-api-access-kb8rt\" (UniqueName: \"kubernetes.io/projected/29507c6d-dd21-4763-849f-a94e4de511ca-kube-api-access-kb8rt\") pod \"29507c6d-dd21-4763-849f-a94e4de511ca\" (UID: \"29507c6d-dd21-4763-849f-a94e4de511ca\") " Nov 03 13:30:34.428112 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:34.428093 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/29507c6d-dd21-4763-849f-a94e4de511ca-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29507c6d-dd21-4763-849f-a94e4de511ca-plugins-config-volume\") pod \"29507c6d-dd21-4763-849f-a94e4de511ca\" (UID: \"29507c6d-dd21-4763-849f-a94e4de511ca\") " Nov 03 13:30:34.428266 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:34.428248 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/29507c6d-dd21-4763-849f-a94e4de511ca-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/29507c6d-dd21-4763-849f-a94e4de511ca-routing-overlay-volume\") pod \"29507c6d-dd21-4763-849f-a94e4de511ca\" (UID: \"29507c6d-dd21-4763-849f-a94e4de511ca\") " Nov 03 13:30:34.428568 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:34.428546 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29507c6d-dd21-4763-849f-a94e4de511ca-routing-overlay-volume" pod "29507c6d-dd21-4763-849f-a94e4de511ca" (UID: "29507c6d-dd21-4763-849f-a94e4de511ca"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:30:34.428626 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:34.428597 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29507c6d-dd21-4763-849f-a94e4de511ca-plugins-config-volume" pod "29507c6d-dd21-4763-849f-a94e4de511ca" (UID: "29507c6d-dd21-4763-849f-a94e4de511ca"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:30:34.429002 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:34.428984 5024 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/29507c6d-dd21-4763-849f-a94e4de511ca-routing-overlay-volume\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:30:34.429039 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:34.429006 5024 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29507c6d-dd21-4763-849f-a94e4de511ca-plugins-config-volume\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:30:34.430212 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:34.430190 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29507c6d-dd21-4763-849f-a94e4de511ca-kube-api-access-kb8rt" pod "29507c6d-dd21-4763-849f-a94e4de511ca" (UID: "29507c6d-dd21-4763-849f-a94e4de511ca"). InnerVolumeSpecName "kube-api-access-kb8rt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:30:34.530986 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:34.530943 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kb8rt\" (UniqueName: \"kubernetes.io/projected/29507c6d-dd21-4763-849f-a94e4de511ca-kube-api-access-kb8rt\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:30:34.836982 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:34.836936 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/29507c6d-dd21-4763-849f-a94e4de511ca-provider-credentials\") pod \"payload-processing-external-model-5766f45f4c-4c24z\" (UID: \"29507c6d-dd21-4763-849f-a94e4de511ca\") " pod="models-as-a-service/payload-processing-external-model-5766f45f4c-4c24z" Nov 03 13:30:34.837174 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:30:34.837059 5024 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-36a6915adee5: secret "agc-cred-36a6915adee5" not found Nov 03 13:30:34.837174 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:30:34.837082 5024 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-5766f45f4c-4c24z: secret "agc-cred-36a6915adee5" not found Nov 03 13:30:34.837174 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:30:34.837145 5024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/29507c6d-dd21-4763-849f-a94e4de511ca-provider-credentials podName:29507c6d-dd21-4763-849f-a94e4de511ca nodeName:}" failed. No retries permitted until 2026-11-03 13:30:36.837130715 +0000 UTC m=+1316.845321970 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/29507c6d-dd21-4763-849f-a94e4de511ca-provider-credentials") pod "payload-processing-external-model-5766f45f4c-4c24z" (UID: "29507c6d-dd21-4763-849f-a94e4de511ca") : secret "agc-cred-36a6915adee5" not found Nov 03 13:30:35.312330 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:35.312276 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5766f45f4c-4c24z" Nov 03 13:30:35.325603 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:35.325563 5024 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-55dcdb647d-l4wrh" podUID="040b6e8c-e334-4b79-ad7c-34bd40bb7d2b" containerName="payload-processing" containerID="cri-o://c23967006a2051bf6687518beb0dcbeab62ba4a3b73be8679b3336654d67ebd0" gracePeriod=25 Nov 03 13:30:35.365031 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:35.364991 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5766f45f4c-4c24z"] Nov 03 13:30:35.371477 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:35.371422 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-5766f45f4c-4c24z"] Nov 03 13:30:35.452823 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:35.452778 5024 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/29507c6d-dd21-4763-849f-a94e4de511ca-provider-credentials\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:30:35.676175 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:35.676136 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-55dcdb647d-l4wrh" Nov 03 13:30:35.772307 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:35.772023 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/040b6e8c-e334-4b79-ad7c-34bd40bb7d2b-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/040b6e8c-e334-4b79-ad7c-34bd40bb7d2b-routing-overlay-volume\") pod \"040b6e8c-e334-4b79-ad7c-34bd40bb7d2b\" (UID: \"040b6e8c-e334-4b79-ad7c-34bd40bb7d2b\") " Nov 03 13:30:35.772307 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:35.772115 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/040b6e8c-e334-4b79-ad7c-34bd40bb7d2b-kube-api-access-bgh9p\" (UniqueName: \"kubernetes.io/projected/040b6e8c-e334-4b79-ad7c-34bd40bb7d2b-kube-api-access-bgh9p\") pod \"040b6e8c-e334-4b79-ad7c-34bd40bb7d2b\" (UID: \"040b6e8c-e334-4b79-ad7c-34bd40bb7d2b\") " Nov 03 13:30:35.772307 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:35.772156 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/040b6e8c-e334-4b79-ad7c-34bd40bb7d2b-provider-credentials\" (UniqueName: \"kubernetes.io/projected/040b6e8c-e334-4b79-ad7c-34bd40bb7d2b-provider-credentials\") pod \"040b6e8c-e334-4b79-ad7c-34bd40bb7d2b\" (UID: \"040b6e8c-e334-4b79-ad7c-34bd40bb7d2b\") " Nov 03 13:30:35.772630 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:35.772331 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/040b6e8c-e334-4b79-ad7c-34bd40bb7d2b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/040b6e8c-e334-4b79-ad7c-34bd40bb7d2b-plugins-config-volume\") pod \"040b6e8c-e334-4b79-ad7c-34bd40bb7d2b\" (UID: \"040b6e8c-e334-4b79-ad7c-34bd40bb7d2b\") " Nov 03 13:30:35.781430 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:35.773912 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/040b6e8c-e334-4b79-ad7c-34bd40bb7d2b-plugins-config-volume" pod "040b6e8c-e334-4b79-ad7c-34bd40bb7d2b" (UID: "040b6e8c-e334-4b79-ad7c-34bd40bb7d2b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:30:35.781430 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:35.776474 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/040b6e8c-e334-4b79-ad7c-34bd40bb7d2b-routing-overlay-volume" pod "040b6e8c-e334-4b79-ad7c-34bd40bb7d2b" (UID: "040b6e8c-e334-4b79-ad7c-34bd40bb7d2b"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:30:35.804520 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:35.804455 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/040b6e8c-e334-4b79-ad7c-34bd40bb7d2b-provider-credentials" pod "040b6e8c-e334-4b79-ad7c-34bd40bb7d2b" (UID: "040b6e8c-e334-4b79-ad7c-34bd40bb7d2b"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:30:35.804859 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:35.804825 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/040b6e8c-e334-4b79-ad7c-34bd40bb7d2b-kube-api-access-bgh9p" pod "040b6e8c-e334-4b79-ad7c-34bd40bb7d2b" (UID: "040b6e8c-e334-4b79-ad7c-34bd40bb7d2b"). InnerVolumeSpecName "kube-api-access-bgh9p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:30:35.883036 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:35.882877 5024 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/040b6e8c-e334-4b79-ad7c-34bd40bb7d2b-routing-overlay-volume\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:30:35.883036 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:35.882920 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bgh9p\" (UniqueName: \"kubernetes.io/projected/040b6e8c-e334-4b79-ad7c-34bd40bb7d2b-kube-api-access-bgh9p\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:30:35.883036 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:35.882938 5024 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/040b6e8c-e334-4b79-ad7c-34bd40bb7d2b-provider-credentials\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:30:35.883036 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:35.882953 5024 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/040b6e8c-e334-4b79-ad7c-34bd40bb7d2b-plugins-config-volume\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:30:36.361314 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:36.358720 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="29507c6d-dd21-4763-849f-a94e4de511ca" path="/var/lib/kubelet/pods/29507c6d-dd21-4763-849f-a94e4de511ca/volumes" Nov 03 13:30:36.438641 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:36.438425 5024 generic.go:356] "Generic (PLEG): container finished" podID="040b6e8c-e334-4b79-ad7c-34bd40bb7d2b" containerID="c23967006a2051bf6687518beb0dcbeab62ba4a3b73be8679b3336654d67ebd0" exitCode=0 Nov 03 13:30:36.438641 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:36.438539 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-55dcdb647d-l4wrh" event={"ID":"040b6e8c-e334-4b79-ad7c-34bd40bb7d2b","Type":"ContainerDied","Data":"c23967006a2051bf6687518beb0dcbeab62ba4a3b73be8679b3336654d67ebd0"} Nov 03 13:30:36.438641 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:36.438591 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-55dcdb647d-l4wrh" event={"ID":"040b6e8c-e334-4b79-ad7c-34bd40bb7d2b","Type":"ContainerDied","Data":"fe4d4a7f08a406fab8068acd634a9d3e33a1c00ac5abcc9834d9f46907a1bc20"} Nov 03 13:30:36.438641 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:36.438622 5024 scope.go:122] "RemoveContainer" containerID="c23967006a2051bf6687518beb0dcbeab62ba4a3b73be8679b3336654d67ebd0" Nov 03 13:30:36.438974 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:36.438814 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-55dcdb647d-l4wrh" Nov 03 13:30:36.542323 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:36.539459 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-55dcdb647d-l4wrh"] Nov 03 13:30:36.552580 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:36.552479 5024 scope.go:122] "RemoveContainer" containerID="c23967006a2051bf6687518beb0dcbeab62ba4a3b73be8679b3336654d67ebd0" Nov 03 13:30:36.553145 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:30:36.553045 5024 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c23967006a2051bf6687518beb0dcbeab62ba4a3b73be8679b3336654d67ebd0\": container with ID starting with c23967006a2051bf6687518beb0dcbeab62ba4a3b73be8679b3336654d67ebd0 not found: ID does not exist" containerID="c23967006a2051bf6687518beb0dcbeab62ba4a3b73be8679b3336654d67ebd0" Nov 03 13:30:36.553145 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:36.553085 5024 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c23967006a2051bf6687518beb0dcbeab62ba4a3b73be8679b3336654d67ebd0"} err="failed to get container status \"c23967006a2051bf6687518beb0dcbeab62ba4a3b73be8679b3336654d67ebd0\": rpc error: code = NotFound desc = could not find container \"c23967006a2051bf6687518beb0dcbeab62ba4a3b73be8679b3336654d67ebd0\": container with ID starting with c23967006a2051bf6687518beb0dcbeab62ba4a3b73be8679b3336654d67ebd0 not found: ID does not exist" Nov 03 13:30:36.557357 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:36.554503 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-55dcdb647d-l4wrh"] Nov 03 13:30:37.517308 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:37.516906 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-08111b-openshift-default-5b6dd8787c-v2hw4"] Nov 03 13:30:37.562309 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:37.518943 5024 cpu_manager.go:414] "containerMap: removing container" podUID="040b6e8c-e334-4b79-ad7c-34bd40bb7d2b" containerName="payload-processing" Nov 03 13:30:37.562309 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:37.518972 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="040b6e8c-e334-4b79-ad7c-34bd40bb7d2b" containerName="payload-processing" Nov 03 13:30:37.562309 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:37.519382 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="040b6e8c-e334-4b79-ad7c-34bd40bb7d2b" containerName="payload-processing" Nov 03 13:30:37.849387 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:37.849302 5024 patch_prober.go:29] interesting pod/e2e-worker-w4-c381a7-openshift-default-78bcd4695-rddcl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 13:30:38.004879 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:38.004833 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-rddcl" podUID="77fc5206-8873-4bcc-89b1-fb637422962b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 13:30:38.005211 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:37.850655 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-08111b-openshift-default-5b6dd8787c-v2hw4"] Nov 03 13:30:38.005434 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:37.980146 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca014f3a-ff92-4bb9-9a7a-8c8d3dabe486-istio-envoy\") pod \"e2e-worker-w1-08111b-openshift-default-5b6dd8787c-v2hw4\" (UID: \"ca014f3a-ff92-4bb9-9a7a-8c8d3dabe486\") " pod="openshift-ingress/e2e-worker-w1-08111b-openshift-default-5b6dd8787c-v2hw4" Nov 03 13:30:38.005819 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:38.005801 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca014f3a-ff92-4bb9-9a7a-8c8d3dabe486-istio-token\") pod \"e2e-worker-w1-08111b-openshift-default-5b6dd8787c-v2hw4\" (UID: \"ca014f3a-ff92-4bb9-9a7a-8c8d3dabe486\") " pod="openshift-ingress/e2e-worker-w1-08111b-openshift-default-5b6dd8787c-v2hw4" Nov 03 13:30:38.013651 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:38.013477 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca014f3a-ff92-4bb9-9a7a-8c8d3dabe486-workload-certs\") pod \"e2e-worker-w1-08111b-openshift-default-5b6dd8787c-v2hw4\" (UID: \"ca014f3a-ff92-4bb9-9a7a-8c8d3dabe486\") " pod="openshift-ingress/e2e-worker-w1-08111b-openshift-default-5b6dd8787c-v2hw4" Nov 03 13:30:38.023008 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:38.022964 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca014f3a-ff92-4bb9-9a7a-8c8d3dabe486-istiod-ca-cert\") pod \"e2e-worker-w1-08111b-openshift-default-5b6dd8787c-v2hw4\" (UID: \"ca014f3a-ff92-4bb9-9a7a-8c8d3dabe486\") " pod="openshift-ingress/e2e-worker-w1-08111b-openshift-default-5b6dd8787c-v2hw4" Nov 03 13:30:38.043376 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:37.849903 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-08111b-openshift-default-5b6dd8787c-v2hw4" Nov 03 13:30:38.045813 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:38.033357 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca014f3a-ff92-4bb9-9a7a-8c8d3dabe486-credential-socket\") pod \"e2e-worker-w1-08111b-openshift-default-5b6dd8787c-v2hw4\" (UID: \"ca014f3a-ff92-4bb9-9a7a-8c8d3dabe486\") " pod="openshift-ingress/e2e-worker-w1-08111b-openshift-default-5b6dd8787c-v2hw4" Nov 03 13:30:38.055738 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:38.055700 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8kdt\" (UniqueName: \"kubernetes.io/projected/ca014f3a-ff92-4bb9-9a7a-8c8d3dabe486-kube-api-access-q8kdt\") pod \"e2e-worker-w1-08111b-openshift-default-5b6dd8787c-v2hw4\" (UID: \"ca014f3a-ff92-4bb9-9a7a-8c8d3dabe486\") " pod="openshift-ingress/e2e-worker-w1-08111b-openshift-default-5b6dd8787c-v2hw4" Nov 03 13:30:38.056182 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:38.056153 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca014f3a-ff92-4bb9-9a7a-8c8d3dabe486-istio-podinfo\") pod \"e2e-worker-w1-08111b-openshift-default-5b6dd8787c-v2hw4\" (UID: \"ca014f3a-ff92-4bb9-9a7a-8c8d3dabe486\") " pod="openshift-ingress/e2e-worker-w1-08111b-openshift-default-5b6dd8787c-v2hw4" Nov 03 13:30:38.057123 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:38.056930 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca014f3a-ff92-4bb9-9a7a-8c8d3dabe486-workload-socket\") pod \"e2e-worker-w1-08111b-openshift-default-5b6dd8787c-v2hw4\" (UID: \"ca014f3a-ff92-4bb9-9a7a-8c8d3dabe486\") " pod="openshift-ingress/e2e-worker-w1-08111b-openshift-default-5b6dd8787c-v2hw4" Nov 03 13:30:38.089133 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:38.089095 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca014f3a-ff92-4bb9-9a7a-8c8d3dabe486-istio-data\") pod \"e2e-worker-w1-08111b-openshift-default-5b6dd8787c-v2hw4\" (UID: \"ca014f3a-ff92-4bb9-9a7a-8c8d3dabe486\") " pod="openshift-ingress/e2e-worker-w1-08111b-openshift-default-5b6dd8787c-v2hw4" Nov 03 13:30:38.090033 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:38.090004 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-08111b-openshift-default-dockercfg-j69cw\"" Nov 03 13:30:38.142780 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:38.142741 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="040b6e8c-e334-4b79-ad7c-34bd40bb7d2b" path="/var/lib/kubelet/pods/040b6e8c-e334-4b79-ad7c-34bd40bb7d2b/volumes" Nov 03 13:30:38.193407 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:38.193353 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca014f3a-ff92-4bb9-9a7a-8c8d3dabe486-istio-envoy\") pod \"e2e-worker-w1-08111b-openshift-default-5b6dd8787c-v2hw4\" (UID: \"ca014f3a-ff92-4bb9-9a7a-8c8d3dabe486\") " pod="openshift-ingress/e2e-worker-w1-08111b-openshift-default-5b6dd8787c-v2hw4" Nov 03 13:30:38.193793 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:38.193772 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca014f3a-ff92-4bb9-9a7a-8c8d3dabe486-istio-token\") pod \"e2e-worker-w1-08111b-openshift-default-5b6dd8787c-v2hw4\" (UID: \"ca014f3a-ff92-4bb9-9a7a-8c8d3dabe486\") " pod="openshift-ingress/e2e-worker-w1-08111b-openshift-default-5b6dd8787c-v2hw4" Nov 03 13:30:38.193980 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:38.193948 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca014f3a-ff92-4bb9-9a7a-8c8d3dabe486-workload-certs\") pod \"e2e-worker-w1-08111b-openshift-default-5b6dd8787c-v2hw4\" (UID: \"ca014f3a-ff92-4bb9-9a7a-8c8d3dabe486\") " pod="openshift-ingress/e2e-worker-w1-08111b-openshift-default-5b6dd8787c-v2hw4" Nov 03 13:30:38.194150 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:38.194123 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca014f3a-ff92-4bb9-9a7a-8c8d3dabe486-istiod-ca-cert\") pod \"e2e-worker-w1-08111b-openshift-default-5b6dd8787c-v2hw4\" (UID: \"ca014f3a-ff92-4bb9-9a7a-8c8d3dabe486\") " pod="openshift-ingress/e2e-worker-w1-08111b-openshift-default-5b6dd8787c-v2hw4" Nov 03 13:30:38.194361 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:38.194344 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca014f3a-ff92-4bb9-9a7a-8c8d3dabe486-credential-socket\") pod \"e2e-worker-w1-08111b-openshift-default-5b6dd8787c-v2hw4\" (UID: \"ca014f3a-ff92-4bb9-9a7a-8c8d3dabe486\") " pod="openshift-ingress/e2e-worker-w1-08111b-openshift-default-5b6dd8787c-v2hw4" Nov 03 13:30:38.194629 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:38.194606 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8kdt\" (UniqueName: \"kubernetes.io/projected/ca014f3a-ff92-4bb9-9a7a-8c8d3dabe486-kube-api-access-q8kdt\") pod \"e2e-worker-w1-08111b-openshift-default-5b6dd8787c-v2hw4\" (UID: \"ca014f3a-ff92-4bb9-9a7a-8c8d3dabe486\") " pod="openshift-ingress/e2e-worker-w1-08111b-openshift-default-5b6dd8787c-v2hw4" Nov 03 13:30:38.213398 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:38.198561 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca014f3a-ff92-4bb9-9a7a-8c8d3dabe486-istio-podinfo\") pod \"e2e-worker-w1-08111b-openshift-default-5b6dd8787c-v2hw4\" (UID: \"ca014f3a-ff92-4bb9-9a7a-8c8d3dabe486\") " pod="openshift-ingress/e2e-worker-w1-08111b-openshift-default-5b6dd8787c-v2hw4" Nov 03 13:30:38.215317 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:38.215260 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca014f3a-ff92-4bb9-9a7a-8c8d3dabe486-workload-socket\") pod \"e2e-worker-w1-08111b-openshift-default-5b6dd8787c-v2hw4\" (UID: \"ca014f3a-ff92-4bb9-9a7a-8c8d3dabe486\") " pod="openshift-ingress/e2e-worker-w1-08111b-openshift-default-5b6dd8787c-v2hw4" Nov 03 13:30:38.215612 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:38.215588 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca014f3a-ff92-4bb9-9a7a-8c8d3dabe486-istio-data\") pod \"e2e-worker-w1-08111b-openshift-default-5b6dd8787c-v2hw4\" (UID: \"ca014f3a-ff92-4bb9-9a7a-8c8d3dabe486\") " pod="openshift-ingress/e2e-worker-w1-08111b-openshift-default-5b6dd8787c-v2hw4" Nov 03 13:30:38.221016 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:38.216259 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca014f3a-ff92-4bb9-9a7a-8c8d3dabe486-istiod-ca-cert\") pod \"e2e-worker-w1-08111b-openshift-default-5b6dd8787c-v2hw4\" (UID: \"ca014f3a-ff92-4bb9-9a7a-8c8d3dabe486\") " pod="openshift-ingress/e2e-worker-w1-08111b-openshift-default-5b6dd8787c-v2hw4" Nov 03 13:30:38.346272 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:38.216534 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca014f3a-ff92-4bb9-9a7a-8c8d3dabe486-istio-data\") pod \"e2e-worker-w1-08111b-openshift-default-5b6dd8787c-v2hw4\" (UID: \"ca014f3a-ff92-4bb9-9a7a-8c8d3dabe486\") " pod="openshift-ingress/e2e-worker-w1-08111b-openshift-default-5b6dd8787c-v2hw4" Nov 03 13:30:38.346272 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:38.216802 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca014f3a-ff92-4bb9-9a7a-8c8d3dabe486-workload-socket\") pod \"e2e-worker-w1-08111b-openshift-default-5b6dd8787c-v2hw4\" (UID: \"ca014f3a-ff92-4bb9-9a7a-8c8d3dabe486\") " pod="openshift-ingress/e2e-worker-w1-08111b-openshift-default-5b6dd8787c-v2hw4" Nov 03 13:30:38.346272 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:38.217736 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca014f3a-ff92-4bb9-9a7a-8c8d3dabe486-workload-certs\") pod \"e2e-worker-w1-08111b-openshift-default-5b6dd8787c-v2hw4\" (UID: \"ca014f3a-ff92-4bb9-9a7a-8c8d3dabe486\") " pod="openshift-ingress/e2e-worker-w1-08111b-openshift-default-5b6dd8787c-v2hw4" Nov 03 13:30:38.346272 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:38.217773 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca014f3a-ff92-4bb9-9a7a-8c8d3dabe486-credential-socket\") pod \"e2e-worker-w1-08111b-openshift-default-5b6dd8787c-v2hw4\" (UID: \"ca014f3a-ff92-4bb9-9a7a-8c8d3dabe486\") " pod="openshift-ingress/e2e-worker-w1-08111b-openshift-default-5b6dd8787c-v2hw4" Nov 03 13:30:38.346272 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:38.247922 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca014f3a-ff92-4bb9-9a7a-8c8d3dabe486-istio-envoy\") pod \"e2e-worker-w1-08111b-openshift-default-5b6dd8787c-v2hw4\" (UID: \"ca014f3a-ff92-4bb9-9a7a-8c8d3dabe486\") " pod="openshift-ingress/e2e-worker-w1-08111b-openshift-default-5b6dd8787c-v2hw4" Nov 03 13:30:38.439853 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:38.439810 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca014f3a-ff92-4bb9-9a7a-8c8d3dabe486-istio-podinfo\") pod \"e2e-worker-w1-08111b-openshift-default-5b6dd8787c-v2hw4\" (UID: \"ca014f3a-ff92-4bb9-9a7a-8c8d3dabe486\") " pod="openshift-ingress/e2e-worker-w1-08111b-openshift-default-5b6dd8787c-v2hw4" Nov 03 13:30:38.464056 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:38.451011 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8kdt\" (UniqueName: \"kubernetes.io/projected/ca014f3a-ff92-4bb9-9a7a-8c8d3dabe486-kube-api-access-q8kdt\") pod \"e2e-worker-w1-08111b-openshift-default-5b6dd8787c-v2hw4\" (UID: \"ca014f3a-ff92-4bb9-9a7a-8c8d3dabe486\") " pod="openshift-ingress/e2e-worker-w1-08111b-openshift-default-5b6dd8787c-v2hw4" Nov 03 13:30:38.617347 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:38.594577 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca014f3a-ff92-4bb9-9a7a-8c8d3dabe486-istio-token\") pod \"e2e-worker-w1-08111b-openshift-default-5b6dd8787c-v2hw4\" (UID: \"ca014f3a-ff92-4bb9-9a7a-8c8d3dabe486\") " pod="openshift-ingress/e2e-worker-w1-08111b-openshift-default-5b6dd8787c-v2hw4" Nov 03 13:30:38.738888 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:38.738794 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-08111b-openshift-default-5b6dd8787c-v2hw4" Nov 03 13:30:39.045511 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:39.045347 5024 generic.go:356] "Generic (PLEG): container finished" podID="4eeaf0ad-8a27-44c2-8641-88d4d64e7969" containerID="fbdcddf96d6c74e3a56a93df3868b856b4423d9bf7ad5fa43d10d6b5cd79be48" exitCode=0 Nov 03 13:30:39.045857 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:39.045401 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-kjbs5" event={"ID":"4eeaf0ad-8a27-44c2-8641-88d4d64e7969","Type":"ContainerDied","Data":"fbdcddf96d6c74e3a56a93df3868b856b4423d9bf7ad5fa43d10d6b5cd79be48"} Nov 03 13:30:39.130561 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:39.130491 5024 generic.go:356] "Generic (PLEG): container finished" podID="77fc5206-8873-4bcc-89b1-fb637422962b" containerID="aebdfbf8569969e34481bf8e94bc420143339f71a7e38b7baa320243eb1d2097" exitCode=0 Nov 03 13:30:39.131101 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:39.131056 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-rddcl" event={"ID":"77fc5206-8873-4bcc-89b1-fb637422962b","Type":"ContainerDied","Data":"aebdfbf8569969e34481bf8e94bc420143339f71a7e38b7baa320243eb1d2097"} Nov 03 13:30:39.131562 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:39.131368 5024 scope.go:122] "RemoveContainer" containerID="8c2a0b398d8cdb497635818df8d9a1db39abe3b060e5bbcd9d30c33667bdafc3" Nov 03 13:30:39.303516 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:39.303378 5024 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-6958bdd4ff-mjcfh" podUID="0561186c-eec8-4b5f-aac2-7aedf3f060cd" containerName="payload-processing" containerID="cri-o://431b2852c65533851b4988d4bb5e16f30814a6104809be4f26940e5317443ad0" gracePeriod=25 Nov 03 13:30:41.009377 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:30:41.002914 5024 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d7mwz" containerName="extractor" Nov 03 13:30:41.108326 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:41.090264 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-08111b-openshift-default-5b6dd8787c-v2hw4"] Nov 03 13:30:41.108326 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:41.102511 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-rddcl" Nov 03 13:30:41.139346 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:41.134997 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-5bffndnqx" podUID="6908d762-9f04-4823-a7f2-00a4e703b601" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 13:30:41.283870 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:41.276788 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:30:41.283870 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:41.276924 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:30:41.283870 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:41.276972 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:30:41.283870 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:41.278434 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77fc5206-8873-4bcc-89b1-fb637422962b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77fc5206-8873-4bcc-89b1-fb637422962b-workload-certs\") pod \"77fc5206-8873-4bcc-89b1-fb637422962b\" (UID: \"77fc5206-8873-4bcc-89b1-fb637422962b\") " Nov 03 13:30:41.283870 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:41.278533 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/77fc5206-8873-4bcc-89b1-fb637422962b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77fc5206-8873-4bcc-89b1-fb637422962b-istiod-ca-cert\") pod \"77fc5206-8873-4bcc-89b1-fb637422962b\" (UID: \"77fc5206-8873-4bcc-89b1-fb637422962b\") " Nov 03 13:30:41.283870 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:41.278687 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77fc5206-8873-4bcc-89b1-fb637422962b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77fc5206-8873-4bcc-89b1-fb637422962b-istio-envoy\") pod \"77fc5206-8873-4bcc-89b1-fb637422962b\" (UID: \"77fc5206-8873-4bcc-89b1-fb637422962b\") " Nov 03 13:30:41.283870 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:41.278748 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77fc5206-8873-4bcc-89b1-fb637422962b-istio-token\" (UniqueName: \"kubernetes.io/projected/77fc5206-8873-4bcc-89b1-fb637422962b-istio-token\") pod \"77fc5206-8873-4bcc-89b1-fb637422962b\" (UID: \"77fc5206-8873-4bcc-89b1-fb637422962b\") " Nov 03 13:30:41.283870 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:41.278798 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77fc5206-8873-4bcc-89b1-fb637422962b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77fc5206-8873-4bcc-89b1-fb637422962b-istio-data\") pod \"77fc5206-8873-4bcc-89b1-fb637422962b\" (UID: \"77fc5206-8873-4bcc-89b1-fb637422962b\") " Nov 03 13:30:41.283870 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:41.278851 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77fc5206-8873-4bcc-89b1-fb637422962b-kube-api-access-j6pt7\" (UniqueName: \"kubernetes.io/projected/77fc5206-8873-4bcc-89b1-fb637422962b-kube-api-access-j6pt7\") pod \"77fc5206-8873-4bcc-89b1-fb637422962b\" (UID: \"77fc5206-8873-4bcc-89b1-fb637422962b\") " Nov 03 13:30:41.283870 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:41.278885 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77fc5206-8873-4bcc-89b1-fb637422962b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77fc5206-8873-4bcc-89b1-fb637422962b-workload-socket\") pod \"77fc5206-8873-4bcc-89b1-fb637422962b\" (UID: \"77fc5206-8873-4bcc-89b1-fb637422962b\") " Nov 03 13:30:41.283870 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:41.278943 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/77fc5206-8873-4bcc-89b1-fb637422962b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77fc5206-8873-4bcc-89b1-fb637422962b-istio-podinfo\") pod \"77fc5206-8873-4bcc-89b1-fb637422962b\" (UID: \"77fc5206-8873-4bcc-89b1-fb637422962b\") " Nov 03 13:30:41.283870 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:41.279070 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77fc5206-8873-4bcc-89b1-fb637422962b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77fc5206-8873-4bcc-89b1-fb637422962b-credential-socket\") pod \"77fc5206-8873-4bcc-89b1-fb637422962b\" (UID: \"77fc5206-8873-4bcc-89b1-fb637422962b\") " Nov 03 13:30:41.283870 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:41.280704 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77fc5206-8873-4bcc-89b1-fb637422962b-workload-certs" pod "77fc5206-8873-4bcc-89b1-fb637422962b" (UID: "77fc5206-8873-4bcc-89b1-fb637422962b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:30:41.283870 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:41.281104 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77fc5206-8873-4bcc-89b1-fb637422962b-istiod-ca-cert" pod "77fc5206-8873-4bcc-89b1-fb637422962b" (UID: "77fc5206-8873-4bcc-89b1-fb637422962b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:30:41.283870 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:41.281673 5024 generic.go:356] "Generic (PLEG): container finished" podID="0561186c-eec8-4b5f-aac2-7aedf3f060cd" containerID="431b2852c65533851b4988d4bb5e16f30814a6104809be4f26940e5317443ad0" exitCode=0 Nov 03 13:30:41.283870 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:41.281932 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6958bdd4ff-mjcfh" event={"ID":"0561186c-eec8-4b5f-aac2-7aedf3f060cd","Type":"ContainerDied","Data":"431b2852c65533851b4988d4bb5e16f30814a6104809be4f26940e5317443ad0"} Nov 03 13:30:41.403101 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:41.319450 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77fc5206-8873-4bcc-89b1-fb637422962b-workload-socket" pod "77fc5206-8873-4bcc-89b1-fb637422962b" (UID: "77fc5206-8873-4bcc-89b1-fb637422962b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:30:41.403101 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:41.349018 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77fc5206-8873-4bcc-89b1-fb637422962b-credential-socket" pod "77fc5206-8873-4bcc-89b1-fb637422962b" (UID: "77fc5206-8873-4bcc-89b1-fb637422962b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:30:41.403101 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:41.349320 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77fc5206-8873-4bcc-89b1-fb637422962b-istio-data" pod "77fc5206-8873-4bcc-89b1-fb637422962b" (UID: "77fc5206-8873-4bcc-89b1-fb637422962b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:30:41.403101 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:41.392898 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77fc5206-8873-4bcc-89b1-fb637422962b-istio-token" pod "77fc5206-8873-4bcc-89b1-fb637422962b" (UID: "77fc5206-8873-4bcc-89b1-fb637422962b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:30:41.516448 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:41.420593 5024 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77fc5206-8873-4bcc-89b1-fb637422962b-istio-data\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:30:41.516448 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:41.420631 5024 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77fc5206-8873-4bcc-89b1-fb637422962b-workload-socket\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:30:41.516448 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:41.420648 5024 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77fc5206-8873-4bcc-89b1-fb637422962b-credential-socket\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:30:41.516448 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:41.420663 5024 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77fc5206-8873-4bcc-89b1-fb637422962b-workload-certs\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:30:41.516448 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:41.420679 5024 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77fc5206-8873-4bcc-89b1-fb637422962b-istiod-ca-cert\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:30:41.516448 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:41.420693 5024 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77fc5206-8873-4bcc-89b1-fb637422962b-istio-token\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:30:41.550348 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:41.548922 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/77fc5206-8873-4bcc-89b1-fb637422962b-istio-podinfo" pod "77fc5206-8873-4bcc-89b1-fb637422962b" (UID: "77fc5206-8873-4bcc-89b1-fb637422962b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 13:30:41.628310 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:41.624602 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-kjbs5" Nov 03 13:30:41.686306 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:41.665075 5024 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77fc5206-8873-4bcc-89b1-fb637422962b-istio-podinfo\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:30:41.688508 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:41.688459 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77fc5206-8873-4bcc-89b1-fb637422962b-kube-api-access-j6pt7" pod "77fc5206-8873-4bcc-89b1-fb637422962b" (UID: "77fc5206-8873-4bcc-89b1-fb637422962b"). InnerVolumeSpecName "kube-api-access-j6pt7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:30:41.753862 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:41.752647 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77fc5206-8873-4bcc-89b1-fb637422962b-istio-envoy" pod "77fc5206-8873-4bcc-89b1-fb637422962b" (UID: "77fc5206-8873-4bcc-89b1-fb637422962b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:30:41.770422 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:41.766379 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4eeaf0ad-8a27-44c2-8641-88d4d64e7969-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4eeaf0ad-8a27-44c2-8641-88d4d64e7969-istio-podinfo\") pod \"4eeaf0ad-8a27-44c2-8641-88d4d64e7969\" (UID: \"4eeaf0ad-8a27-44c2-8641-88d4d64e7969\") " Nov 03 13:30:41.770422 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:41.766483 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4eeaf0ad-8a27-44c2-8641-88d4d64e7969-kube-api-access-hbh6f\" (UniqueName: \"kubernetes.io/projected/4eeaf0ad-8a27-44c2-8641-88d4d64e7969-kube-api-access-hbh6f\") pod \"4eeaf0ad-8a27-44c2-8641-88d4d64e7969\" (UID: \"4eeaf0ad-8a27-44c2-8641-88d4d64e7969\") " Nov 03 13:30:41.770422 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:41.766513 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4eeaf0ad-8a27-44c2-8641-88d4d64e7969-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4eeaf0ad-8a27-44c2-8641-88d4d64e7969-istio-data\") pod \"4eeaf0ad-8a27-44c2-8641-88d4d64e7969\" (UID: \"4eeaf0ad-8a27-44c2-8641-88d4d64e7969\") " Nov 03 13:30:41.770422 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:41.766569 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4eeaf0ad-8a27-44c2-8641-88d4d64e7969-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4eeaf0ad-8a27-44c2-8641-88d4d64e7969-workload-socket\") pod \"4eeaf0ad-8a27-44c2-8641-88d4d64e7969\" (UID: \"4eeaf0ad-8a27-44c2-8641-88d4d64e7969\") " Nov 03 13:30:41.770422 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:41.766605 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4eeaf0ad-8a27-44c2-8641-88d4d64e7969-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4eeaf0ad-8a27-44c2-8641-88d4d64e7969-istiod-ca-cert\") pod \"4eeaf0ad-8a27-44c2-8641-88d4d64e7969\" (UID: \"4eeaf0ad-8a27-44c2-8641-88d4d64e7969\") " Nov 03 13:30:41.770422 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:41.766677 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4eeaf0ad-8a27-44c2-8641-88d4d64e7969-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4eeaf0ad-8a27-44c2-8641-88d4d64e7969-workload-certs\") pod \"4eeaf0ad-8a27-44c2-8641-88d4d64e7969\" (UID: \"4eeaf0ad-8a27-44c2-8641-88d4d64e7969\") " Nov 03 13:30:41.770422 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:41.766736 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4eeaf0ad-8a27-44c2-8641-88d4d64e7969-istio-token\" (UniqueName: \"kubernetes.io/projected/4eeaf0ad-8a27-44c2-8641-88d4d64e7969-istio-token\") pod \"4eeaf0ad-8a27-44c2-8641-88d4d64e7969\" (UID: \"4eeaf0ad-8a27-44c2-8641-88d4d64e7969\") " Nov 03 13:30:41.770422 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:41.766847 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4eeaf0ad-8a27-44c2-8641-88d4d64e7969-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4eeaf0ad-8a27-44c2-8641-88d4d64e7969-credential-socket\") pod \"4eeaf0ad-8a27-44c2-8641-88d4d64e7969\" (UID: \"4eeaf0ad-8a27-44c2-8641-88d4d64e7969\") " Nov 03 13:30:41.770422 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:41.766943 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4eeaf0ad-8a27-44c2-8641-88d4d64e7969-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4eeaf0ad-8a27-44c2-8641-88d4d64e7969-istio-envoy\") pod \"4eeaf0ad-8a27-44c2-8641-88d4d64e7969\" (UID: \"4eeaf0ad-8a27-44c2-8641-88d4d64e7969\") " Nov 03 13:30:41.770422 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:41.768772 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4eeaf0ad-8a27-44c2-8641-88d4d64e7969-workload-socket" pod "4eeaf0ad-8a27-44c2-8641-88d4d64e7969" (UID: "4eeaf0ad-8a27-44c2-8641-88d4d64e7969"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:30:41.777834 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:41.777796 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4eeaf0ad-8a27-44c2-8641-88d4d64e7969-kube-api-access-hbh6f" pod "4eeaf0ad-8a27-44c2-8641-88d4d64e7969" (UID: "4eeaf0ad-8a27-44c2-8641-88d4d64e7969"). InnerVolumeSpecName "kube-api-access-hbh6f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:30:41.778250 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:41.778229 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4eeaf0ad-8a27-44c2-8641-88d4d64e7969-workload-certs" pod "4eeaf0ad-8a27-44c2-8641-88d4d64e7969" (UID: "4eeaf0ad-8a27-44c2-8641-88d4d64e7969"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:30:41.779559 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:41.779532 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4eeaf0ad-8a27-44c2-8641-88d4d64e7969-istiod-ca-cert" pod "4eeaf0ad-8a27-44c2-8641-88d4d64e7969" (UID: "4eeaf0ad-8a27-44c2-8641-88d4d64e7969"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:30:41.780087 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:41.780063 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4eeaf0ad-8a27-44c2-8641-88d4d64e7969-istio-data" pod "4eeaf0ad-8a27-44c2-8641-88d4d64e7969" (UID: "4eeaf0ad-8a27-44c2-8641-88d4d64e7969"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:30:41.784114 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:41.783506 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4eeaf0ad-8a27-44c2-8641-88d4d64e7969-istio-token" pod "4eeaf0ad-8a27-44c2-8641-88d4d64e7969" (UID: "4eeaf0ad-8a27-44c2-8641-88d4d64e7969"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:30:41.793497 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:41.784262 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4eeaf0ad-8a27-44c2-8641-88d4d64e7969-credential-socket" pod "4eeaf0ad-8a27-44c2-8641-88d4d64e7969" (UID: "4eeaf0ad-8a27-44c2-8641-88d4d64e7969"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:30:41.970315 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:41.931468 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hbh6f\" (UniqueName: \"kubernetes.io/projected/4eeaf0ad-8a27-44c2-8641-88d4d64e7969-kube-api-access-hbh6f\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:30:42.214476 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:42.203069 5024 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4eeaf0ad-8a27-44c2-8641-88d4d64e7969-istio-data\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:30:42.215127 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:42.215088 5024 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4eeaf0ad-8a27-44c2-8641-88d4d64e7969-workload-socket\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:30:42.215308 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:42.215273 5024 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4eeaf0ad-8a27-44c2-8641-88d4d64e7969-istiod-ca-cert\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:30:42.215424 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:42.215409 5024 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4eeaf0ad-8a27-44c2-8641-88d4d64e7969-workload-certs\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:30:42.215540 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:42.215526 5024 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4eeaf0ad-8a27-44c2-8641-88d4d64e7969-istio-token\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:30:42.215680 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:42.215657 5024 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4eeaf0ad-8a27-44c2-8641-88d4d64e7969-credential-socket\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:30:42.220408 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:42.220367 5024 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77fc5206-8873-4bcc-89b1-fb637422962b-istio-envoy\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:30:42.220622 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:42.220599 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j6pt7\" (UniqueName: \"kubernetes.io/projected/77fc5206-8873-4bcc-89b1-fb637422962b-kube-api-access-j6pt7\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:30:42.368597 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:42.359958 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4eeaf0ad-8a27-44c2-8641-88d4d64e7969-istio-envoy" pod "4eeaf0ad-8a27-44c2-8641-88d4d64e7969" (UID: "4eeaf0ad-8a27-44c2-8641-88d4d64e7969"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:30:42.399916 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:42.399878 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4eeaf0ad-8a27-44c2-8641-88d4d64e7969-istio-podinfo" pod "4eeaf0ad-8a27-44c2-8641-88d4d64e7969" (UID: "4eeaf0ad-8a27-44c2-8641-88d4d64e7969"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 13:30:42.420391 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:42.413659 5024 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4eeaf0ad-8a27-44c2-8641-88d4d64e7969-istio-envoy\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:30:42.420391 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:42.413721 5024 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4eeaf0ad-8a27-44c2-8641-88d4d64e7969-istio-podinfo\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:30:42.463452 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:42.463403 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-08111b-openshift-default-5b6dd8787c-v2hw4" event={"ID":"ca014f3a-ff92-4bb9-9a7a-8c8d3dabe486","Type":"ContainerStarted","Data":"5a0d74c33d3dea8b8445327a18b596aa922754e42b80e3915e17f7f73bbb73b0"} Nov 03 13:30:42.463751 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:42.463730 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-08111b-openshift-default-5b6dd8787c-v2hw4" event={"ID":"ca014f3a-ff92-4bb9-9a7a-8c8d3dabe486","Type":"ContainerStarted","Data":"975c0ba90f891e0704532a6fc22c9f2065279f7e614128867cbdf569902563d1"} Nov 03 13:30:42.539346 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:42.538804 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-kjbs5" event={"ID":"4eeaf0ad-8a27-44c2-8641-88d4d64e7969","Type":"ContainerDied","Data":"3a6652419812eb4faea52e9e5742073ecbd9197cac6473f2bd01b26496379afe"} Nov 03 13:30:42.539346 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:42.538885 5024 scope.go:122] "RemoveContainer" containerID="fbdcddf96d6c74e3a56a93df3868b856b4423d9bf7ad5fa43d10d6b5cd79be48" Nov 03 13:30:42.539346 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:42.538994 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-kjbs5" Nov 03 13:30:42.579637 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:42.579555 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-08111b-openshift-default-5b6dd8787c-v2hw4" podStartSLOduration=5.579528993 podStartE2EDuration="5.579528993s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:30:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:30:42.534859329 +0000 UTC m=+1322.543050607" watchObservedRunningTime="2026-11-03 13:30:42.579528993 +0000 UTC m=+1322.587720273" Nov 03 13:30:42.584803 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:42.584745 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-rddcl" event={"ID":"77fc5206-8873-4bcc-89b1-fb637422962b","Type":"ContainerDied","Data":"c74240ad967a5e722fe522abd24071bdbf6326c3b7fb4dc6811653679d42a35e"} Nov 03 13:30:42.585243 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:42.585220 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-rddcl" Nov 03 13:30:42.601926 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:42.601895 5024 scope.go:122] "RemoveContainer" containerID="aebdfbf8569969e34481bf8e94bc420143339f71a7e38b7baa320243eb1d2097" Nov 03 13:30:42.663491 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:42.663458 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6958bdd4ff-mjcfh" Nov 03 13:30:42.797310 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:42.766639 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-08111b-openshift-default-5b6dd8787c-v2hw4" Nov 03 13:30:42.797310 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:42.767276 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0561186c-eec8-4b5f-aac2-7aedf3f060cd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0561186c-eec8-4b5f-aac2-7aedf3f060cd-plugins-config-volume\") pod \"0561186c-eec8-4b5f-aac2-7aedf3f060cd\" (UID: \"0561186c-eec8-4b5f-aac2-7aedf3f060cd\") " Nov 03 13:30:42.797310 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:42.767409 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0561186c-eec8-4b5f-aac2-7aedf3f060cd-kube-api-access-6mckk\" (UniqueName: \"kubernetes.io/projected/0561186c-eec8-4b5f-aac2-7aedf3f060cd-kube-api-access-6mckk\") pod \"0561186c-eec8-4b5f-aac2-7aedf3f060cd\" (UID: \"0561186c-eec8-4b5f-aac2-7aedf3f060cd\") " Nov 03 13:30:42.797310 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:42.767566 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0561186c-eec8-4b5f-aac2-7aedf3f060cd-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0561186c-eec8-4b5f-aac2-7aedf3f060cd-routing-overlay-volume\") pod \"0561186c-eec8-4b5f-aac2-7aedf3f060cd\" (UID: \"0561186c-eec8-4b5f-aac2-7aedf3f060cd\") " Nov 03 13:30:42.797310 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:42.767667 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0561186c-eec8-4b5f-aac2-7aedf3f060cd-provider-credentials\" (UniqueName: \"kubernetes.io/projected/0561186c-eec8-4b5f-aac2-7aedf3f060cd-provider-credentials\") pod \"0561186c-eec8-4b5f-aac2-7aedf3f060cd\" (UID: \"0561186c-eec8-4b5f-aac2-7aedf3f060cd\") " Nov 03 13:30:42.843569 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:42.797246 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0561186c-eec8-4b5f-aac2-7aedf3f060cd-plugins-config-volume" pod "0561186c-eec8-4b5f-aac2-7aedf3f060cd" (UID: "0561186c-eec8-4b5f-aac2-7aedf3f060cd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:30:42.843569 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:42.798548 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0561186c-eec8-4b5f-aac2-7aedf3f060cd-routing-overlay-volume" pod "0561186c-eec8-4b5f-aac2-7aedf3f060cd" (UID: "0561186c-eec8-4b5f-aac2-7aedf3f060cd"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:30:42.843569 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:42.805078 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0561186c-eec8-4b5f-aac2-7aedf3f060cd-kube-api-access-6mckk" pod "0561186c-eec8-4b5f-aac2-7aedf3f060cd" (UID: "0561186c-eec8-4b5f-aac2-7aedf3f060cd"). InnerVolumeSpecName "kube-api-access-6mckk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:30:42.855837 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:42.855788 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-rddcl"] Nov 03 13:30:42.865817 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:42.865178 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-rddcl"] Nov 03 13:30:42.904947 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:42.904857 5024 patch_prober.go:29] interesting pod/e2e-worker-w1-08111b-openshift-default-5b6dd8787c-v2hw4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" start-of-body= Nov 03 13:30:42.906166 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:42.906123 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-08111b-openshift-default-5b6dd8787c-v2hw4" podUID="ca014f3a-ff92-4bb9-9a7a-8c8d3dabe486" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" Nov 03 13:30:42.906588 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:42.906555 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0561186c-eec8-4b5f-aac2-7aedf3f060cd-provider-credentials" pod "0561186c-eec8-4b5f-aac2-7aedf3f060cd" (UID: "0561186c-eec8-4b5f-aac2-7aedf3f060cd"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:30:42.913313 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:42.912005 5024 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0561186c-eec8-4b5f-aac2-7aedf3f060cd-plugins-config-volume\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:30:42.913313 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:42.912042 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6mckk\" (UniqueName: \"kubernetes.io/projected/0561186c-eec8-4b5f-aac2-7aedf3f060cd-kube-api-access-6mckk\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:30:42.913313 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:42.912062 5024 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0561186c-eec8-4b5f-aac2-7aedf3f060cd-routing-overlay-volume\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:30:42.913313 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:42.912077 5024 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0561186c-eec8-4b5f-aac2-7aedf3f060cd-provider-credentials\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:30:43.054970 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:43.053226 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-kjbs5"] Nov 03 13:30:43.058716 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:43.058675 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-kjbs5"] Nov 03 13:30:43.644771 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:43.644721 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6958bdd4ff-mjcfh" event={"ID":"0561186c-eec8-4b5f-aac2-7aedf3f060cd","Type":"ContainerDied","Data":"9911afbe427087b1b806421a27db075c3d33751c81679c4366b6f14d473c2ae4"} Nov 03 13:30:43.645452 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:43.645428 5024 scope.go:122] "RemoveContainer" containerID="431b2852c65533851b4988d4bb5e16f30814a6104809be4f26940e5317443ad0" Nov 03 13:30:43.645625 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:43.644936 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6958bdd4ff-mjcfh" Nov 03 13:30:43.751584 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:43.751553 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-6958bdd4ff-mjcfh"] Nov 03 13:30:43.769331 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:43.764681 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-08111b-openshift-default-5b6dd8787c-v2hw4" Nov 03 13:30:43.769331 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:43.764742 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-6958bdd4ff-mjcfh"] Nov 03 13:30:44.112733 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:44.112691 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0561186c-eec8-4b5f-aac2-7aedf3f060cd" path="/var/lib/kubelet/pods/0561186c-eec8-4b5f-aac2-7aedf3f060cd/volumes" Nov 03 13:30:44.113402 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:44.113379 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4eeaf0ad-8a27-44c2-8641-88d4d64e7969" path="/var/lib/kubelet/pods/4eeaf0ad-8a27-44c2-8641-88d4d64e7969/volumes" Nov 03 13:30:44.113955 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:44.113935 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="77fc5206-8873-4bcc-89b1-fb637422962b" path="/var/lib/kubelet/pods/77fc5206-8873-4bcc-89b1-fb637422962b/volumes" Nov 03 13:30:44.654612 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:44.654576 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-08111b-openshift-default-5b6dd8787c-v2hw4" Nov 03 13:30:44.655794 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:44.655770 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-08111b-openshift-default-5b6dd8787c-v2hw4" Nov 03 13:30:46.367232 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:46.367188 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-08111b-744df4c7d7-b5xcs"] Nov 03 13:30:46.367939 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:46.367920 5024 cpu_manager.go:414] "containerMap: removing container" podUID="77fc5206-8873-4bcc-89b1-fb637422962b" containerName="istio-proxy" Nov 03 13:30:46.367939 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:46.367936 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77fc5206-8873-4bcc-89b1-fb637422962b" containerName="istio-proxy" Nov 03 13:30:46.368108 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:46.367950 5024 cpu_manager.go:414] "containerMap: removing container" podUID="4eeaf0ad-8a27-44c2-8641-88d4d64e7969" containerName="istio-proxy" Nov 03 13:30:46.368108 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:46.367955 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4eeaf0ad-8a27-44c2-8641-88d4d64e7969" containerName="istio-proxy" Nov 03 13:30:46.368108 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:46.367988 5024 cpu_manager.go:414] "containerMap: removing container" podUID="77fc5206-8873-4bcc-89b1-fb637422962b" containerName="istio-proxy" Nov 03 13:30:46.368108 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:46.367994 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77fc5206-8873-4bcc-89b1-fb637422962b" containerName="istio-proxy" Nov 03 13:30:46.368108 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:46.368024 5024 cpu_manager.go:414] "containerMap: removing container" podUID="0561186c-eec8-4b5f-aac2-7aedf3f060cd" containerName="payload-processing" Nov 03 13:30:46.368108 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:46.368029 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0561186c-eec8-4b5f-aac2-7aedf3f060cd" containerName="payload-processing" Nov 03 13:30:46.368451 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:46.368120 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="0561186c-eec8-4b5f-aac2-7aedf3f060cd" containerName="payload-processing" Nov 03 13:30:46.368451 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:46.368131 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="77fc5206-8873-4bcc-89b1-fb637422962b" containerName="istio-proxy" Nov 03 13:30:46.368451 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:46.368138 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="4eeaf0ad-8a27-44c2-8641-88d4d64e7969" containerName="istio-proxy" Nov 03 13:30:46.371930 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:46.371908 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-08111b-744df4c7d7-b5xcs" Nov 03 13:30:46.383686 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:46.383651 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w1-08111b\"" Nov 03 13:30:46.383910 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:46.383802 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-08111b-744df4c7d7-b5xcs"] Nov 03 13:30:46.385545 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:46.385513 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w1-08111b-dockercfg-hsxmg\"" Nov 03 13:30:46.404372 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:46.404341 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-08111b-9cc448f95-n6ttb"] Nov 03 13:30:46.406067 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:46.406030 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="77fc5206-8873-4bcc-89b1-fb637422962b" containerName="istio-proxy" Nov 03 13:30:46.434392 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:46.434329 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-08111b-9cc448f95-n6ttb"] Nov 03 13:30:46.434677 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:46.434452 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-08111b-9cc448f95-n6ttb" Nov 03 13:30:46.598306 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:46.591340 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8j7s\" (UniqueName: \"kubernetes.io/projected/459af5d0-4e46-4ad0-b123-5642ceae6e74-kube-api-access-b8j7s\") pod \"payload-pre-processing-e2e-worker-w1-08111b-744df4c7d7-b5xcs\" (UID: \"459af5d0-4e46-4ad0-b123-5642ceae6e74\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-08111b-744df4c7d7-b5xcs" Nov 03 13:30:46.598306 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:46.591399 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9zclm\" (UniqueName: \"kubernetes.io/projected/c4490dbf-a557-44df-b654-b7a47f5c0604-kube-api-access-9zclm\") pod \"payload-processing-e2e-worker-w1-08111b-9cc448f95-n6ttb\" (UID: \"c4490dbf-a557-44df-b654-b7a47f5c0604\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-08111b-9cc448f95-n6ttb" Nov 03 13:30:46.598306 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:46.591812 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/459af5d0-4e46-4ad0-b123-5642ceae6e74-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-08111b-744df4c7d7-b5xcs\" (UID: \"459af5d0-4e46-4ad0-b123-5642ceae6e74\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-08111b-744df4c7d7-b5xcs" Nov 03 13:30:46.598306 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:46.591869 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4490dbf-a557-44df-b654-b7a47f5c0604-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-08111b-9cc448f95-n6ttb\" (UID: \"c4490dbf-a557-44df-b654-b7a47f5c0604\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-08111b-9cc448f95-n6ttb" Nov 03 13:30:46.711369 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:46.696573 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/459af5d0-4e46-4ad0-b123-5642ceae6e74-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-08111b-744df4c7d7-b5xcs\" (UID: \"459af5d0-4e46-4ad0-b123-5642ceae6e74\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-08111b-744df4c7d7-b5xcs" Nov 03 13:30:46.754796 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:46.754764 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4490dbf-a557-44df-b654-b7a47f5c0604-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-08111b-9cc448f95-n6ttb\" (UID: \"c4490dbf-a557-44df-b654-b7a47f5c0604\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-08111b-9cc448f95-n6ttb" Nov 03 13:30:46.755199 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:46.755171 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8j7s\" (UniqueName: \"kubernetes.io/projected/459af5d0-4e46-4ad0-b123-5642ceae6e74-kube-api-access-b8j7s\") pod \"payload-pre-processing-e2e-worker-w1-08111b-744df4c7d7-b5xcs\" (UID: \"459af5d0-4e46-4ad0-b123-5642ceae6e74\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-08111b-744df4c7d7-b5xcs" Nov 03 13:30:46.762311 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:46.756120 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zclm\" (UniqueName: \"kubernetes.io/projected/c4490dbf-a557-44df-b654-b7a47f5c0604-kube-api-access-9zclm\") pod \"payload-processing-e2e-worker-w1-08111b-9cc448f95-n6ttb\" (UID: \"c4490dbf-a557-44df-b654-b7a47f5c0604\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-08111b-9cc448f95-n6ttb" Nov 03 13:30:46.762950 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:46.754625 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/459af5d0-4e46-4ad0-b123-5642ceae6e74-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-08111b-744df4c7d7-b5xcs\" (UID: \"459af5d0-4e46-4ad0-b123-5642ceae6e74\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-08111b-744df4c7d7-b5xcs" Nov 03 13:30:46.770324 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:46.763882 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4490dbf-a557-44df-b654-b7a47f5c0604-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-08111b-9cc448f95-n6ttb\" (UID: \"c4490dbf-a557-44df-b654-b7a47f5c0604\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-08111b-9cc448f95-n6ttb" Nov 03 13:30:46.845137 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:46.845102 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zclm\" (UniqueName: \"kubernetes.io/projected/c4490dbf-a557-44df-b654-b7a47f5c0604-kube-api-access-9zclm\") pod \"payload-processing-e2e-worker-w1-08111b-9cc448f95-n6ttb\" (UID: \"c4490dbf-a557-44df-b654-b7a47f5c0604\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-08111b-9cc448f95-n6ttb" Nov 03 13:30:46.857828 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:46.857539 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8j7s\" (UniqueName: \"kubernetes.io/projected/459af5d0-4e46-4ad0-b123-5642ceae6e74-kube-api-access-b8j7s\") pod \"payload-pre-processing-e2e-worker-w1-08111b-744df4c7d7-b5xcs\" (UID: \"459af5d0-4e46-4ad0-b123-5642ceae6e74\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-08111b-744df4c7d7-b5xcs" Nov 03 13:30:47.010361 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:47.007583 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-08111b-744df4c7d7-b5xcs" Nov 03 13:30:47.082364 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:47.072681 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-08111b-9cc448f95-n6ttb" Nov 03 13:30:47.517457 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:47.517416 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-08111b-9cc448f95-n6ttb"] Nov 03 13:30:47.529956 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:47.529917 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-08111b-744df4c7d7-b5xcs"] Nov 03 13:30:47.547013 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:30:47.546969 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod459af5d0_4e46_4ad0_b123_5642ceae6e74.slice/crio-4b92eb884b44004c08712a47feb1b3b2035fef8b1233f7ec0989559f820d691b WatchSource:0}: Error finding container 4b92eb884b44004c08712a47feb1b3b2035fef8b1233f7ec0989559f820d691b: Status 404 returned error can't find the container with id 4b92eb884b44004c08712a47feb1b3b2035fef8b1233f7ec0989559f820d691b Nov 03 13:30:47.784351 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:47.784241 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-08111b-744df4c7d7-b5xcs" event={"ID":"459af5d0-4e46-4ad0-b123-5642ceae6e74","Type":"ContainerStarted","Data":"86d550975a2ec9158fd99e9d700a0502ee24107c5ee979b725760987d8f292be"} Nov 03 13:30:47.784506 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:47.784351 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-08111b-744df4c7d7-b5xcs" event={"ID":"459af5d0-4e46-4ad0-b123-5642ceae6e74","Type":"ContainerStarted","Data":"4b92eb884b44004c08712a47feb1b3b2035fef8b1233f7ec0989559f820d691b"} Nov 03 13:30:47.784506 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:47.784392 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-08111b-744df4c7d7-b5xcs" Nov 03 13:30:47.786554 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:47.786528 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-08111b-9cc448f95-n6ttb" event={"ID":"c4490dbf-a557-44df-b654-b7a47f5c0604","Type":"ContainerStarted","Data":"33652f2e5fd1ab20dc35880a4a41b3da56699adebf21290475a1c9d3fba35f1b"} Nov 03 13:30:47.786699 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:47.786573 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-08111b-9cc448f95-n6ttb" event={"ID":"c4490dbf-a557-44df-b654-b7a47f5c0604","Type":"ContainerStarted","Data":"468e3c8558af17ac063448bf1838df60f8c635637ad1e5abc1924e72bac1f7dd"} Nov 03 13:30:47.786699 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:47.786646 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-08111b-9cc448f95-n6ttb" Nov 03 13:30:47.804707 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:47.804645 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-08111b-744df4c7d7-b5xcs" podStartSLOduration=1.8046249859999999 podStartE2EDuration="1.804624986s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:30:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:30:47.803538204 +0000 UTC m=+1327.811729481" watchObservedRunningTime="2026-11-03 13:30:47.804624986 +0000 UTC m=+1327.812816265" Nov 03 13:30:47.828433 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:47.828372 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w1-08111b-9cc448f95-n6ttb" podStartSLOduration=1.828356433 podStartE2EDuration="1.828356433s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:30:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:30:47.824828556 +0000 UTC m=+1327.833019835" watchObservedRunningTime="2026-11-03 13:30:47.828356433 +0000 UTC m=+1327.836547776" Nov 03 13:30:51.113107 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:51.113044 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-5bffndnqx" podUID="6908d762-9f04-4823-a7f2-00a4e703b601" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 13:30:54.839049 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:30:54.839001 5024 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-a62d22/agc-cred-e942f4e7c82f: secret "agc-cred-e942f4e7c82f" not found Nov 03 13:30:54.839049 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:30:54.839039 5024 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-cf4789xvd: secret "agc-cred-e942f4e7c82f" not found Nov 03 13:30:54.839510 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:30:54.839096 5024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9b739270-1c05-44b6-bed6-de7eaadb689f-provider-credentials podName:9b739270-1c05-44b6-bed6-de7eaadb689f nodeName:}" failed. No retries permitted until 2026-11-03 13:32:56.839082169 +0000 UTC m=+1456.847273428 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/9b739270-1c05-44b6-bed6-de7eaadb689f-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-a62d22-cf4789xvd" (UID: "9b739270-1c05-44b6-bed6-de7eaadb689f") : secret "agc-cred-e942f4e7c82f" not found Nov 03 13:30:58.798307 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:58.797821 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-08111b-744df4c7d7-b5xcs" Nov 03 13:30:58.798307 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:30:58.797947 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-08111b-9cc448f95-n6ttb" Nov 03 13:31:01.121988 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:31:01.121945 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-5bffndnqx" podUID="6908d762-9f04-4823-a7f2-00a4e703b601" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 13:31:02.865309 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:31:02.860675 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-d66hs"] Nov 03 13:31:02.865309 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:31:02.860995 5024 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-d66hs" podUID="14bfce12-7ae7-4866-aa46-bce174d28ca6" containerName="istio-proxy" containerID="cri-o://75148bbfc8ef536a74d3997858931cd0358236c4a5f8adbc9a723758f3a07290" gracePeriod=30 Nov 03 13:31:07.446309 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:31:07.422460 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d3ca48/premium-w3-d3ca48-kserve-75ffc959b-zvjbc" podUID="add1e8b9-e7fd-4e4d-8d10-f97d0e2612ba" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 13:31:08.435410 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:31:08.426386 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d3ca48/premium-w3-d3ca48-kserve-75ffc959b-zvjbc" podUID="add1e8b9-e7fd-4e4d-8d10-f97d0e2612ba" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 13:31:08.549339 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:31:08.549259 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-d3ca48/premium-w3-d3ca48-kserve-75ffc959b-zvjbc" Nov 03 13:31:08.549827 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:31:08.549806 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-08111b/simulator-w1-08111b-kserve-7f65c5867-btf5b"] Nov 03 13:31:08.684207 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:31:08.683239 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-d3ca48/premium-w3-d3ca48-kserve-75ffc959b-zvjbc" Nov 03 13:31:08.684207 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:31:08.683300 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-08111b/simulator-w1-08111b-kserve-7f65c5867-btf5b"] Nov 03 13:31:08.684207 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:31:08.683370 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-08111b/simulator-w1-08111b-kserve-7f65c5867-btf5b" Nov 03 13:31:08.689028 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:31:08.688957 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-08111b\"/\"simulator-w1-08111b-kserve-self-signed-certs\"" Nov 03 13:31:08.693360 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:31:08.693330 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-08111b\"/\"openshift-service-ca.crt\"" Nov 03 13:31:08.693680 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:31:08.693659 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-08111b\"/\"default-dockercfg-mbl22\"" Nov 03 13:31:08.693916 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:31:08.693899 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-08111b\"/\"kube-root-ca.crt\"" Nov 03 13:31:08.796307 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:31:08.771010 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e8aba665-e4cc-4e67-8cdd-d09ef7542e15-tmp-dir\") pod \"simulator-w1-08111b-kserve-7f65c5867-btf5b\" (UID: \"e8aba665-e4cc-4e67-8cdd-d09ef7542e15\") " pod="e2e-models-e2e-worker-w1-08111b/simulator-w1-08111b-kserve-7f65c5867-btf5b" Nov 03 13:31:08.935317 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:31:08.930146 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d9fbk\" (UniqueName: \"kubernetes.io/projected/e8aba665-e4cc-4e67-8cdd-d09ef7542e15-kube-api-access-d9fbk\") pod \"simulator-w1-08111b-kserve-7f65c5867-btf5b\" (UID: \"e8aba665-e4cc-4e67-8cdd-d09ef7542e15\") " pod="e2e-models-e2e-worker-w1-08111b/simulator-w1-08111b-kserve-7f65c5867-btf5b" Nov 03 13:31:08.935317 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:31:08.930205 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e8aba665-e4cc-4e67-8cdd-d09ef7542e15-tls-certs\") pod \"simulator-w1-08111b-kserve-7f65c5867-btf5b\" (UID: \"e8aba665-e4cc-4e67-8cdd-d09ef7542e15\") " pod="e2e-models-e2e-worker-w1-08111b/simulator-w1-08111b-kserve-7f65c5867-btf5b" Nov 03 13:31:08.935317 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:31:08.930404 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e8aba665-e4cc-4e67-8cdd-d09ef7542e15-dshm\") pod \"simulator-w1-08111b-kserve-7f65c5867-btf5b\" (UID: \"e8aba665-e4cc-4e67-8cdd-d09ef7542e15\") " pod="e2e-models-e2e-worker-w1-08111b/simulator-w1-08111b-kserve-7f65c5867-btf5b" Nov 03 13:31:08.935317 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:31:08.930558 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e8aba665-e4cc-4e67-8cdd-d09ef7542e15-model-cache\") pod \"simulator-w1-08111b-kserve-7f65c5867-btf5b\" (UID: \"e8aba665-e4cc-4e67-8cdd-d09ef7542e15\") " pod="e2e-models-e2e-worker-w1-08111b/simulator-w1-08111b-kserve-7f65c5867-btf5b" Nov 03 13:31:08.935317 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:31:08.930619 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e8aba665-e4cc-4e67-8cdd-d09ef7542e15-home\") pod \"simulator-w1-08111b-kserve-7f65c5867-btf5b\" (UID: \"e8aba665-e4cc-4e67-8cdd-d09ef7542e15\") " pod="e2e-models-e2e-worker-w1-08111b/simulator-w1-08111b-kserve-7f65c5867-btf5b" Nov 03 13:31:08.977372 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:31:08.969912 5024 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-nr5hn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 13:31:08.977372 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:31:08.969991 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-nr5hn" podUID="3cd0d23e-9823-4988-ac79-0820dcc4c028" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 13:31:09.104334 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:31:09.094052 5024 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-xtsb8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 13:31:09.104334 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:31:09.094122 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-xtsb8" podUID="b46130ce-1911-475e-8ec7-5204881eb5ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 13:31:09.126039 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:31:09.125988 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e8aba665-e4cc-4e67-8cdd-d09ef7542e15-dshm\") pod \"simulator-w1-08111b-kserve-7f65c5867-btf5b\" (UID: \"e8aba665-e4cc-4e67-8cdd-d09ef7542e15\") " pod="e2e-models-e2e-worker-w1-08111b/simulator-w1-08111b-kserve-7f65c5867-btf5b" Nov 03 13:31:09.126383 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:31:09.126349 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e8aba665-e4cc-4e67-8cdd-d09ef7542e15-model-cache\") pod \"simulator-w1-08111b-kserve-7f65c5867-btf5b\" (UID: \"e8aba665-e4cc-4e67-8cdd-d09ef7542e15\") " pod="e2e-models-e2e-worker-w1-08111b/simulator-w1-08111b-kserve-7f65c5867-btf5b" Nov 03 13:31:09.126605 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:31:09.126583 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e8aba665-e4cc-4e67-8cdd-d09ef7542e15-home\") pod \"simulator-w1-08111b-kserve-7f65c5867-btf5b\" (UID: \"e8aba665-e4cc-4e67-8cdd-d09ef7542e15\") " pod="e2e-models-e2e-worker-w1-08111b/simulator-w1-08111b-kserve-7f65c5867-btf5b" Nov 03 13:31:09.126955 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:31:09.126921 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e8aba665-e4cc-4e67-8cdd-d09ef7542e15-tmp-dir\") pod \"simulator-w1-08111b-kserve-7f65c5867-btf5b\" (UID: \"e8aba665-e4cc-4e67-8cdd-d09ef7542e15\") " pod="e2e-models-e2e-worker-w1-08111b/simulator-w1-08111b-kserve-7f65c5867-btf5b" Nov 03 13:31:09.161576 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:31:09.143876 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e8aba665-e4cc-4e67-8cdd-d09ef7542e15-model-cache\") pod \"simulator-w1-08111b-kserve-7f65c5867-btf5b\" (UID: \"e8aba665-e4cc-4e67-8cdd-d09ef7542e15\") " pod="e2e-models-e2e-worker-w1-08111b/simulator-w1-08111b-kserve-7f65c5867-btf5b" Nov 03 13:31:09.170613 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:31:09.144465 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e8aba665-e4cc-4e67-8cdd-d09ef7542e15-home\") pod \"simulator-w1-08111b-kserve-7f65c5867-btf5b\" (UID: \"e8aba665-e4cc-4e67-8cdd-d09ef7542e15\") " pod="e2e-models-e2e-worker-w1-08111b/simulator-w1-08111b-kserve-7f65c5867-btf5b" Nov 03 13:31:09.177352 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:31:09.144705 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e8aba665-e4cc-4e67-8cdd-d09ef7542e15-tmp-dir\") pod \"simulator-w1-08111b-kserve-7f65c5867-btf5b\" (UID: \"e8aba665-e4cc-4e67-8cdd-d09ef7542e15\") " pod="e2e-models-e2e-worker-w1-08111b/simulator-w1-08111b-kserve-7f65c5867-btf5b" Nov 03 13:31:09.177681 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:31:09.164130 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9fbk\" (UniqueName: \"kubernetes.io/projected/e8aba665-e4cc-4e67-8cdd-d09ef7542e15-kube-api-access-d9fbk\") pod \"simulator-w1-08111b-kserve-7f65c5867-btf5b\" (UID: \"e8aba665-e4cc-4e67-8cdd-d09ef7542e15\") " pod="e2e-models-e2e-worker-w1-08111b/simulator-w1-08111b-kserve-7f65c5867-btf5b" Nov 03 13:31:09.191427 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:31:09.191372 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e8aba665-e4cc-4e67-8cdd-d09ef7542e15-tls-certs\") pod \"simulator-w1-08111b-kserve-7f65c5867-btf5b\" (UID: \"e8aba665-e4cc-4e67-8cdd-d09ef7542e15\") " pod="e2e-models-e2e-worker-w1-08111b/simulator-w1-08111b-kserve-7f65c5867-btf5b" Nov 03 13:31:09.302573 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:31:09.302468 5024 generic.go:356] "Generic (PLEG): container finished" podID="14bfce12-7ae7-4866-aa46-bce174d28ca6" containerID="75148bbfc8ef536a74d3997858931cd0358236c4a5f8adbc9a723758f3a07290" exitCode=0 Nov 03 13:31:09.340895 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:31:09.322642 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-d66hs" event={"ID":"14bfce12-7ae7-4866-aa46-bce174d28ca6","Type":"ContainerDied","Data":"75148bbfc8ef536a74d3997858931cd0358236c4a5f8adbc9a723758f3a07290"} Nov 03 13:31:09.369313 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:31:09.365342 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e8aba665-e4cc-4e67-8cdd-d09ef7542e15-dshm\") pod \"simulator-w1-08111b-kserve-7f65c5867-btf5b\" (UID: \"e8aba665-e4cc-4e67-8cdd-d09ef7542e15\") " pod="e2e-models-e2e-worker-w1-08111b/simulator-w1-08111b-kserve-7f65c5867-btf5b" Nov 03 13:31:09.495340 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:31:09.487474 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e8aba665-e4cc-4e67-8cdd-d09ef7542e15-tls-certs\") pod \"simulator-w1-08111b-kserve-7f65c5867-btf5b\" (UID: \"e8aba665-e4cc-4e67-8cdd-d09ef7542e15\") " pod="e2e-models-e2e-worker-w1-08111b/simulator-w1-08111b-kserve-7f65c5867-btf5b" Nov 03 13:31:09.916313 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:31:09.884075 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9fbk\" (UniqueName: \"kubernetes.io/projected/e8aba665-e4cc-4e67-8cdd-d09ef7542e15-kube-api-access-d9fbk\") pod \"simulator-w1-08111b-kserve-7f65c5867-btf5b\" (UID: \"e8aba665-e4cc-4e67-8cdd-d09ef7542e15\") " pod="e2e-models-e2e-worker-w1-08111b/simulator-w1-08111b-kserve-7f65c5867-btf5b" Nov 03 13:31:10.249702 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:31:10.217963 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-08111b/simulator-w1-08111b-kserve-7f65c5867-btf5b" Nov 03 13:31:10.333881 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:31:10.333817 5024 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d7mwz" containerName="extractor" Nov 03 13:31:10.619651 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:31:10.619607 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-d66hs" event={"ID":"14bfce12-7ae7-4866-aa46-bce174d28ca6","Type":"ContainerDied","Data":"db87f7221ec232574aa0ddb54f0eb29f4c55ca4b81071f0f877f24fef00749be"} Nov 03 13:31:10.619894 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:31:10.619873 5024 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="db87f7221ec232574aa0ddb54f0eb29f4c55ca4b81071f0f877f24fef00749be" Nov 03 13:31:10.745053 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:31:10.744701 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-d66hs" Nov 03 13:31:10.889991 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:31:10.889929 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/14bfce12-7ae7-4866-aa46-bce174d28ca6-istio-token\" (UniqueName: \"kubernetes.io/projected/14bfce12-7ae7-4866-aa46-bce174d28ca6-istio-token\") pod \"14bfce12-7ae7-4866-aa46-bce174d28ca6\" (UID: \"14bfce12-7ae7-4866-aa46-bce174d28ca6\") " Nov 03 13:31:10.890180 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:31:10.890007 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14bfce12-7ae7-4866-aa46-bce174d28ca6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14bfce12-7ae7-4866-aa46-bce174d28ca6-istio-data\") pod \"14bfce12-7ae7-4866-aa46-bce174d28ca6\" (UID: \"14bfce12-7ae7-4866-aa46-bce174d28ca6\") " Nov 03 13:31:10.890180 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:31:10.890059 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14bfce12-7ae7-4866-aa46-bce174d28ca6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14bfce12-7ae7-4866-aa46-bce174d28ca6-workload-socket\") pod \"14bfce12-7ae7-4866-aa46-bce174d28ca6\" (UID: \"14bfce12-7ae7-4866-aa46-bce174d28ca6\") " Nov 03 13:31:10.890180 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:31:10.890136 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14bfce12-7ae7-4866-aa46-bce174d28ca6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14bfce12-7ae7-4866-aa46-bce174d28ca6-workload-certs\") pod \"14bfce12-7ae7-4866-aa46-bce174d28ca6\" (UID: \"14bfce12-7ae7-4866-aa46-bce174d28ca6\") " Nov 03 13:31:10.890180 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:31:10.890167 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14bfce12-7ae7-4866-aa46-bce174d28ca6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14bfce12-7ae7-4866-aa46-bce174d28ca6-credential-socket\") pod \"14bfce12-7ae7-4866-aa46-bce174d28ca6\" (UID: \"14bfce12-7ae7-4866-aa46-bce174d28ca6\") " Nov 03 13:31:10.890472 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:31:10.890431 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/14bfce12-7ae7-4866-aa46-bce174d28ca6-kube-api-access-gv797\" (UniqueName: \"kubernetes.io/projected/14bfce12-7ae7-4866-aa46-bce174d28ca6-kube-api-access-gv797\") pod \"14bfce12-7ae7-4866-aa46-bce174d28ca6\" (UID: \"14bfce12-7ae7-4866-aa46-bce174d28ca6\") " Nov 03 13:31:10.890533 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:31:10.890475 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14bfce12-7ae7-4866-aa46-bce174d28ca6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14bfce12-7ae7-4866-aa46-bce174d28ca6-istio-envoy\") pod \"14bfce12-7ae7-4866-aa46-bce174d28ca6\" (UID: \"14bfce12-7ae7-4866-aa46-bce174d28ca6\") " Nov 03 13:31:10.890533 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:31:10.890510 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/14bfce12-7ae7-4866-aa46-bce174d28ca6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14bfce12-7ae7-4866-aa46-bce174d28ca6-istio-podinfo\") pod \"14bfce12-7ae7-4866-aa46-bce174d28ca6\" (UID: \"14bfce12-7ae7-4866-aa46-bce174d28ca6\") " Nov 03 13:31:10.931348 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:31:10.890631 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/14bfce12-7ae7-4866-aa46-bce174d28ca6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14bfce12-7ae7-4866-aa46-bce174d28ca6-istiod-ca-cert\") pod \"14bfce12-7ae7-4866-aa46-bce174d28ca6\" (UID: \"14bfce12-7ae7-4866-aa46-bce174d28ca6\") " Nov 03 13:31:10.931348 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:31:10.915215 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14bfce12-7ae7-4866-aa46-bce174d28ca6-istiod-ca-cert" pod "14bfce12-7ae7-4866-aa46-bce174d28ca6" (UID: "14bfce12-7ae7-4866-aa46-bce174d28ca6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:31:10.932693 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:31:10.932634 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14bfce12-7ae7-4866-aa46-bce174d28ca6-credential-socket" pod "14bfce12-7ae7-4866-aa46-bce174d28ca6" (UID: "14bfce12-7ae7-4866-aa46-bce174d28ca6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:31:10.933089 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:31:10.933068 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14bfce12-7ae7-4866-aa46-bce174d28ca6-workload-certs" pod "14bfce12-7ae7-4866-aa46-bce174d28ca6" (UID: "14bfce12-7ae7-4866-aa46-bce174d28ca6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:31:10.934664 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:31:10.934639 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14bfce12-7ae7-4866-aa46-bce174d28ca6-istio-data" pod "14bfce12-7ae7-4866-aa46-bce174d28ca6" (UID: "14bfce12-7ae7-4866-aa46-bce174d28ca6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:31:10.935350 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:31:10.935327 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14bfce12-7ae7-4866-aa46-bce174d28ca6-workload-socket" pod "14bfce12-7ae7-4866-aa46-bce174d28ca6" (UID: "14bfce12-7ae7-4866-aa46-bce174d28ca6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:31:10.941311 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:31:10.938435 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/14bfce12-7ae7-4866-aa46-bce174d28ca6-istio-podinfo" pod "14bfce12-7ae7-4866-aa46-bce174d28ca6" (UID: "14bfce12-7ae7-4866-aa46-bce174d28ca6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 13:31:10.942262 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:31:10.941826 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14bfce12-7ae7-4866-aa46-bce174d28ca6-istio-token" pod "14bfce12-7ae7-4866-aa46-bce174d28ca6" (UID: "14bfce12-7ae7-4866-aa46-bce174d28ca6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:31:10.945302 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:31:10.943155 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14bfce12-7ae7-4866-aa46-bce174d28ca6-kube-api-access-gv797" pod "14bfce12-7ae7-4866-aa46-bce174d28ca6" (UID: "14bfce12-7ae7-4866-aa46-bce174d28ca6"). InnerVolumeSpecName "kube-api-access-gv797". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:31:10.945302 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:31:10.943639 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14bfce12-7ae7-4866-aa46-bce174d28ca6-istio-envoy" pod "14bfce12-7ae7-4866-aa46-bce174d28ca6" (UID: "14bfce12-7ae7-4866-aa46-bce174d28ca6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:31:11.016820 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:31:11.016772 5024 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/14bfce12-7ae7-4866-aa46-bce174d28ca6-istio-token\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:31:11.016820 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:31:11.016814 5024 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14bfce12-7ae7-4866-aa46-bce174d28ca6-istio-data\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:31:11.017050 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:31:11.016830 5024 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14bfce12-7ae7-4866-aa46-bce174d28ca6-workload-socket\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:31:11.017050 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:31:11.016846 5024 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14bfce12-7ae7-4866-aa46-bce174d28ca6-workload-certs\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:31:11.017050 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:31:11.016860 5024 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14bfce12-7ae7-4866-aa46-bce174d28ca6-credential-socket\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:31:11.017050 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:31:11.016875 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gv797\" (UniqueName: \"kubernetes.io/projected/14bfce12-7ae7-4866-aa46-bce174d28ca6-kube-api-access-gv797\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:31:11.017050 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:31:11.016889 5024 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14bfce12-7ae7-4866-aa46-bce174d28ca6-istio-envoy\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:31:11.017050 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:31:11.016903 5024 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14bfce12-7ae7-4866-aa46-bce174d28ca6-istio-podinfo\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:31:11.017050 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:31:11.016916 5024 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14bfce12-7ae7-4866-aa46-bce174d28ca6-istiod-ca-cert\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:31:11.074789 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:31:11.074748 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-08111b/simulator-w1-08111b-kserve-7f65c5867-btf5b"] Nov 03 13:31:11.156588 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:31:11.146839 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-5bffndnqx" podUID="6908d762-9f04-4823-a7f2-00a4e703b601" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 13:31:11.631950 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:31:11.631907 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-08111b/simulator-w1-08111b-kserve-7f65c5867-btf5b" event={"ID":"e8aba665-e4cc-4e67-8cdd-d09ef7542e15","Type":"ContainerStarted","Data":"448f466488f569306e0803889b69dcea09f90745995e4723f29cb7c4088320cb"} Nov 03 13:31:11.631950 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:31:11.631959 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-08111b/simulator-w1-08111b-kserve-7f65c5867-btf5b" event={"ID":"e8aba665-e4cc-4e67-8cdd-d09ef7542e15","Type":"ContainerStarted","Data":"0fa375b76abce0e4f4c39c5b380958c0f5d6b2c8ef7519ae2dc9695bd5c37dd3"} Nov 03 13:31:11.632249 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:31:11.631999 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-d66hs" Nov 03 13:31:11.669425 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:31:11.669361 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-08111b/simulator-w1-08111b-kserve-7f65c5867-btf5b" podStartSLOduration=4.6693414650000005 podStartE2EDuration="4.669341465s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:31:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:31:11.662761421 +0000 UTC m=+1351.670952725" watchObservedRunningTime="2026-11-03 13:31:11.669341465 +0000 UTC m=+1351.677532747" Nov 03 13:31:11.721731 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:31:11.721689 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-d66hs"] Nov 03 13:31:11.727567 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:31:11.727532 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-d66hs"] Nov 03 13:31:12.157757 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:31:12.157716 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="14bfce12-7ae7-4866-aa46-bce174d28ca6" path="/var/lib/kubelet/pods/14bfce12-7ae7-4866-aa46-bce174d28ca6/volumes" Nov 03 13:31:12.256227 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:31:12.256186 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-08111b/simulator-w1-08111b-kserve-7f65c5867-btf5b" Nov 03 13:31:13.103618 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:31:13.103575 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-5bffndnqx" podUID="6908d762-9f04-4823-a7f2-00a4e703b601" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 13:31:15.696564 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:31:15.696450 5024 patch_prober.go:29] interesting pod/e2e-worker-w1-08111b-openshift-default-5b6dd8787c-v2hw4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 13:31:15.696564 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:31:15.696508 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-08111b-openshift-default-5b6dd8787c-v2hw4" podUID="ca014f3a-ff92-4bb9-9a7a-8c8d3dabe486" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 13:31:18.821922 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:31:18.821879 5024 generic.go:356] "Generic (PLEG): container finished" podID="ca014f3a-ff92-4bb9-9a7a-8c8d3dabe486" containerID="5a0d74c33d3dea8b8445327a18b596aa922754e42b80e3915e17f7f73bbb73b0" exitCode=0 Nov 03 13:31:18.822882 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:31:18.822546 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-08111b-openshift-default-5b6dd8787c-v2hw4" event={"ID":"ca014f3a-ff92-4bb9-9a7a-8c8d3dabe486","Type":"ContainerDied","Data":"5a0d74c33d3dea8b8445327a18b596aa922754e42b80e3915e17f7f73bbb73b0"} Nov 03 13:31:18.824387 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:31:18.824345 5024 scope.go:122] "RemoveContainer" containerID="5a0d74c33d3dea8b8445327a18b596aa922754e42b80e3915e17f7f73bbb73b0" Nov 03 13:31:18.846904 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:31:18.846858 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:31:18.847196 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:31:18.847173 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:31:18.847356 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:31:18.847338 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:31:19.833738 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:31:19.833701 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-08111b-openshift-default-5b6dd8787c-v2hw4" event={"ID":"ca014f3a-ff92-4bb9-9a7a-8c8d3dabe486","Type":"ContainerStarted","Data":"1c031b5792bffc146b78656c47e3d1e7873d61a5ce2371294eaf19c77a57a974"} Nov 03 13:31:20.256535 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:31:20.256498 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-08111b/simulator-w1-08111b-kserve-7f65c5867-btf5b" Nov 03 13:31:20.269604 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:31:20.269569 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-08111b/simulator-w1-08111b-kserve-7f65c5867-btf5b" Nov 03 13:31:20.739734 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:31:20.739699 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-08111b-openshift-default-5b6dd8787c-v2hw4" Nov 03 13:31:20.739930 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:31:20.739905 5024 patch_prober.go:29] interesting pod/e2e-worker-w1-08111b-openshift-default-5b6dd8787c-v2hw4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" start-of-body= Nov 03 13:31:20.739980 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:31:20.739955 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-08111b-openshift-default-5b6dd8787c-v2hw4" podUID="ca014f3a-ff92-4bb9-9a7a-8c8d3dabe486" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" Nov 03 13:31:20.968135 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:31:20.968095 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-08111b/simulator-w1-08111b-kserve-7f65c5867-btf5b" Nov 03 13:31:21.756313 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:31:21.740758 5024 patch_prober.go:29] interesting pod/e2e-worker-w1-08111b-openshift-default-5b6dd8787c-v2hw4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" start-of-body= Nov 03 13:31:21.756313 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:31:21.740825 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-08111b-openshift-default-5b6dd8787c-v2hw4" podUID="ca014f3a-ff92-4bb9-9a7a-8c8d3dabe486" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" Nov 03 13:31:22.739895 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:31:22.739851 5024 patch_prober.go:29] interesting pod/e2e-worker-w1-08111b-openshift-default-5b6dd8787c-v2hw4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" start-of-body= Nov 03 13:31:22.740381 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:31:22.739923 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-08111b-openshift-default-5b6dd8787c-v2hw4" podUID="ca014f3a-ff92-4bb9-9a7a-8c8d3dabe486" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" Nov 03 13:31:23.103991 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:31:23.103949 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-5bffndnqx" podUID="6908d762-9f04-4823-a7f2-00a4e703b601" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 13:31:23.740520 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:31:23.740479 5024 patch_prober.go:29] interesting pod/e2e-worker-w1-08111b-openshift-default-5b6dd8787c-v2hw4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" start-of-body= Nov 03 13:31:23.740941 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:31:23.740575 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-08111b-openshift-default-5b6dd8787c-v2hw4" podUID="ca014f3a-ff92-4bb9-9a7a-8c8d3dabe486" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" Nov 03 13:31:24.143357 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:31:24.143320 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-08111b/premium-w1-08111b-kserve-57555d598-dx9g5"] Nov 03 13:31:24.144214 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:31:24.144190 5024 cpu_manager.go:414] "containerMap: removing container" podUID="14bfce12-7ae7-4866-aa46-bce174d28ca6" containerName="istio-proxy" Nov 03 13:31:24.144214 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:31:24.144214 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="14bfce12-7ae7-4866-aa46-bce174d28ca6" containerName="istio-proxy" Nov 03 13:31:24.144455 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:31:24.144440 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="14bfce12-7ae7-4866-aa46-bce174d28ca6" containerName="istio-proxy" Nov 03 13:31:24.147518 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:31:24.147490 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-08111b/premium-w1-08111b-kserve-57555d598-dx9g5" Nov 03 13:31:24.155503 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:31:24.155476 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-08111b/premium-w1-08111b-kserve-57555d598-dx9g5"] Nov 03 13:31:24.156335 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:31:24.156276 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-08111b\"/\"premium-w1-08111b-kserve-self-signed-certs\"" Nov 03 13:31:24.235244 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:31:24.235194 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-825gk\" (UniqueName: \"kubernetes.io/projected/e45c569a-243f-49ab-8d74-dd08cbb6127c-kube-api-access-825gk\") pod \"premium-w1-08111b-kserve-57555d598-dx9g5\" (UID: \"e45c569a-243f-49ab-8d74-dd08cbb6127c\") " pod="e2e-models-e2e-worker-w1-08111b/premium-w1-08111b-kserve-57555d598-dx9g5" Nov 03 13:31:24.235502 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:31:24.235298 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e45c569a-243f-49ab-8d74-dd08cbb6127c-tls-certs\") pod \"premium-w1-08111b-kserve-57555d598-dx9g5\" (UID: \"e45c569a-243f-49ab-8d74-dd08cbb6127c\") " pod="e2e-models-e2e-worker-w1-08111b/premium-w1-08111b-kserve-57555d598-dx9g5" Nov 03 13:31:24.235852 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:31:24.235827 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e45c569a-243f-49ab-8d74-dd08cbb6127c-dshm\") pod \"premium-w1-08111b-kserve-57555d598-dx9g5\" (UID: \"e45c569a-243f-49ab-8d74-dd08cbb6127c\") " pod="e2e-models-e2e-worker-w1-08111b/premium-w1-08111b-kserve-57555d598-dx9g5" Nov 03 13:31:24.235959 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:31:24.235938 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e45c569a-243f-49ab-8d74-dd08cbb6127c-tmp-dir\") pod \"premium-w1-08111b-kserve-57555d598-dx9g5\" (UID: \"e45c569a-243f-49ab-8d74-dd08cbb6127c\") " pod="e2e-models-e2e-worker-w1-08111b/premium-w1-08111b-kserve-57555d598-dx9g5" Nov 03 13:31:24.236029 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:31:24.236010 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e45c569a-243f-49ab-8d74-dd08cbb6127c-model-cache\") pod \"premium-w1-08111b-kserve-57555d598-dx9g5\" (UID: \"e45c569a-243f-49ab-8d74-dd08cbb6127c\") " pod="e2e-models-e2e-worker-w1-08111b/premium-w1-08111b-kserve-57555d598-dx9g5" Nov 03 13:31:24.236834 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:31:24.236063 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e45c569a-243f-49ab-8d74-dd08cbb6127c-home\") pod \"premium-w1-08111b-kserve-57555d598-dx9g5\" (UID: \"e45c569a-243f-49ab-8d74-dd08cbb6127c\") " pod="e2e-models-e2e-worker-w1-08111b/premium-w1-08111b-kserve-57555d598-dx9g5" Nov 03 13:31:24.350728 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:31:24.348461 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e45c569a-243f-49ab-8d74-dd08cbb6127c-model-cache\") pod \"premium-w1-08111b-kserve-57555d598-dx9g5\" (UID: \"e45c569a-243f-49ab-8d74-dd08cbb6127c\") " pod="e2e-models-e2e-worker-w1-08111b/premium-w1-08111b-kserve-57555d598-dx9g5" Nov 03 13:31:24.350728 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:31:24.348548 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e45c569a-243f-49ab-8d74-dd08cbb6127c-home\") pod \"premium-w1-08111b-kserve-57555d598-dx9g5\" (UID: \"e45c569a-243f-49ab-8d74-dd08cbb6127c\") " pod="e2e-models-e2e-worker-w1-08111b/premium-w1-08111b-kserve-57555d598-dx9g5" Nov 03 13:31:24.350728 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:31:24.348711 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-825gk\" (UniqueName: \"kubernetes.io/projected/e45c569a-243f-49ab-8d74-dd08cbb6127c-kube-api-access-825gk\") pod \"premium-w1-08111b-kserve-57555d598-dx9g5\" (UID: \"e45c569a-243f-49ab-8d74-dd08cbb6127c\") " pod="e2e-models-e2e-worker-w1-08111b/premium-w1-08111b-kserve-57555d598-dx9g5" Nov 03 13:31:24.350728 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:31:24.348767 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e45c569a-243f-49ab-8d74-dd08cbb6127c-tls-certs\") pod \"premium-w1-08111b-kserve-57555d598-dx9g5\" (UID: \"e45c569a-243f-49ab-8d74-dd08cbb6127c\") " pod="e2e-models-e2e-worker-w1-08111b/premium-w1-08111b-kserve-57555d598-dx9g5" Nov 03 13:31:24.350728 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:31:24.348911 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e45c569a-243f-49ab-8d74-dd08cbb6127c-dshm\") pod \"premium-w1-08111b-kserve-57555d598-dx9g5\" (UID: \"e45c569a-243f-49ab-8d74-dd08cbb6127c\") " pod="e2e-models-e2e-worker-w1-08111b/premium-w1-08111b-kserve-57555d598-dx9g5" Nov 03 13:31:24.350728 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:31:24.349012 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e45c569a-243f-49ab-8d74-dd08cbb6127c-tmp-dir\") pod \"premium-w1-08111b-kserve-57555d598-dx9g5\" (UID: \"e45c569a-243f-49ab-8d74-dd08cbb6127c\") " pod="e2e-models-e2e-worker-w1-08111b/premium-w1-08111b-kserve-57555d598-dx9g5" Nov 03 13:31:24.350728 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:31:24.349560 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e45c569a-243f-49ab-8d74-dd08cbb6127c-tmp-dir\") pod \"premium-w1-08111b-kserve-57555d598-dx9g5\" (UID: \"e45c569a-243f-49ab-8d74-dd08cbb6127c\") " pod="e2e-models-e2e-worker-w1-08111b/premium-w1-08111b-kserve-57555d598-dx9g5" Nov 03 13:31:24.350728 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:31:24.349813 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e45c569a-243f-49ab-8d74-dd08cbb6127c-model-cache\") pod \"premium-w1-08111b-kserve-57555d598-dx9g5\" (UID: \"e45c569a-243f-49ab-8d74-dd08cbb6127c\") " pod="e2e-models-e2e-worker-w1-08111b/premium-w1-08111b-kserve-57555d598-dx9g5" Nov 03 13:31:24.350728 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:31:24.350037 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e45c569a-243f-49ab-8d74-dd08cbb6127c-home\") pod \"premium-w1-08111b-kserve-57555d598-dx9g5\" (UID: \"e45c569a-243f-49ab-8d74-dd08cbb6127c\") " pod="e2e-models-e2e-worker-w1-08111b/premium-w1-08111b-kserve-57555d598-dx9g5" Nov 03 13:31:24.371715 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:31:24.371603 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e45c569a-243f-49ab-8d74-dd08cbb6127c-tls-certs\") pod \"premium-w1-08111b-kserve-57555d598-dx9g5\" (UID: \"e45c569a-243f-49ab-8d74-dd08cbb6127c\") " pod="e2e-models-e2e-worker-w1-08111b/premium-w1-08111b-kserve-57555d598-dx9g5" Nov 03 13:31:24.398521 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:31:24.395274 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e45c569a-243f-49ab-8d74-dd08cbb6127c-dshm\") pod \"premium-w1-08111b-kserve-57555d598-dx9g5\" (UID: \"e45c569a-243f-49ab-8d74-dd08cbb6127c\") " pod="e2e-models-e2e-worker-w1-08111b/premium-w1-08111b-kserve-57555d598-dx9g5" Nov 03 13:31:24.416304 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:31:24.406915 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-825gk\" (UniqueName: \"kubernetes.io/projected/e45c569a-243f-49ab-8d74-dd08cbb6127c-kube-api-access-825gk\") pod \"premium-w1-08111b-kserve-57555d598-dx9g5\" (UID: \"e45c569a-243f-49ab-8d74-dd08cbb6127c\") " pod="e2e-models-e2e-worker-w1-08111b/premium-w1-08111b-kserve-57555d598-dx9g5" Nov 03 13:31:24.470371 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:31:24.467603 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-08111b/premium-w1-08111b-kserve-57555d598-dx9g5" Nov 03 13:31:24.753367 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:31:24.750784 5024 patch_prober.go:29] interesting pod/e2e-worker-w1-08111b-openshift-default-5b6dd8787c-v2hw4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" start-of-body= Nov 03 13:31:24.753367 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:31:24.750848 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-08111b-openshift-default-5b6dd8787c-v2hw4" podUID="ca014f3a-ff92-4bb9-9a7a-8c8d3dabe486" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" Nov 03 13:31:25.040341 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:31:25.040306 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-08111b/premium-w1-08111b-kserve-57555d598-dx9g5"] Nov 03 13:31:25.041823 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:31:25.041774 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode45c569a_243f_49ab_8d74_dd08cbb6127c.slice/crio-9ef8259972cb6062da22404b19f33369b60a1e1e9e188b5172dfed0946980748 WatchSource:0}: Error finding container 9ef8259972cb6062da22404b19f33369b60a1e1e9e188b5172dfed0946980748: Status 404 returned error can't find the container with id 9ef8259972cb6062da22404b19f33369b60a1e1e9e188b5172dfed0946980748 Nov 03 13:31:25.744366 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:31:25.744334 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-08111b-openshift-default-5b6dd8787c-v2hw4" Nov 03 13:31:25.744628 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:31:25.744598 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-08111b-openshift-default-5b6dd8787c-v2hw4" Nov 03 13:31:25.745376 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:31:25.745360 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-08111b-openshift-default-5b6dd8787c-v2hw4" Nov 03 13:31:26.033501 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:31:26.033406 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-08111b/premium-w1-08111b-kserve-57555d598-dx9g5" event={"ID":"e45c569a-243f-49ab-8d74-dd08cbb6127c","Type":"ContainerStarted","Data":"f260365e7cd791f7ee4cb0c4526ddb1047ed8821c14214cef7e33c1fae97190f"} Nov 03 13:31:26.033976 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:31:26.033948 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-08111b/premium-w1-08111b-kserve-57555d598-dx9g5" event={"ID":"e45c569a-243f-49ab-8d74-dd08cbb6127c","Type":"ContainerStarted","Data":"9ef8259972cb6062da22404b19f33369b60a1e1e9e188b5172dfed0946980748"} Nov 03 13:31:26.056492 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:31:26.056435 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-08111b/premium-w1-08111b-kserve-57555d598-dx9g5" podStartSLOduration=2.056418676 podStartE2EDuration="2.056418676s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:31:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:31:26.053353052 +0000 UTC m=+1366.061544322" watchObservedRunningTime="2026-11-03 13:31:26.056418676 +0000 UTC m=+1366.064609979" Nov 03 13:31:26.468636 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:31:26.468594 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-08111b/premium-w1-08111b-kserve-57555d598-dx9g5" Nov 03 13:31:33.121896 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:31:33.120118 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-5bffndnqx" podUID="6908d762-9f04-4823-a7f2-00a4e703b601" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 13:31:34.471206 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:31:34.471165 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-08111b/premium-w1-08111b-kserve-57555d598-dx9g5" Nov 03 13:31:35.474219 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:31:35.474171 5024 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-models-e2e-worker-w1-08111b/premium-w1-08111b-kserve-57555d598-dx9g5" podUID="e45c569a-243f-49ab-8d74-dd08cbb6127c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.134:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 13:31:37.030493 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:31:37.027306 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-extmod-shr-user-805818/e2e-usage-sim-ea7ebc-85b488466-wxqgk"] Nov 03 13:31:37.030493 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:31:37.027589 5024 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-extmod-shr-user-805818/e2e-usage-sim-ea7ebc-85b488466-wxqgk" podUID="b0aac0fd-ce21-463d-b6f8-4830f81582e2" containerName="sim" containerID="cri-o://3681e931878f9d104470811577ceaab09b1b6d7103af06ea4b59f54d8d3744ec" gracePeriod=30 Nov 03 13:31:37.802375 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:31:37.802121 5024 generic.go:356] "Generic (PLEG): container finished" podID="b0aac0fd-ce21-463d-b6f8-4830f81582e2" containerID="3681e931878f9d104470811577ceaab09b1b6d7103af06ea4b59f54d8d3744ec" exitCode=0 Nov 03 13:31:37.802844 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:31:37.802818 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-extmod-shr-user-805818/e2e-usage-sim-ea7ebc-85b488466-wxqgk" event={"ID":"b0aac0fd-ce21-463d-b6f8-4830f81582e2","Type":"ContainerDied","Data":"3681e931878f9d104470811577ceaab09b1b6d7103af06ea4b59f54d8d3744ec"} Nov 03 13:31:37.974893 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:31:37.974864 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-extmod-shr-user-805818/e2e-usage-sim-ea7ebc-85b488466-wxqgk" Nov 03 13:31:38.065508 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:31:38.065354 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b0aac0fd-ce21-463d-b6f8-4830f81582e2-tls\" (UniqueName: \"kubernetes.io/secret/b0aac0fd-ce21-463d-b6f8-4830f81582e2-tls\") pod \"b0aac0fd-ce21-463d-b6f8-4830f81582e2\" (UID: \"b0aac0fd-ce21-463d-b6f8-4830f81582e2\") " Nov 03 13:31:38.065935 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:31:38.065737 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0aac0fd-ce21-463d-b6f8-4830f81582e2-kube-api-access-dj5v6\" (UniqueName: \"kubernetes.io/projected/b0aac0fd-ce21-463d-b6f8-4830f81582e2-kube-api-access-dj5v6\") pod \"b0aac0fd-ce21-463d-b6f8-4830f81582e2\" (UID: \"b0aac0fd-ce21-463d-b6f8-4830f81582e2\") " Nov 03 13:31:38.068419 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:31:38.068382 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0aac0fd-ce21-463d-b6f8-4830f81582e2-kube-api-access-dj5v6" pod "b0aac0fd-ce21-463d-b6f8-4830f81582e2" (UID: "b0aac0fd-ce21-463d-b6f8-4830f81582e2"). InnerVolumeSpecName "kube-api-access-dj5v6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:31:38.068559 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:31:38.068439 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b0aac0fd-ce21-463d-b6f8-4830f81582e2-tls" pod "b0aac0fd-ce21-463d-b6f8-4830f81582e2" (UID: "b0aac0fd-ce21-463d-b6f8-4830f81582e2"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 13:31:38.188686 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:31:38.183390 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dj5v6\" (UniqueName: \"kubernetes.io/projected/b0aac0fd-ce21-463d-b6f8-4830f81582e2-kube-api-access-dj5v6\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:31:38.188686 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:31:38.183438 5024 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/b0aac0fd-ce21-463d-b6f8-4830f81582e2-tls\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:31:38.854257 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:31:38.854215 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-extmod-shr-user-805818/e2e-usage-sim-ea7ebc-85b488466-wxqgk" event={"ID":"b0aac0fd-ce21-463d-b6f8-4830f81582e2","Type":"ContainerDied","Data":"9d51d381262f2148ea0b5c4bf3a3502de0c81216b393509280f89040340db28a"} Nov 03 13:31:38.854556 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:31:38.854536 5024 scope.go:122] "RemoveContainer" containerID="3681e931878f9d104470811577ceaab09b1b6d7103af06ea4b59f54d8d3744ec" Nov 03 13:31:38.854842 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:31:38.854825 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-extmod-shr-user-805818/e2e-usage-sim-ea7ebc-85b488466-wxqgk" Nov 03 13:31:38.903467 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:31:38.903401 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-extmod-shr-user-805818/e2e-usage-sim-ea7ebc-85b488466-wxqgk"] Nov 03 13:31:38.920297 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:31:38.920234 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-extmod-shr-user-805818/e2e-usage-sim-ea7ebc-85b488466-wxqgk"] Nov 03 13:31:39.509309 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:31:39.506384 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-1a18c8/payload-processing-external-model-e2e-extmod-b-1a18c8-8676lrgff"] Nov 03 13:31:39.509309 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:31:39.507669 5024 cpu_manager.go:414] "containerMap: removing container" podUID="b0aac0fd-ce21-463d-b6f8-4830f81582e2" containerName="sim" Nov 03 13:31:39.509309 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:31:39.507695 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0aac0fd-ce21-463d-b6f8-4830f81582e2" containerName="sim" Nov 03 13:31:39.509309 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:31:39.507903 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0aac0fd-ce21-463d-b6f8-4830f81582e2" containerName="sim" Nov 03 13:31:39.529312 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:31:39.525133 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-1a18c8/payload-processing-external-model-e2e-extmod-b-1a18c8-8676lrgff"] Nov 03 13:31:39.529312 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:31:39.525234 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-1a18c8/payload-processing-external-model-e2e-extmod-b-1a18c8-8676lrgff" Nov 03 13:31:39.530694 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:31:39.530672 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-1a18c8\"/\"openshift-service-ca.crt\"" Nov 03 13:31:39.531332 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:31:39.531306 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-1a18c8\"/\"agc-cred-31f9f21f7966\"" Nov 03 13:31:39.533409 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:31:39.533361 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-1a18c8\"/\"agc-cred-de83083e156c\"" Nov 03 13:31:39.534608 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:31:39.533498 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-1a18c8\"/\"payload-processing-external-model-e2e--d832c86e-dockercfg-sq45c\"" Nov 03 13:31:39.534608 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:31:39.533951 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-1a18c8\"/\"payload-processing-external-model-plugins-e2e-extmod-b-1a18c8\"" Nov 03 13:31:39.536837 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:31:39.536817 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-1a18c8\"/\"kube-root-ca.crt\"" Nov 03 13:31:39.536945 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:31:39.536863 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-1a18c8\"/\"routing-overlay\"" Nov 03 13:31:39.544891 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:31:39.543798 5024 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d7mwz" containerName="extractor" Nov 03 13:31:39.641117 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:31:39.641081 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/fa6220b6-0c80-431e-8689-d6e63e5653e0-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-1a18c8-8676lrgff\" (UID: \"fa6220b6-0c80-431e-8689-d6e63e5653e0\") " pod="ai-tenant-e2e-extmod-b-1a18c8/payload-processing-external-model-e2e-extmod-b-1a18c8-8676lrgff" Nov 03 13:31:39.641345 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:31:39.641147 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa6220b6-0c80-431e-8689-d6e63e5653e0-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-1a18c8-8676lrgff\" (UID: \"fa6220b6-0c80-431e-8689-d6e63e5653e0\") " pod="ai-tenant-e2e-extmod-b-1a18c8/payload-processing-external-model-e2e-extmod-b-1a18c8-8676lrgff" Nov 03 13:31:39.641426 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:31:39.641359 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/fa6220b6-0c80-431e-8689-d6e63e5653e0-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-1a18c8-8676lrgff\" (UID: \"fa6220b6-0c80-431e-8689-d6e63e5653e0\") " pod="ai-tenant-e2e-extmod-b-1a18c8/payload-processing-external-model-e2e-extmod-b-1a18c8-8676lrgff" Nov 03 13:31:39.641603 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:31:39.641576 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t6hlj\" (UniqueName: \"kubernetes.io/projected/fa6220b6-0c80-431e-8689-d6e63e5653e0-kube-api-access-t6hlj\") pod \"payload-processing-external-model-e2e-extmod-b-1a18c8-8676lrgff\" (UID: \"fa6220b6-0c80-431e-8689-d6e63e5653e0\") " pod="ai-tenant-e2e-extmod-b-1a18c8/payload-processing-external-model-e2e-extmod-b-1a18c8-8676lrgff" Nov 03 13:31:39.775783 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:31:39.775007 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6hlj\" (UniqueName: \"kubernetes.io/projected/fa6220b6-0c80-431e-8689-d6e63e5653e0-kube-api-access-t6hlj\") pod \"payload-processing-external-model-e2e-extmod-b-1a18c8-8676lrgff\" (UID: \"fa6220b6-0c80-431e-8689-d6e63e5653e0\") " pod="ai-tenant-e2e-extmod-b-1a18c8/payload-processing-external-model-e2e-extmod-b-1a18c8-8676lrgff" Nov 03 13:31:39.775783 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:31:39.775386 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/fa6220b6-0c80-431e-8689-d6e63e5653e0-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-1a18c8-8676lrgff\" (UID: \"fa6220b6-0c80-431e-8689-d6e63e5653e0\") " pod="ai-tenant-e2e-extmod-b-1a18c8/payload-processing-external-model-e2e-extmod-b-1a18c8-8676lrgff" Nov 03 13:31:39.800309 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:31:39.790609 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa6220b6-0c80-431e-8689-d6e63e5653e0-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-1a18c8-8676lrgff\" (UID: \"fa6220b6-0c80-431e-8689-d6e63e5653e0\") " pod="ai-tenant-e2e-extmod-b-1a18c8/payload-processing-external-model-e2e-extmod-b-1a18c8-8676lrgff" Nov 03 13:31:39.800309 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:31:39.790961 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/fa6220b6-0c80-431e-8689-d6e63e5653e0-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-1a18c8-8676lrgff\" (UID: \"fa6220b6-0c80-431e-8689-d6e63e5653e0\") " pod="ai-tenant-e2e-extmod-b-1a18c8/payload-processing-external-model-e2e-extmod-b-1a18c8-8676lrgff" Nov 03 13:31:39.800309 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:31:39.792023 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/fa6220b6-0c80-431e-8689-d6e63e5653e0-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-1a18c8-8676lrgff\" (UID: \"fa6220b6-0c80-431e-8689-d6e63e5653e0\") " pod="ai-tenant-e2e-extmod-b-1a18c8/payload-processing-external-model-e2e-extmod-b-1a18c8-8676lrgff" Nov 03 13:31:39.800309 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:31:39.800216 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa6220b6-0c80-431e-8689-d6e63e5653e0-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-1a18c8-8676lrgff\" (UID: \"fa6220b6-0c80-431e-8689-d6e63e5653e0\") " pod="ai-tenant-e2e-extmod-b-1a18c8/payload-processing-external-model-e2e-extmod-b-1a18c8-8676lrgff" Nov 03 13:31:39.845309 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:31:39.842827 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t6hlj\" (UniqueName: \"kubernetes.io/projected/fa6220b6-0c80-431e-8689-d6e63e5653e0-kube-api-access-t6hlj\") pod \"payload-processing-external-model-e2e-extmod-b-1a18c8-8676lrgff\" (UID: \"fa6220b6-0c80-431e-8689-d6e63e5653e0\") " pod="ai-tenant-e2e-extmod-b-1a18c8/payload-processing-external-model-e2e-extmod-b-1a18c8-8676lrgff" Nov 03 13:31:39.845309 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:31:39.843597 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/fa6220b6-0c80-431e-8689-d6e63e5653e0-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-1a18c8-8676lrgff\" (UID: \"fa6220b6-0c80-431e-8689-d6e63e5653e0\") " pod="ai-tenant-e2e-extmod-b-1a18c8/payload-processing-external-model-e2e-extmod-b-1a18c8-8676lrgff" Nov 03 13:31:39.889359 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:31:39.875945 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-1a18c8/payload-processing-external-model-e2e-extmod-b-1a18c8-8676lrgff" Nov 03 13:31:40.332430 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:31:40.330827 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b0aac0fd-ce21-463d-b6f8-4830f81582e2" path="/var/lib/kubelet/pods/b0aac0fd-ce21-463d-b6f8-4830f81582e2/volumes" Nov 03 13:31:40.332430 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:31:40.331191 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-fdc7999b7-xb26w"] Nov 03 13:31:40.339436 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:31:40.339404 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-fdc7999b7-xb26w"] Nov 03 13:31:40.339755 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:31:40.339664 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-fdc7999b7-xb26w" Nov 03 13:31:40.396824 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:31:40.394052 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-vx42v\"" Nov 03 13:31:40.396824 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:31:40.394152 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\"" Nov 03 13:31:40.396824 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:31:40.394213 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\"" Nov 03 13:31:40.396824 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:31:40.394329 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\"" Nov 03 13:31:40.396824 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:31:40.394418 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\"" Nov 03 13:31:40.396824 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:31:40.394491 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-1d111babc549\"" Nov 03 13:31:40.534362 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:31:40.533939 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/fa2f0750-c048-4c12-ba76-1d54f2abb5f7-provider-credentials\") pod \"payload-processing-external-model-fdc7999b7-xb26w\" (UID: \"fa2f0750-c048-4c12-ba76-1d54f2abb5f7\") " pod="models-as-a-service/payload-processing-external-model-fdc7999b7-xb26w" Nov 03 13:31:40.534362 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:31:40.534126 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/fa2f0750-c048-4c12-ba76-1d54f2abb5f7-routing-overlay-volume\") pod \"payload-processing-external-model-fdc7999b7-xb26w\" (UID: \"fa2f0750-c048-4c12-ba76-1d54f2abb5f7\") " pod="models-as-a-service/payload-processing-external-model-fdc7999b7-xb26w" Nov 03 13:31:40.534362 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:31:40.534333 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa2f0750-c048-4c12-ba76-1d54f2abb5f7-plugins-config-volume\") pod \"payload-processing-external-model-fdc7999b7-xb26w\" (UID: \"fa2f0750-c048-4c12-ba76-1d54f2abb5f7\") " pod="models-as-a-service/payload-processing-external-model-fdc7999b7-xb26w" Nov 03 13:31:40.534979 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:31:40.534525 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-874j8\" (UniqueName: \"kubernetes.io/projected/fa2f0750-c048-4c12-ba76-1d54f2abb5f7-kube-api-access-874j8\") pod \"payload-processing-external-model-fdc7999b7-xb26w\" (UID: \"fa2f0750-c048-4c12-ba76-1d54f2abb5f7\") " pod="models-as-a-service/payload-processing-external-model-fdc7999b7-xb26w" Nov 03 13:31:40.588889 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:31:40.588804 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-1a18c8/payload-processing-external-model-e2e-extmod-b-1a18c8-8676lrgff"] Nov 03 13:31:40.594811 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:31:40.594774 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa6220b6_0c80_431e_8689_d6e63e5653e0.slice/crio-5749e0ae13e695cdf5f3cc68c9bf4cf279f018bb5a4c05b85cbc0046023d3b54 WatchSource:0}: Error finding container 5749e0ae13e695cdf5f3cc68c9bf4cf279f018bb5a4c05b85cbc0046023d3b54: Status 404 returned error can't find the container with id 5749e0ae13e695cdf5f3cc68c9bf4cf279f018bb5a4c05b85cbc0046023d3b54 Nov 03 13:31:40.635968 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:31:40.635938 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/fa2f0750-c048-4c12-ba76-1d54f2abb5f7-routing-overlay-volume\") pod \"payload-processing-external-model-fdc7999b7-xb26w\" (UID: \"fa2f0750-c048-4c12-ba76-1d54f2abb5f7\") " pod="models-as-a-service/payload-processing-external-model-fdc7999b7-xb26w" Nov 03 13:31:40.636107 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:31:40.636044 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa2f0750-c048-4c12-ba76-1d54f2abb5f7-plugins-config-volume\") pod \"payload-processing-external-model-fdc7999b7-xb26w\" (UID: \"fa2f0750-c048-4c12-ba76-1d54f2abb5f7\") " pod="models-as-a-service/payload-processing-external-model-fdc7999b7-xb26w" Nov 03 13:31:40.636161 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:31:40.636122 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-874j8\" (UniqueName: \"kubernetes.io/projected/fa2f0750-c048-4c12-ba76-1d54f2abb5f7-kube-api-access-874j8\") pod \"payload-processing-external-model-fdc7999b7-xb26w\" (UID: \"fa2f0750-c048-4c12-ba76-1d54f2abb5f7\") " pod="models-as-a-service/payload-processing-external-model-fdc7999b7-xb26w" Nov 03 13:31:40.636244 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:31:40.636229 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/fa2f0750-c048-4c12-ba76-1d54f2abb5f7-provider-credentials\") pod \"payload-processing-external-model-fdc7999b7-xb26w\" (UID: \"fa2f0750-c048-4c12-ba76-1d54f2abb5f7\") " pod="models-as-a-service/payload-processing-external-model-fdc7999b7-xb26w" Nov 03 13:31:40.636684 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:31:40.636661 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/fa2f0750-c048-4c12-ba76-1d54f2abb5f7-routing-overlay-volume\") pod \"payload-processing-external-model-fdc7999b7-xb26w\" (UID: \"fa2f0750-c048-4c12-ba76-1d54f2abb5f7\") " pod="models-as-a-service/payload-processing-external-model-fdc7999b7-xb26w" Nov 03 13:31:40.637040 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:31:40.637021 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa2f0750-c048-4c12-ba76-1d54f2abb5f7-plugins-config-volume\") pod \"payload-processing-external-model-fdc7999b7-xb26w\" (UID: \"fa2f0750-c048-4c12-ba76-1d54f2abb5f7\") " pod="models-as-a-service/payload-processing-external-model-fdc7999b7-xb26w" Nov 03 13:31:40.638712 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:31:40.638689 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/fa2f0750-c048-4c12-ba76-1d54f2abb5f7-provider-credentials\") pod \"payload-processing-external-model-fdc7999b7-xb26w\" (UID: \"fa2f0750-c048-4c12-ba76-1d54f2abb5f7\") " pod="models-as-a-service/payload-processing-external-model-fdc7999b7-xb26w" Nov 03 13:31:40.645591 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:31:40.645565 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-874j8\" (UniqueName: \"kubernetes.io/projected/fa2f0750-c048-4c12-ba76-1d54f2abb5f7-kube-api-access-874j8\") pod \"payload-processing-external-model-fdc7999b7-xb26w\" (UID: \"fa2f0750-c048-4c12-ba76-1d54f2abb5f7\") " pod="models-as-a-service/payload-processing-external-model-fdc7999b7-xb26w" Nov 03 13:31:40.667887 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:31:40.667858 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-fdc7999b7-xb26w" Nov 03 13:31:40.720178 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:31:40.720139 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-ef1693/payload-processing-external-model-e2e-extmod-a-ef1693-5578zqwf8"] Nov 03 13:31:40.725013 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:31:40.724985 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-ef1693/payload-processing-external-model-e2e-extmod-a-ef1693-5578zqwf8" Nov 03 13:31:40.732709 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:31:40.732670 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-ef1693\"/\"kube-root-ca.crt\"" Nov 03 13:31:40.733535 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:31:40.733258 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-ef1693\"/\"payload-processing-external-model-e2e--393fd9ed-dockercfg-fnc52\"" Nov 03 13:31:40.733535 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:31:40.733355 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-ef1693\"/\"openshift-service-ca.crt\"" Nov 03 13:31:40.733740 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:31:40.733377 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-ef1693\"/\"agc-cred-4d65870da959\"" Nov 03 13:31:40.733740 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:31:40.733416 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-ef1693\"/\"routing-overlay\"" Nov 03 13:31:40.733740 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:31:40.733474 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-ef1693\"/\"payload-processing-external-model-plugins-e2e-extmod-a-ef1693\"" Nov 03 13:31:40.733740 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:31:40.733514 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-ef1693\"/\"agc-cred-5e1bd9e90045\"" Nov 03 13:31:40.739799 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:31:40.739771 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-ef1693/payload-processing-external-model-e2e-extmod-a-ef1693-5578zqwf8"] Nov 03 13:31:40.842683 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:31:40.840377 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26c02b21-f8e5-441b-96db-6512b211606e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-ef1693-5578zqwf8\" (UID: \"26c02b21-f8e5-441b-96db-6512b211606e\") " pod="ai-tenant-e2e-extmod-a-ef1693/payload-processing-external-model-e2e-extmod-a-ef1693-5578zqwf8" Nov 03 13:31:40.842683 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:31:40.840585 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w9k47\" (UniqueName: \"kubernetes.io/projected/26c02b21-f8e5-441b-96db-6512b211606e-kube-api-access-w9k47\") pod \"payload-processing-external-model-e2e-extmod-a-ef1693-5578zqwf8\" (UID: \"26c02b21-f8e5-441b-96db-6512b211606e\") " pod="ai-tenant-e2e-extmod-a-ef1693/payload-processing-external-model-e2e-extmod-a-ef1693-5578zqwf8" Nov 03 13:31:40.842683 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:31:40.840810 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/26c02b21-f8e5-441b-96db-6512b211606e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-ef1693-5578zqwf8\" (UID: \"26c02b21-f8e5-441b-96db-6512b211606e\") " pod="ai-tenant-e2e-extmod-a-ef1693/payload-processing-external-model-e2e-extmod-a-ef1693-5578zqwf8" Nov 03 13:31:40.842683 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:31:40.840970 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/26c02b21-f8e5-441b-96db-6512b211606e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-ef1693-5578zqwf8\" (UID: \"26c02b21-f8e5-441b-96db-6512b211606e\") " pod="ai-tenant-e2e-extmod-a-ef1693/payload-processing-external-model-e2e-extmod-a-ef1693-5578zqwf8" Nov 03 13:31:40.942229 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:31:40.942197 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/26c02b21-f8e5-441b-96db-6512b211606e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-ef1693-5578zqwf8\" (UID: \"26c02b21-f8e5-441b-96db-6512b211606e\") " pod="ai-tenant-e2e-extmod-a-ef1693/payload-processing-external-model-e2e-extmod-a-ef1693-5578zqwf8" Nov 03 13:31:40.942457 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:31:40.942314 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/26c02b21-f8e5-441b-96db-6512b211606e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-ef1693-5578zqwf8\" (UID: \"26c02b21-f8e5-441b-96db-6512b211606e\") " pod="ai-tenant-e2e-extmod-a-ef1693/payload-processing-external-model-e2e-extmod-a-ef1693-5578zqwf8" Nov 03 13:31:40.942457 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:31:40.942388 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26c02b21-f8e5-441b-96db-6512b211606e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-ef1693-5578zqwf8\" (UID: \"26c02b21-f8e5-441b-96db-6512b211606e\") " pod="ai-tenant-e2e-extmod-a-ef1693/payload-processing-external-model-e2e-extmod-a-ef1693-5578zqwf8" Nov 03 13:31:40.942565 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:31:40.942546 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w9k47\" (UniqueName: \"kubernetes.io/projected/26c02b21-f8e5-441b-96db-6512b211606e-kube-api-access-w9k47\") pod \"payload-processing-external-model-e2e-extmod-a-ef1693-5578zqwf8\" (UID: \"26c02b21-f8e5-441b-96db-6512b211606e\") " pod="ai-tenant-e2e-extmod-a-ef1693/payload-processing-external-model-e2e-extmod-a-ef1693-5578zqwf8" Nov 03 13:31:40.943246 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:31:40.943191 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/26c02b21-f8e5-441b-96db-6512b211606e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-ef1693-5578zqwf8\" (UID: \"26c02b21-f8e5-441b-96db-6512b211606e\") " pod="ai-tenant-e2e-extmod-a-ef1693/payload-processing-external-model-e2e-extmod-a-ef1693-5578zqwf8" Nov 03 13:31:40.943409 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:31:40.943356 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26c02b21-f8e5-441b-96db-6512b211606e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-ef1693-5578zqwf8\" (UID: \"26c02b21-f8e5-441b-96db-6512b211606e\") " pod="ai-tenant-e2e-extmod-a-ef1693/payload-processing-external-model-e2e-extmod-a-ef1693-5578zqwf8" Nov 03 13:31:40.945753 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:31:40.945726 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/26c02b21-f8e5-441b-96db-6512b211606e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-ef1693-5578zqwf8\" (UID: \"26c02b21-f8e5-441b-96db-6512b211606e\") " pod="ai-tenant-e2e-extmod-a-ef1693/payload-processing-external-model-e2e-extmod-a-ef1693-5578zqwf8" Nov 03 13:31:40.953531 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:31:40.953502 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w9k47\" (UniqueName: \"kubernetes.io/projected/26c02b21-f8e5-441b-96db-6512b211606e-kube-api-access-w9k47\") pod \"payload-processing-external-model-e2e-extmod-a-ef1693-5578zqwf8\" (UID: \"26c02b21-f8e5-441b-96db-6512b211606e\") " pod="ai-tenant-e2e-extmod-a-ef1693/payload-processing-external-model-e2e-extmod-a-ef1693-5578zqwf8" Nov 03 13:31:40.968621 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:31:40.968591 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-fdc7999b7-xb26w"] Nov 03 13:31:40.969702 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:31:40.969676 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa2f0750_c048_4c12_ba76_1d54f2abb5f7.slice/crio-dbdbd7bcfbb826f67d902cd256399f856b33b742eeb6440221a75ac1da200e78 WatchSource:0}: Error finding container dbdbd7bcfbb826f67d902cd256399f856b33b742eeb6440221a75ac1da200e78: Status 404 returned error can't find the container with id dbdbd7bcfbb826f67d902cd256399f856b33b742eeb6440221a75ac1da200e78 Nov 03 13:31:41.048147 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:31:41.048121 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-ef1693/payload-processing-external-model-e2e-extmod-a-ef1693-5578zqwf8" Nov 03 13:31:41.287628 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:31:41.281057 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-fdc7999b7-xb26w" event={"ID":"fa2f0750-c048-4c12-ba76-1d54f2abb5f7","Type":"ContainerStarted","Data":"7d2c88da1906c77a344ef117f9fee5acbf1da79d9d7e8b2a184496a60068a991"} Nov 03 13:31:41.287628 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:31:41.281183 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-fdc7999b7-xb26w" event={"ID":"fa2f0750-c048-4c12-ba76-1d54f2abb5f7","Type":"ContainerStarted","Data":"dbdbd7bcfbb826f67d902cd256399f856b33b742eeb6440221a75ac1da200e78"} Nov 03 13:31:41.287628 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:31:41.282797 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-fdc7999b7-xb26w" Nov 03 13:31:41.289002 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:31:41.288965 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-1a18c8/payload-processing-external-model-e2e-extmod-b-1a18c8-8676lrgff" event={"ID":"fa6220b6-0c80-431e-8689-d6e63e5653e0","Type":"ContainerStarted","Data":"19210be1a553e292b693b44829623bca3a646678f0d82141fbb2d8b29c9a2013"} Nov 03 13:31:41.289140 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:31:41.289030 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-1a18c8/payload-processing-external-model-e2e-extmod-b-1a18c8-8676lrgff" event={"ID":"fa6220b6-0c80-431e-8689-d6e63e5653e0","Type":"ContainerStarted","Data":"5749e0ae13e695cdf5f3cc68c9bf4cf279f018bb5a4c05b85cbc0046023d3b54"} Nov 03 13:31:41.290016 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:31:41.289988 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-1a18c8/payload-processing-external-model-e2e-extmod-b-1a18c8-8676lrgff" Nov 03 13:31:41.320425 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:31:41.320370 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-fdc7999b7-xb26w" podStartSLOduration=1.3203548010000001 podStartE2EDuration="1.320354801s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:31:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:31:41.317415649 +0000 UTC m=+1381.325606933" watchObservedRunningTime="2026-11-03 13:31:41.320354801 +0000 UTC m=+1381.328546080" Nov 03 13:31:41.348741 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:31:41.348678 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-1a18c8/payload-processing-external-model-e2e-extmod-b-1a18c8-8676lrgff" podStartSLOduration=2.348660007 podStartE2EDuration="2.348660007s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:31:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:31:41.345011958 +0000 UTC m=+1381.353203235" watchObservedRunningTime="2026-11-03 13:31:41.348660007 +0000 UTC m=+1381.356851285" Nov 03 13:31:41.453807 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:31:41.453772 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-ef1693/payload-processing-external-model-e2e-extmod-a-ef1693-5578zqwf8"] Nov 03 13:31:41.454145 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:31:41.454111 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26c02b21_f8e5_441b_96db_6512b211606e.slice/crio-7a3bad7eeefbdcfc73955b404eb01d21f93bb1661e7d36869ff33573e7a89aa9 WatchSource:0}: Error finding container 7a3bad7eeefbdcfc73955b404eb01d21f93bb1661e7d36869ff33573e7a89aa9: Status 404 returned error can't find the container with id 7a3bad7eeefbdcfc73955b404eb01d21f93bb1661e7d36869ff33573e7a89aa9 Nov 03 13:31:42.300034 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:31:42.299989 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-ef1693/payload-processing-external-model-e2e-extmod-a-ef1693-5578zqwf8" event={"ID":"26c02b21-f8e5-441b-96db-6512b211606e","Type":"ContainerStarted","Data":"d0b068305cc8569b5405a2459cce581924aa2cf2031cc1160bb1da48c29e35a0"} Nov 03 13:31:42.300503 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:31:42.300059 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-ef1693/payload-processing-external-model-e2e-extmod-a-ef1693-5578zqwf8" event={"ID":"26c02b21-f8e5-441b-96db-6512b211606e","Type":"ContainerStarted","Data":"7a3bad7eeefbdcfc73955b404eb01d21f93bb1661e7d36869ff33573e7a89aa9"} Nov 03 13:31:42.300855 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:31:42.300822 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-ef1693/payload-processing-external-model-e2e-extmod-a-ef1693-5578zqwf8" Nov 03 13:31:42.329896 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:31:42.329835 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-ef1693/payload-processing-external-model-e2e-extmod-a-ef1693-5578zqwf8" podStartSLOduration=2.329818868 podStartE2EDuration="2.329818868s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:31:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:31:42.323735967 +0000 UTC m=+1382.331927244" watchObservedRunningTime="2026-11-03 13:31:42.329818868 +0000 UTC m=+1382.338010199" Nov 03 13:31:43.113086 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:31:43.113044 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-5bffndnqx" podUID="6908d762-9f04-4823-a7f2-00a4e703b601" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 13:31:43.841715 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:31:43.841677 5024 scope.go:122] "RemoveContainer" containerID="75148bbfc8ef536a74d3997858931cd0358236c4a5f8adbc9a723758f3a07290" Nov 03 13:31:44.481388 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:31:44.481355 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-08111b/premium-w1-08111b-kserve-57555d598-dx9g5" Nov 03 13:31:44.492955 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:31:44.492918 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-08111b/premium-w1-08111b-kserve-57555d598-dx9g5" Nov 03 13:31:53.103392 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:31:53.103344 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-5bffndnqx" podUID="6908d762-9f04-4823-a7f2-00a4e703b601" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 13:31:53.312995 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:31:53.312961 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-1a18c8/payload-processing-external-model-e2e-extmod-b-1a18c8-8676lrgff" Nov 03 13:31:53.313182 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:31:53.313042 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-fdc7999b7-xb26w" Nov 03 13:31:54.376543 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:31:54.376501 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-ef1693/payload-processing-external-model-e2e-extmod-a-ef1693-5578zqwf8" Nov 03 13:31:58.494313 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:31:58.490373 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-08111b/premium-w1-08111b-kserve-57555d598-dx9g5" podUID="e45c569a-243f-49ab-8d74-dd08cbb6127c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.134:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 13:32:03.122940 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:03.122466 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-5bffndnqx" podUID="6908d762-9f04-4823-a7f2-00a4e703b601" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 13:32:04.432411 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:04.432368 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-5bffndnqx"] Nov 03 13:32:04.435010 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:04.434970 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-5bffndnqx" podUID="6908d762-9f04-4823-a7f2-00a4e703b601" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 13:32:04.463505 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:04.463461 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-7554n79pc"] Nov 03 13:32:04.469699 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:04.469640 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-7554n79pc" Nov 03 13:32:04.475553 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:04.475519 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-a62d22\"/\"agc-cred-c03d3bf9f10f\"" Nov 03 13:32:04.477629 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:04.477592 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-7554n79pc"] Nov 03 13:32:04.546625 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:04.546580 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2hzmw\" (UniqueName: \"kubernetes.io/projected/aaad66b5-4b6d-4f05-a2f9-a8d34e6036a4-kube-api-access-2hzmw\") pod \"payload-processing-external-model-e2e-extmod-a-a62d22-7554n79pc\" (UID: \"aaad66b5-4b6d-4f05-a2f9-a8d34e6036a4\") " pod="ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-7554n79pc" Nov 03 13:32:04.546842 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:04.546786 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/aaad66b5-4b6d-4f05-a2f9-a8d34e6036a4-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-a62d22-7554n79pc\" (UID: \"aaad66b5-4b6d-4f05-a2f9-a8d34e6036a4\") " pod="ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-7554n79pc" Nov 03 13:32:04.547037 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:04.547013 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/aaad66b5-4b6d-4f05-a2f9-a8d34e6036a4-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a62d22-7554n79pc\" (UID: \"aaad66b5-4b6d-4f05-a2f9-a8d34e6036a4\") " pod="ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-7554n79pc" Nov 03 13:32:04.547151 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:04.547124 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aaad66b5-4b6d-4f05-a2f9-a8d34e6036a4-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a62d22-7554n79pc\" (UID: \"aaad66b5-4b6d-4f05-a2f9-a8d34e6036a4\") " pod="ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-7554n79pc" Nov 03 13:32:04.656308 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:04.650701 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/aaad66b5-4b6d-4f05-a2f9-a8d34e6036a4-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a62d22-7554n79pc\" (UID: \"aaad66b5-4b6d-4f05-a2f9-a8d34e6036a4\") " pod="ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-7554n79pc" Nov 03 13:32:04.656308 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:04.650830 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aaad66b5-4b6d-4f05-a2f9-a8d34e6036a4-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a62d22-7554n79pc\" (UID: \"aaad66b5-4b6d-4f05-a2f9-a8d34e6036a4\") " pod="ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-7554n79pc" Nov 03 13:32:04.656308 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:04.650999 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hzmw\" (UniqueName: \"kubernetes.io/projected/aaad66b5-4b6d-4f05-a2f9-a8d34e6036a4-kube-api-access-2hzmw\") pod \"payload-processing-external-model-e2e-extmod-a-a62d22-7554n79pc\" (UID: \"aaad66b5-4b6d-4f05-a2f9-a8d34e6036a4\") " pod="ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-7554n79pc" Nov 03 13:32:04.656308 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:04.651150 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/aaad66b5-4b6d-4f05-a2f9-a8d34e6036a4-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-a62d22-7554n79pc\" (UID: \"aaad66b5-4b6d-4f05-a2f9-a8d34e6036a4\") " pod="ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-7554n79pc" Nov 03 13:32:04.678399 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:04.678360 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aaad66b5-4b6d-4f05-a2f9-a8d34e6036a4-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a62d22-7554n79pc\" (UID: \"aaad66b5-4b6d-4f05-a2f9-a8d34e6036a4\") " pod="ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-7554n79pc" Nov 03 13:32:04.679326 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:04.679298 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/aaad66b5-4b6d-4f05-a2f9-a8d34e6036a4-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a62d22-7554n79pc\" (UID: \"aaad66b5-4b6d-4f05-a2f9-a8d34e6036a4\") " pod="ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-7554n79pc" Nov 03 13:32:04.712587 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:04.712398 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/aaad66b5-4b6d-4f05-a2f9-a8d34e6036a4-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-a62d22-7554n79pc\" (UID: \"aaad66b5-4b6d-4f05-a2f9-a8d34e6036a4\") " pod="ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-7554n79pc" Nov 03 13:32:04.713404 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:04.713372 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2hzmw\" (UniqueName: \"kubernetes.io/projected/aaad66b5-4b6d-4f05-a2f9-a8d34e6036a4-kube-api-access-2hzmw\") pod \"payload-processing-external-model-e2e-extmod-a-a62d22-7554n79pc\" (UID: \"aaad66b5-4b6d-4f05-a2f9-a8d34e6036a4\") " pod="ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-7554n79pc" Nov 03 13:32:04.806015 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:04.805971 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-7554n79pc" Nov 03 13:32:05.102407 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:05.102374 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-7554n79pc"] Nov 03 13:32:05.528984 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:05.528946 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-7554n79pc" event={"ID":"aaad66b5-4b6d-4f05-a2f9-a8d34e6036a4","Type":"ContainerStarted","Data":"f2e2cf6705e996b618b47c9fc81e82626dc9852d81cfaace9136ae8f0b093ace"} Nov 03 13:32:05.529606 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:05.529003 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-7554n79pc" event={"ID":"aaad66b5-4b6d-4f05-a2f9-a8d34e6036a4","Type":"ContainerStarted","Data":"133deb9ea6077ec3de907940382721cd53ca2a47ba3c60045098c1db4ae3e470"} Nov 03 13:32:05.529606 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:05.529027 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-7554n79pc" Nov 03 13:32:05.556818 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:05.556752 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-7554n79pc" podStartSLOduration=1.556733404 podStartE2EDuration="1.556733404s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:32:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:32:05.551078252 +0000 UTC m=+1405.559269531" watchObservedRunningTime="2026-11-03 13:32:05.556733404 +0000 UTC m=+1405.564924681" Nov 03 13:32:09.441729 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:09.433843 5024 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-5bffndnqx" podUID="6908d762-9f04-4823-a7f2-00a4e703b601" containerName="payload-processing" containerID="cri-o://7a22fa57cc4e724021af1499f19bda4391dbf49842f8946a064c0dee05b5ab55" gracePeriod=25 Nov 03 13:32:09.515113 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:32:09.515061 5024 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d7mwz" containerName="extractor" Nov 03 13:32:09.610120 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:09.610083 5024 generic.go:356] "Generic (PLEG): container finished" podID="6908d762-9f04-4823-a7f2-00a4e703b601" containerID="7a22fa57cc4e724021af1499f19bda4391dbf49842f8946a064c0dee05b5ab55" exitCode=0 Nov 03 13:32:09.610333 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:09.610159 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-5bffndnqx" event={"ID":"6908d762-9f04-4823-a7f2-00a4e703b601","Type":"ContainerDied","Data":"7a22fa57cc4e724021af1499f19bda4391dbf49842f8946a064c0dee05b5ab55"} Nov 03 13:32:09.747547 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:09.747509 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-5bffndnqx" Nov 03 13:32:09.888343 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:09.888307 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6908d762-9f04-4823-a7f2-00a4e703b601-provider-credentials\" (UniqueName: \"kubernetes.io/projected/6908d762-9f04-4823-a7f2-00a4e703b601-provider-credentials\") pod \"6908d762-9f04-4823-a7f2-00a4e703b601\" (UID: \"6908d762-9f04-4823-a7f2-00a4e703b601\") " Nov 03 13:32:09.888553 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:09.888379 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6908d762-9f04-4823-a7f2-00a4e703b601-kube-api-access-7zw9c\" (UniqueName: \"kubernetes.io/projected/6908d762-9f04-4823-a7f2-00a4e703b601-kube-api-access-7zw9c\") pod \"6908d762-9f04-4823-a7f2-00a4e703b601\" (UID: \"6908d762-9f04-4823-a7f2-00a4e703b601\") " Nov 03 13:32:09.888623 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:09.888573 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6908d762-9f04-4823-a7f2-00a4e703b601-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6908d762-9f04-4823-a7f2-00a4e703b601-plugins-config-volume\") pod \"6908d762-9f04-4823-a7f2-00a4e703b601\" (UID: \"6908d762-9f04-4823-a7f2-00a4e703b601\") " Nov 03 13:32:09.888690 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:09.888655 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6908d762-9f04-4823-a7f2-00a4e703b601-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6908d762-9f04-4823-a7f2-00a4e703b601-routing-overlay-volume\") pod \"6908d762-9f04-4823-a7f2-00a4e703b601\" (UID: \"6908d762-9f04-4823-a7f2-00a4e703b601\") " Nov 03 13:32:09.889122 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:09.889087 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6908d762-9f04-4823-a7f2-00a4e703b601-plugins-config-volume" pod "6908d762-9f04-4823-a7f2-00a4e703b601" (UID: "6908d762-9f04-4823-a7f2-00a4e703b601"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:32:09.889232 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:09.889124 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6908d762-9f04-4823-a7f2-00a4e703b601-routing-overlay-volume" pod "6908d762-9f04-4823-a7f2-00a4e703b601" (UID: "6908d762-9f04-4823-a7f2-00a4e703b601"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:32:09.889609 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:09.889584 5024 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6908d762-9f04-4823-a7f2-00a4e703b601-plugins-config-volume\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:32:09.889709 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:09.889612 5024 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6908d762-9f04-4823-a7f2-00a4e703b601-routing-overlay-volume\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:32:09.891232 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:09.891178 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6908d762-9f04-4823-a7f2-00a4e703b601-kube-api-access-7zw9c" pod "6908d762-9f04-4823-a7f2-00a4e703b601" (UID: "6908d762-9f04-4823-a7f2-00a4e703b601"). InnerVolumeSpecName "kube-api-access-7zw9c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:32:09.891357 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:09.891340 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6908d762-9f04-4823-a7f2-00a4e703b601-provider-credentials" pod "6908d762-9f04-4823-a7f2-00a4e703b601" (UID: "6908d762-9f04-4823-a7f2-00a4e703b601"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:32:09.992364 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:09.991307 5024 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6908d762-9f04-4823-a7f2-00a4e703b601-provider-credentials\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:32:09.992364 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:09.991351 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7zw9c\" (UniqueName: \"kubernetes.io/projected/6908d762-9f04-4823-a7f2-00a4e703b601-kube-api-access-7zw9c\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:32:10.620806 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:10.620767 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-5bffndnqx" Nov 03 13:32:10.621363 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:10.620777 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-5bffndnqx" event={"ID":"6908d762-9f04-4823-a7f2-00a4e703b601","Type":"ContainerDied","Data":"034c244e6819a2710b027639a0e42e6254be3a34e995d3d416e47181f94bfcc1"} Nov 03 13:32:10.621363 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:10.620908 5024 scope.go:122] "RemoveContainer" containerID="7a22fa57cc4e724021af1499f19bda4391dbf49842f8946a064c0dee05b5ab55" Nov 03 13:32:10.643735 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:10.643691 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-5bffndnqx"] Nov 03 13:32:10.651031 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:10.651001 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-5bffndnqx"] Nov 03 13:32:12.244365 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:12.238097 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6908d762-9f04-4823-a7f2-00a4e703b601" path="/var/lib/kubelet/pods/6908d762-9f04-4823-a7f2-00a4e703b601/volumes" Nov 03 13:32:12.977309 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:12.974312 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-08111b/simulator-w1-08111b-kserve-7f65c5867-btf5b" podUID="e8aba665-e4cc-4e67-8cdd-d09ef7542e15" containerName="main" probeResult="failure" output="Get \"https://10.132.0.133:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 13:32:15.243309 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:15.216637 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-fdc7999b7-xb26w"] Nov 03 13:32:15.636328 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:15.635660 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-ef1693/payload-processing-external-model-e2e-extmod-a-ef1693-5578zqwf8"] Nov 03 13:32:16.417338 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:16.409844 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-1a18c8/payload-processing-external-model-e2e-extmod-b-1a18c8-8676lrgff"] Nov 03 13:32:16.597308 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:16.581510 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-7554n79pc" Nov 03 13:32:16.958311 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:16.957855 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-cf4789xvd"] Nov 03 13:32:20.217442 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:20.217392 5024 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-fdc7999b7-xb26w" podUID="fa2f0750-c048-4c12-ba76-1d54f2abb5f7" containerName="payload-processing" containerID="cri-o://7d2c88da1906c77a344ef117f9fee5acbf1da79d9d7e8b2a184496a60068a991" gracePeriod=25 Nov 03 13:32:20.532607 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:20.532572 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-fdc7999b7-xb26w" Nov 03 13:32:20.619023 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:20.618976 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa2f0750-c048-4c12-ba76-1d54f2abb5f7-provider-credentials\" (UniqueName: \"kubernetes.io/projected/fa2f0750-c048-4c12-ba76-1d54f2abb5f7-provider-credentials\") pod \"fa2f0750-c048-4c12-ba76-1d54f2abb5f7\" (UID: \"fa2f0750-c048-4c12-ba76-1d54f2abb5f7\") " Nov 03 13:32:20.619227 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:20.619101 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fa2f0750-c048-4c12-ba76-1d54f2abb5f7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa2f0750-c048-4c12-ba76-1d54f2abb5f7-plugins-config-volume\") pod \"fa2f0750-c048-4c12-ba76-1d54f2abb5f7\" (UID: \"fa2f0750-c048-4c12-ba76-1d54f2abb5f7\") " Nov 03 13:32:20.619227 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:20.619202 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa2f0750-c048-4c12-ba76-1d54f2abb5f7-kube-api-access-874j8\" (UniqueName: \"kubernetes.io/projected/fa2f0750-c048-4c12-ba76-1d54f2abb5f7-kube-api-access-874j8\") pod \"fa2f0750-c048-4c12-ba76-1d54f2abb5f7\" (UID: \"fa2f0750-c048-4c12-ba76-1d54f2abb5f7\") " Nov 03 13:32:20.619399 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:20.619332 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fa2f0750-c048-4c12-ba76-1d54f2abb5f7-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/fa2f0750-c048-4c12-ba76-1d54f2abb5f7-routing-overlay-volume\") pod \"fa2f0750-c048-4c12-ba76-1d54f2abb5f7\" (UID: \"fa2f0750-c048-4c12-ba76-1d54f2abb5f7\") " Nov 03 13:32:20.620008 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:20.619980 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa2f0750-c048-4c12-ba76-1d54f2abb5f7-plugins-config-volume" pod "fa2f0750-c048-4c12-ba76-1d54f2abb5f7" (UID: "fa2f0750-c048-4c12-ba76-1d54f2abb5f7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:32:20.620396 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:20.620366 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa2f0750-c048-4c12-ba76-1d54f2abb5f7-routing-overlay-volume" pod "fa2f0750-c048-4c12-ba76-1d54f2abb5f7" (UID: "fa2f0750-c048-4c12-ba76-1d54f2abb5f7"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:32:20.620570 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:20.620547 5024 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/fa2f0750-c048-4c12-ba76-1d54f2abb5f7-routing-overlay-volume\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:32:20.620647 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:20.620577 5024 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa2f0750-c048-4c12-ba76-1d54f2abb5f7-plugins-config-volume\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:32:20.623046 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:20.623016 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa2f0750-c048-4c12-ba76-1d54f2abb5f7-provider-credentials" pod "fa2f0750-c048-4c12-ba76-1d54f2abb5f7" (UID: "fa2f0750-c048-4c12-ba76-1d54f2abb5f7"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:32:20.623350 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:20.623319 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa2f0750-c048-4c12-ba76-1d54f2abb5f7-kube-api-access-874j8" pod "fa2f0750-c048-4c12-ba76-1d54f2abb5f7" (UID: "fa2f0750-c048-4c12-ba76-1d54f2abb5f7"). InnerVolumeSpecName "kube-api-access-874j8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:32:20.636947 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:20.636900 5024 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-ef1693/payload-processing-external-model-e2e-extmod-a-ef1693-5578zqwf8" podUID="26c02b21-f8e5-441b-96db-6512b211606e" containerName="payload-processing" containerID="cri-o://d0b068305cc8569b5405a2459cce581924aa2cf2031cc1160bb1da48c29e35a0" gracePeriod=25 Nov 03 13:32:20.745623 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:20.742486 5024 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/fa2f0750-c048-4c12-ba76-1d54f2abb5f7-provider-credentials\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:32:20.745623 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:20.742531 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-874j8\" (UniqueName: \"kubernetes.io/projected/fa2f0750-c048-4c12-ba76-1d54f2abb5f7-kube-api-access-874j8\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:32:21.002935 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:21.002899 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-ef1693/payload-processing-external-model-e2e-extmod-a-ef1693-5578zqwf8" Nov 03 13:32:21.047201 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:21.047163 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/26c02b21-f8e5-441b-96db-6512b211606e-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/26c02b21-f8e5-441b-96db-6512b211606e-routing-overlay-volume\") pod \"26c02b21-f8e5-441b-96db-6512b211606e\" (UID: \"26c02b21-f8e5-441b-96db-6512b211606e\") " Nov 03 13:32:21.047414 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:21.047398 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26c02b21-f8e5-441b-96db-6512b211606e-provider-credentials\" (UniqueName: \"kubernetes.io/projected/26c02b21-f8e5-441b-96db-6512b211606e-provider-credentials\") pod \"26c02b21-f8e5-441b-96db-6512b211606e\" (UID: \"26c02b21-f8e5-441b-96db-6512b211606e\") " Nov 03 13:32:21.047490 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:21.047447 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26c02b21-f8e5-441b-96db-6512b211606e-kube-api-access-w9k47\" (UniqueName: \"kubernetes.io/projected/26c02b21-f8e5-441b-96db-6512b211606e-kube-api-access-w9k47\") pod \"26c02b21-f8e5-441b-96db-6512b211606e\" (UID: \"26c02b21-f8e5-441b-96db-6512b211606e\") " Nov 03 13:32:21.047630 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:21.047607 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/26c02b21-f8e5-441b-96db-6512b211606e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26c02b21-f8e5-441b-96db-6512b211606e-plugins-config-volume\") pod \"26c02b21-f8e5-441b-96db-6512b211606e\" (UID: \"26c02b21-f8e5-441b-96db-6512b211606e\") " Nov 03 13:32:21.047630 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:21.047606 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26c02b21-f8e5-441b-96db-6512b211606e-routing-overlay-volume" pod "26c02b21-f8e5-441b-96db-6512b211606e" (UID: "26c02b21-f8e5-441b-96db-6512b211606e"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:32:21.048203 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:21.048178 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26c02b21-f8e5-441b-96db-6512b211606e-plugins-config-volume" pod "26c02b21-f8e5-441b-96db-6512b211606e" (UID: "26c02b21-f8e5-441b-96db-6512b211606e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:32:21.048325 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:21.048212 5024 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/26c02b21-f8e5-441b-96db-6512b211606e-routing-overlay-volume\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:32:21.049952 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:21.049898 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26c02b21-f8e5-441b-96db-6512b211606e-provider-credentials" pod "26c02b21-f8e5-441b-96db-6512b211606e" (UID: "26c02b21-f8e5-441b-96db-6512b211606e"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:32:21.051044 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:21.051023 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26c02b21-f8e5-441b-96db-6512b211606e-kube-api-access-w9k47" pod "26c02b21-f8e5-441b-96db-6512b211606e" (UID: "26c02b21-f8e5-441b-96db-6512b211606e"). InnerVolumeSpecName "kube-api-access-w9k47". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:32:21.153701 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:21.153664 5024 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/26c02b21-f8e5-441b-96db-6512b211606e-provider-credentials\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:32:21.153701 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:21.153698 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w9k47\" (UniqueName: \"kubernetes.io/projected/26c02b21-f8e5-441b-96db-6512b211606e-kube-api-access-w9k47\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:32:21.153701 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:21.153708 5024 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26c02b21-f8e5-441b-96db-6512b211606e-plugins-config-volume\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:32:21.246034 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:21.245993 5024 generic.go:356] "Generic (PLEG): container finished" podID="26c02b21-f8e5-441b-96db-6512b211606e" containerID="d0b068305cc8569b5405a2459cce581924aa2cf2031cc1160bb1da48c29e35a0" exitCode=0 Nov 03 13:32:21.246561 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:21.246048 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-ef1693/payload-processing-external-model-e2e-extmod-a-ef1693-5578zqwf8" Nov 03 13:32:21.246561 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:21.246049 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-ef1693/payload-processing-external-model-e2e-extmod-a-ef1693-5578zqwf8" event={"ID":"26c02b21-f8e5-441b-96db-6512b211606e","Type":"ContainerDied","Data":"d0b068305cc8569b5405a2459cce581924aa2cf2031cc1160bb1da48c29e35a0"} Nov 03 13:32:21.246561 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:21.246129 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-ef1693/payload-processing-external-model-e2e-extmod-a-ef1693-5578zqwf8" event={"ID":"26c02b21-f8e5-441b-96db-6512b211606e","Type":"ContainerDied","Data":"7a3bad7eeefbdcfc73955b404eb01d21f93bb1661e7d36869ff33573e7a89aa9"} Nov 03 13:32:21.246561 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:21.246163 5024 scope.go:122] "RemoveContainer" containerID="d0b068305cc8569b5405a2459cce581924aa2cf2031cc1160bb1da48c29e35a0" Nov 03 13:32:21.248215 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:21.248182 5024 generic.go:356] "Generic (PLEG): container finished" podID="fa2f0750-c048-4c12-ba76-1d54f2abb5f7" containerID="7d2c88da1906c77a344ef117f9fee5acbf1da79d9d7e8b2a184496a60068a991" exitCode=0 Nov 03 13:32:21.248369 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:21.248233 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-fdc7999b7-xb26w" Nov 03 13:32:21.248369 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:21.248339 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-fdc7999b7-xb26w" event={"ID":"fa2f0750-c048-4c12-ba76-1d54f2abb5f7","Type":"ContainerDied","Data":"7d2c88da1906c77a344ef117f9fee5acbf1da79d9d7e8b2a184496a60068a991"} Nov 03 13:32:21.248473 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:21.248406 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-fdc7999b7-xb26w" event={"ID":"fa2f0750-c048-4c12-ba76-1d54f2abb5f7","Type":"ContainerDied","Data":"dbdbd7bcfbb826f67d902cd256399f856b33b742eeb6440221a75ac1da200e78"} Nov 03 13:32:21.275496 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:21.275442 5024 scope.go:122] "RemoveContainer" containerID="d0b068305cc8569b5405a2459cce581924aa2cf2031cc1160bb1da48c29e35a0" Nov 03 13:32:21.275933 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:32:21.275901 5024 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d0b068305cc8569b5405a2459cce581924aa2cf2031cc1160bb1da48c29e35a0\": container with ID starting with d0b068305cc8569b5405a2459cce581924aa2cf2031cc1160bb1da48c29e35a0 not found: ID does not exist" containerID="d0b068305cc8569b5405a2459cce581924aa2cf2031cc1160bb1da48c29e35a0" Nov 03 13:32:21.276213 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:21.275937 5024 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0b068305cc8569b5405a2459cce581924aa2cf2031cc1160bb1da48c29e35a0"} err="failed to get container status \"d0b068305cc8569b5405a2459cce581924aa2cf2031cc1160bb1da48c29e35a0\": rpc error: code = NotFound desc = could not find container \"d0b068305cc8569b5405a2459cce581924aa2cf2031cc1160bb1da48c29e35a0\": container with ID starting with d0b068305cc8569b5405a2459cce581924aa2cf2031cc1160bb1da48c29e35a0 not found: ID does not exist" Nov 03 13:32:21.276213 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:21.275962 5024 scope.go:122] "RemoveContainer" containerID="7d2c88da1906c77a344ef117f9fee5acbf1da79d9d7e8b2a184496a60068a991" Nov 03 13:32:21.287821 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:21.287785 5024 scope.go:122] "RemoveContainer" containerID="7d2c88da1906c77a344ef117f9fee5acbf1da79d9d7e8b2a184496a60068a991" Nov 03 13:32:21.288228 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:32:21.288189 5024 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d2c88da1906c77a344ef117f9fee5acbf1da79d9d7e8b2a184496a60068a991\": container with ID starting with 7d2c88da1906c77a344ef117f9fee5acbf1da79d9d7e8b2a184496a60068a991 not found: ID does not exist" containerID="7d2c88da1906c77a344ef117f9fee5acbf1da79d9d7e8b2a184496a60068a991" Nov 03 13:32:21.288357 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:21.288246 5024 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d2c88da1906c77a344ef117f9fee5acbf1da79d9d7e8b2a184496a60068a991"} err="failed to get container status \"7d2c88da1906c77a344ef117f9fee5acbf1da79d9d7e8b2a184496a60068a991\": rpc error: code = NotFound desc = could not find container \"7d2c88da1906c77a344ef117f9fee5acbf1da79d9d7e8b2a184496a60068a991\": container with ID starting with 7d2c88da1906c77a344ef117f9fee5acbf1da79d9d7e8b2a184496a60068a991 not found: ID does not exist" Nov 03 13:32:21.293885 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:21.293830 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-fdc7999b7-xb26w"] Nov 03 13:32:21.299662 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:21.299623 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-fdc7999b7-xb26w"] Nov 03 13:32:21.316614 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:21.316539 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-ef1693/payload-processing-external-model-e2e-extmod-a-ef1693-5578zqwf8"] Nov 03 13:32:21.320219 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:21.320191 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-ef1693/payload-processing-external-model-e2e-extmod-a-ef1693-5578zqwf8"] Nov 03 13:32:21.410817 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:21.410713 5024 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-1a18c8/payload-processing-external-model-e2e-extmod-b-1a18c8-8676lrgff" podUID="fa6220b6-0c80-431e-8689-d6e63e5653e0" containerName="payload-processing" containerID="cri-o://19210be1a553e292b693b44829623bca3a646678f0d82141fbb2d8b29c9a2013" gracePeriod=25 Nov 03 13:32:21.790198 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:21.790170 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-1a18c8/payload-processing-external-model-e2e-extmod-b-1a18c8-8676lrgff" Nov 03 13:32:21.866332 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:21.866274 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa6220b6-0c80-431e-8689-d6e63e5653e0-provider-credentials\" (UniqueName: \"kubernetes.io/projected/fa6220b6-0c80-431e-8689-d6e63e5653e0-provider-credentials\") pod \"fa6220b6-0c80-431e-8689-d6e63e5653e0\" (UID: \"fa6220b6-0c80-431e-8689-d6e63e5653e0\") " Nov 03 13:32:21.866543 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:21.866419 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fa6220b6-0c80-431e-8689-d6e63e5653e0-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/fa6220b6-0c80-431e-8689-d6e63e5653e0-routing-overlay-volume\") pod \"fa6220b6-0c80-431e-8689-d6e63e5653e0\" (UID: \"fa6220b6-0c80-431e-8689-d6e63e5653e0\") " Nov 03 13:32:21.866543 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:21.866502 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fa6220b6-0c80-431e-8689-d6e63e5653e0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa6220b6-0c80-431e-8689-d6e63e5653e0-plugins-config-volume\") pod \"fa6220b6-0c80-431e-8689-d6e63e5653e0\" (UID: \"fa6220b6-0c80-431e-8689-d6e63e5653e0\") " Nov 03 13:32:21.866928 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:21.866893 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa6220b6-0c80-431e-8689-d6e63e5653e0-routing-overlay-volume" pod "fa6220b6-0c80-431e-8689-d6e63e5653e0" (UID: "fa6220b6-0c80-431e-8689-d6e63e5653e0"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:32:21.867022 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:21.866990 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa6220b6-0c80-431e-8689-d6e63e5653e0-plugins-config-volume" pod "fa6220b6-0c80-431e-8689-d6e63e5653e0" (UID: "fa6220b6-0c80-431e-8689-d6e63e5653e0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:32:21.867089 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:21.867032 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa6220b6-0c80-431e-8689-d6e63e5653e0-kube-api-access-t6hlj\" (UniqueName: \"kubernetes.io/projected/fa6220b6-0c80-431e-8689-d6e63e5653e0-kube-api-access-t6hlj\") pod \"fa6220b6-0c80-431e-8689-d6e63e5653e0\" (UID: \"fa6220b6-0c80-431e-8689-d6e63e5653e0\") " Nov 03 13:32:21.867884 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:21.867856 5024 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/fa6220b6-0c80-431e-8689-d6e63e5653e0-routing-overlay-volume\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:32:21.868001 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:21.867886 5024 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa6220b6-0c80-431e-8689-d6e63e5653e0-plugins-config-volume\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:32:21.868723 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:21.868698 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa6220b6-0c80-431e-8689-d6e63e5653e0-provider-credentials" pod "fa6220b6-0c80-431e-8689-d6e63e5653e0" (UID: "fa6220b6-0c80-431e-8689-d6e63e5653e0"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:32:21.869241 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:21.869224 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa6220b6-0c80-431e-8689-d6e63e5653e0-kube-api-access-t6hlj" pod "fa6220b6-0c80-431e-8689-d6e63e5653e0" (UID: "fa6220b6-0c80-431e-8689-d6e63e5653e0"). InnerVolumeSpecName "kube-api-access-t6hlj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:32:21.959019 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:21.958910 5024 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-cf4789xvd" podUID="9b739270-1c05-44b6-bed6-de7eaadb689f" containerName="payload-processing" containerID="cri-o://d0901a47ddce81ec654fecc910a2c87fe5c16dd9d7b1d0f27056796f7e761cc7" gracePeriod=25 Nov 03 13:32:21.969656 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:21.969620 5024 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/fa6220b6-0c80-431e-8689-d6e63e5653e0-provider-credentials\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:32:21.969656 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:21.969653 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t6hlj\" (UniqueName: \"kubernetes.io/projected/fa6220b6-0c80-431e-8689-d6e63e5653e0-kube-api-access-t6hlj\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:32:22.113054 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:22.113012 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="26c02b21-f8e5-441b-96db-6512b211606e" path="/var/lib/kubelet/pods/26c02b21-f8e5-441b-96db-6512b211606e/volumes" Nov 03 13:32:22.113906 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:22.113877 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fa2f0750-c048-4c12-ba76-1d54f2abb5f7" path="/var/lib/kubelet/pods/fa2f0750-c048-4c12-ba76-1d54f2abb5f7/volumes" Nov 03 13:32:22.258087 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:22.258046 5024 generic.go:356] "Generic (PLEG): container finished" podID="fa6220b6-0c80-431e-8689-d6e63e5653e0" containerID="19210be1a553e292b693b44829623bca3a646678f0d82141fbb2d8b29c9a2013" exitCode=0 Nov 03 13:32:22.258532 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:22.258114 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-1a18c8/payload-processing-external-model-e2e-extmod-b-1a18c8-8676lrgff" Nov 03 13:32:22.258532 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:22.258138 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-1a18c8/payload-processing-external-model-e2e-extmod-b-1a18c8-8676lrgff" event={"ID":"fa6220b6-0c80-431e-8689-d6e63e5653e0","Type":"ContainerDied","Data":"19210be1a553e292b693b44829623bca3a646678f0d82141fbb2d8b29c9a2013"} Nov 03 13:32:22.258532 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:22.258196 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-1a18c8/payload-processing-external-model-e2e-extmod-b-1a18c8-8676lrgff" event={"ID":"fa6220b6-0c80-431e-8689-d6e63e5653e0","Type":"ContainerDied","Data":"5749e0ae13e695cdf5f3cc68c9bf4cf279f018bb5a4c05b85cbc0046023d3b54"} Nov 03 13:32:22.258532 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:22.258227 5024 scope.go:122] "RemoveContainer" containerID="19210be1a553e292b693b44829623bca3a646678f0d82141fbb2d8b29c9a2013" Nov 03 13:32:22.262119 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:22.262094 5024 generic.go:356] "Generic (PLEG): container finished" podID="9b739270-1c05-44b6-bed6-de7eaadb689f" containerID="d0901a47ddce81ec654fecc910a2c87fe5c16dd9d7b1d0f27056796f7e761cc7" exitCode=0 Nov 03 13:32:22.262254 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:22.262181 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-cf4789xvd" event={"ID":"9b739270-1c05-44b6-bed6-de7eaadb689f","Type":"ContainerDied","Data":"d0901a47ddce81ec654fecc910a2c87fe5c16dd9d7b1d0f27056796f7e761cc7"} Nov 03 13:32:22.262254 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:22.262243 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-cf4789xvd" event={"ID":"9b739270-1c05-44b6-bed6-de7eaadb689f","Type":"ContainerDied","Data":"d33922f116839a0da1281c0ee59114b73542a6e08889467ad5192e20d1ae63be"} Nov 03 13:32:22.262415 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:22.262271 5024 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d33922f116839a0da1281c0ee59114b73542a6e08889467ad5192e20d1ae63be" Nov 03 13:32:22.280860 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:22.280802 5024 scope.go:122] "RemoveContainer" containerID="19210be1a553e292b693b44829623bca3a646678f0d82141fbb2d8b29c9a2013" Nov 03 13:32:22.281161 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:32:22.281134 5024 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"19210be1a553e292b693b44829623bca3a646678f0d82141fbb2d8b29c9a2013\": container with ID starting with 19210be1a553e292b693b44829623bca3a646678f0d82141fbb2d8b29c9a2013 not found: ID does not exist" containerID="19210be1a553e292b693b44829623bca3a646678f0d82141fbb2d8b29c9a2013" Nov 03 13:32:22.281378 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:22.281167 5024 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19210be1a553e292b693b44829623bca3a646678f0d82141fbb2d8b29c9a2013"} err="failed to get container status \"19210be1a553e292b693b44829623bca3a646678f0d82141fbb2d8b29c9a2013\": rpc error: code = NotFound desc = could not find container \"19210be1a553e292b693b44829623bca3a646678f0d82141fbb2d8b29c9a2013\": container with ID starting with 19210be1a553e292b693b44829623bca3a646678f0d82141fbb2d8b29c9a2013 not found: ID does not exist" Nov 03 13:32:22.285118 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:22.285093 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-cf4789xvd" Nov 03 13:32:22.299961 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:22.299929 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-1a18c8/payload-processing-external-model-e2e-extmod-b-1a18c8-8676lrgff"] Nov 03 13:32:22.303824 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:22.303793 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-1a18c8/payload-processing-external-model-e2e-extmod-b-1a18c8-8676lrgff"] Nov 03 13:32:22.376987 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:22.376946 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b739270-1c05-44b6-bed6-de7eaadb689f-kube-api-access-wktb5\" (UniqueName: \"kubernetes.io/projected/9b739270-1c05-44b6-bed6-de7eaadb689f-kube-api-access-wktb5\") pod \"9b739270-1c05-44b6-bed6-de7eaadb689f\" (UID: \"9b739270-1c05-44b6-bed6-de7eaadb689f\") " Nov 03 13:32:22.377197 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:22.377098 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b739270-1c05-44b6-bed6-de7eaadb689f-provider-credentials\" (UniqueName: \"kubernetes.io/projected/9b739270-1c05-44b6-bed6-de7eaadb689f-provider-credentials\") pod \"9b739270-1c05-44b6-bed6-de7eaadb689f\" (UID: \"9b739270-1c05-44b6-bed6-de7eaadb689f\") " Nov 03 13:32:22.377255 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:22.377221 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9b739270-1c05-44b6-bed6-de7eaadb689f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b739270-1c05-44b6-bed6-de7eaadb689f-plugins-config-volume\") pod \"9b739270-1c05-44b6-bed6-de7eaadb689f\" (UID: \"9b739270-1c05-44b6-bed6-de7eaadb689f\") " Nov 03 13:32:22.377366 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:22.377271 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9b739270-1c05-44b6-bed6-de7eaadb689f-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9b739270-1c05-44b6-bed6-de7eaadb689f-routing-overlay-volume\") pod \"9b739270-1c05-44b6-bed6-de7eaadb689f\" (UID: \"9b739270-1c05-44b6-bed6-de7eaadb689f\") " Nov 03 13:32:22.378168 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:22.378133 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b739270-1c05-44b6-bed6-de7eaadb689f-routing-overlay-volume" pod "9b739270-1c05-44b6-bed6-de7eaadb689f" (UID: "9b739270-1c05-44b6-bed6-de7eaadb689f"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:32:22.378598 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:22.378572 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b739270-1c05-44b6-bed6-de7eaadb689f-plugins-config-volume" pod "9b739270-1c05-44b6-bed6-de7eaadb689f" (UID: "9b739270-1c05-44b6-bed6-de7eaadb689f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:32:22.379852 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:22.379816 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b739270-1c05-44b6-bed6-de7eaadb689f-kube-api-access-wktb5" pod "9b739270-1c05-44b6-bed6-de7eaadb689f" (UID: "9b739270-1c05-44b6-bed6-de7eaadb689f"). InnerVolumeSpecName "kube-api-access-wktb5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:32:22.380836 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:22.380812 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b739270-1c05-44b6-bed6-de7eaadb689f-provider-credentials" pod "9b739270-1c05-44b6-bed6-de7eaadb689f" (UID: "9b739270-1c05-44b6-bed6-de7eaadb689f"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:32:22.503363 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:22.489965 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wktb5\" (UniqueName: \"kubernetes.io/projected/9b739270-1c05-44b6-bed6-de7eaadb689f-kube-api-access-wktb5\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:32:22.503363 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:22.490004 5024 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9b739270-1c05-44b6-bed6-de7eaadb689f-provider-credentials\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:32:22.503363 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:22.490018 5024 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b739270-1c05-44b6-bed6-de7eaadb689f-plugins-config-volume\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:32:22.503363 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:22.490033 5024 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9b739270-1c05-44b6-bed6-de7eaadb689f-routing-overlay-volume\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:32:23.329208 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:23.328460 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-cf4789xvd" Nov 03 13:32:23.356649 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:23.356611 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-cf4789xvd"] Nov 03 13:32:23.360877 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:23.360848 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-cf4789xvd"] Nov 03 13:32:24.122070 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:24.122028 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9b739270-1c05-44b6-bed6-de7eaadb689f" path="/var/lib/kubelet/pods/9b739270-1c05-44b6-bed6-de7eaadb689f/volumes" Nov 03 13:32:24.122643 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:24.122618 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fa6220b6-0c80-431e-8689-d6e63e5653e0" path="/var/lib/kubelet/pods/fa6220b6-0c80-431e-8689-d6e63e5653e0/volumes" Nov 03 13:32:32.337208 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:32.337076 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-5546676475-98nhd"] Nov 03 13:32:32.339600 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:32.339570 5024 cpu_manager.go:414] "containerMap: removing container" podUID="fa2f0750-c048-4c12-ba76-1d54f2abb5f7" containerName="payload-processing" Nov 03 13:32:32.339821 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:32.339783 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa2f0750-c048-4c12-ba76-1d54f2abb5f7" containerName="payload-processing" Nov 03 13:32:32.339973 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:32.339960 5024 cpu_manager.go:414] "containerMap: removing container" podUID="fa6220b6-0c80-431e-8689-d6e63e5653e0" containerName="payload-processing" Nov 03 13:32:32.340052 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:32.340041 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa6220b6-0c80-431e-8689-d6e63e5653e0" containerName="payload-processing" Nov 03 13:32:32.340165 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:32.340154 5024 cpu_manager.go:414] "containerMap: removing container" podUID="9b739270-1c05-44b6-bed6-de7eaadb689f" containerName="payload-processing" Nov 03 13:32:32.340243 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:32.340233 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b739270-1c05-44b6-bed6-de7eaadb689f" containerName="payload-processing" Nov 03 13:32:32.340380 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:32.340365 5024 cpu_manager.go:414] "containerMap: removing container" podUID="6908d762-9f04-4823-a7f2-00a4e703b601" containerName="payload-processing" Nov 03 13:32:32.340480 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:32.340467 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6908d762-9f04-4823-a7f2-00a4e703b601" containerName="payload-processing" Nov 03 13:32:32.340604 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:32.340593 5024 cpu_manager.go:414] "containerMap: removing container" podUID="26c02b21-f8e5-441b-96db-6512b211606e" containerName="payload-processing" Nov 03 13:32:32.340682 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:32.340671 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26c02b21-f8e5-441b-96db-6512b211606e" containerName="payload-processing" Nov 03 13:32:32.340940 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:32.340925 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="26c02b21-f8e5-441b-96db-6512b211606e" containerName="payload-processing" Nov 03 13:32:32.341038 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:32.341026 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa2f0750-c048-4c12-ba76-1d54f2abb5f7" containerName="payload-processing" Nov 03 13:32:32.341130 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:32.341118 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa6220b6-0c80-431e-8689-d6e63e5653e0" containerName="payload-processing" Nov 03 13:32:32.341217 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:32.341207 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b739270-1c05-44b6-bed6-de7eaadb689f" containerName="payload-processing" Nov 03 13:32:32.341410 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:32.341396 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="6908d762-9f04-4823-a7f2-00a4e703b601" containerName="payload-processing" Nov 03 13:32:32.348244 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:32.348215 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5546676475-98nhd"] Nov 03 13:32:32.348593 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:32.348575 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5546676475-98nhd" Nov 03 13:32:32.383673 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:32.383627 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\"" Nov 03 13:32:32.384000 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:32.383964 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\"" Nov 03 13:32:32.384147 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:32.384100 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\"" Nov 03 13:32:32.385374 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:32.385354 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\"" Nov 03 13:32:32.420609 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:32.420498 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-c398a670ca11\"" Nov 03 13:32:32.420609 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:32.420513 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-vx42v\"" Nov 03 13:32:32.420609 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:32.420572 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-e7798d3231e5\"" Nov 03 13:32:32.420609 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:32.420591 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-3de2312914d9\"" Nov 03 13:32:32.436190 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:32.436151 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/49f61653-f10c-4692-bc8f-a379cfd3b073-provider-credentials\") pod \"payload-processing-external-model-5546676475-98nhd\" (UID: \"49f61653-f10c-4692-bc8f-a379cfd3b073\") " pod="models-as-a-service/payload-processing-external-model-5546676475-98nhd" Nov 03 13:32:32.436413 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:32.436352 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dh74s\" (UniqueName: \"kubernetes.io/projected/49f61653-f10c-4692-bc8f-a379cfd3b073-kube-api-access-dh74s\") pod \"payload-processing-external-model-5546676475-98nhd\" (UID: \"49f61653-f10c-4692-bc8f-a379cfd3b073\") " pod="models-as-a-service/payload-processing-external-model-5546676475-98nhd" Nov 03 13:32:32.436547 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:32.436522 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/49f61653-f10c-4692-bc8f-a379cfd3b073-routing-overlay-volume\") pod \"payload-processing-external-model-5546676475-98nhd\" (UID: \"49f61653-f10c-4692-bc8f-a379cfd3b073\") " pod="models-as-a-service/payload-processing-external-model-5546676475-98nhd" Nov 03 13:32:32.436624 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:32.436612 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49f61653-f10c-4692-bc8f-a379cfd3b073-plugins-config-volume\") pod \"payload-processing-external-model-5546676475-98nhd\" (UID: \"49f61653-f10c-4692-bc8f-a379cfd3b073\") " pod="models-as-a-service/payload-processing-external-model-5546676475-98nhd" Nov 03 13:32:32.538030 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:32.537992 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/49f61653-f10c-4692-bc8f-a379cfd3b073-routing-overlay-volume\") pod \"payload-processing-external-model-5546676475-98nhd\" (UID: \"49f61653-f10c-4692-bc8f-a379cfd3b073\") " pod="models-as-a-service/payload-processing-external-model-5546676475-98nhd" Nov 03 13:32:32.538223 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:32.538115 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49f61653-f10c-4692-bc8f-a379cfd3b073-plugins-config-volume\") pod \"payload-processing-external-model-5546676475-98nhd\" (UID: \"49f61653-f10c-4692-bc8f-a379cfd3b073\") " pod="models-as-a-service/payload-processing-external-model-5546676475-98nhd" Nov 03 13:32:32.538345 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:32.538325 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/49f61653-f10c-4692-bc8f-a379cfd3b073-provider-credentials\") pod \"payload-processing-external-model-5546676475-98nhd\" (UID: \"49f61653-f10c-4692-bc8f-a379cfd3b073\") " pod="models-as-a-service/payload-processing-external-model-5546676475-98nhd" Nov 03 13:32:32.538441 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:32.538425 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dh74s\" (UniqueName: \"kubernetes.io/projected/49f61653-f10c-4692-bc8f-a379cfd3b073-kube-api-access-dh74s\") pod \"payload-processing-external-model-5546676475-98nhd\" (UID: \"49f61653-f10c-4692-bc8f-a379cfd3b073\") " pod="models-as-a-service/payload-processing-external-model-5546676475-98nhd" Nov 03 13:32:32.538849 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:32.538827 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/49f61653-f10c-4692-bc8f-a379cfd3b073-routing-overlay-volume\") pod \"payload-processing-external-model-5546676475-98nhd\" (UID: \"49f61653-f10c-4692-bc8f-a379cfd3b073\") " pod="models-as-a-service/payload-processing-external-model-5546676475-98nhd" Nov 03 13:32:32.538988 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:32.538969 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49f61653-f10c-4692-bc8f-a379cfd3b073-plugins-config-volume\") pod \"payload-processing-external-model-5546676475-98nhd\" (UID: \"49f61653-f10c-4692-bc8f-a379cfd3b073\") " pod="models-as-a-service/payload-processing-external-model-5546676475-98nhd" Nov 03 13:32:32.540981 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:32.540956 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/49f61653-f10c-4692-bc8f-a379cfd3b073-provider-credentials\") pod \"payload-processing-external-model-5546676475-98nhd\" (UID: \"49f61653-f10c-4692-bc8f-a379cfd3b073\") " pod="models-as-a-service/payload-processing-external-model-5546676475-98nhd" Nov 03 13:32:32.548000 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:32.547971 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dh74s\" (UniqueName: \"kubernetes.io/projected/49f61653-f10c-4692-bc8f-a379cfd3b073-kube-api-access-dh74s\") pod \"payload-processing-external-model-5546676475-98nhd\" (UID: \"49f61653-f10c-4692-bc8f-a379cfd3b073\") " pod="models-as-a-service/payload-processing-external-model-5546676475-98nhd" Nov 03 13:32:32.669563 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:32.669525 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5546676475-98nhd" Nov 03 13:32:33.002308 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:33.001114 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5546676475-98nhd"] Nov 03 13:32:33.548401 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:33.548362 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5546676475-98nhd" event={"ID":"49f61653-f10c-4692-bc8f-a379cfd3b073","Type":"ContainerStarted","Data":"6b3ee2edf63c11e57730fbdd335caddd3e364aded43884df66dd1c1cbb610b8d"} Nov 03 13:32:33.548774 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:33.548452 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5546676475-98nhd" event={"ID":"49f61653-f10c-4692-bc8f-a379cfd3b073","Type":"ContainerStarted","Data":"d0aebfd5855b17467ebb3d0c7519ee23caba2b659b60d50afe608ee7176ed6bc"} Nov 03 13:32:33.548774 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:33.548506 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-5546676475-98nhd" Nov 03 13:32:33.577742 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:33.577666 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-5546676475-98nhd" podStartSLOduration=1.577644813 podStartE2EDuration="1.577644813s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:32:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:32:33.570923852 +0000 UTC m=+1433.579115173" watchObservedRunningTime="2026-11-03 13:32:33.577644813 +0000 UTC m=+1433.585836092" Nov 03 13:32:34.508109 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:34.508064 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-dd2bbf-openshift-default-7fd7f76f49-gf5kt"] Nov 03 13:32:34.548717 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:34.548676 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-dd2bbf-openshift-default-7fd7f76f49-gf5kt" Nov 03 13:32:34.557450 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:34.557417 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-dd2bbf-openshift-default-7fd7f76f49-gf5kt"] Nov 03 13:32:34.595261 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:34.595227 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-dd2bbf-openshift-default-dockercfg-kx9fd\"" Nov 03 13:32:34.805443 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:34.801635 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b6d7058-b5f5-4b35-a0ad-967187e9ba3b-istiod-ca-cert\") pod \"e2e-api-a-dd2bbf-openshift-default-7fd7f76f49-gf5kt\" (UID: \"8b6d7058-b5f5-4b35-a0ad-967187e9ba3b\") " pod="openshift-ingress/e2e-api-a-dd2bbf-openshift-default-7fd7f76f49-gf5kt" Nov 03 13:32:34.805443 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:34.801715 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b6d7058-b5f5-4b35-a0ad-967187e9ba3b-workload-socket\") pod \"e2e-api-a-dd2bbf-openshift-default-7fd7f76f49-gf5kt\" (UID: \"8b6d7058-b5f5-4b35-a0ad-967187e9ba3b\") " pod="openshift-ingress/e2e-api-a-dd2bbf-openshift-default-7fd7f76f49-gf5kt" Nov 03 13:32:34.805443 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:34.801745 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b6d7058-b5f5-4b35-a0ad-967187e9ba3b-workload-certs\") pod \"e2e-api-a-dd2bbf-openshift-default-7fd7f76f49-gf5kt\" (UID: \"8b6d7058-b5f5-4b35-a0ad-967187e9ba3b\") " pod="openshift-ingress/e2e-api-a-dd2bbf-openshift-default-7fd7f76f49-gf5kt" Nov 03 13:32:34.805443 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:34.801800 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b6d7058-b5f5-4b35-a0ad-967187e9ba3b-istio-envoy\") pod \"e2e-api-a-dd2bbf-openshift-default-7fd7f76f49-gf5kt\" (UID: \"8b6d7058-b5f5-4b35-a0ad-967187e9ba3b\") " pod="openshift-ingress/e2e-api-a-dd2bbf-openshift-default-7fd7f76f49-gf5kt" Nov 03 13:32:34.805443 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:34.801905 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b6d7058-b5f5-4b35-a0ad-967187e9ba3b-istio-podinfo\") pod \"e2e-api-a-dd2bbf-openshift-default-7fd7f76f49-gf5kt\" (UID: \"8b6d7058-b5f5-4b35-a0ad-967187e9ba3b\") " pod="openshift-ingress/e2e-api-a-dd2bbf-openshift-default-7fd7f76f49-gf5kt" Nov 03 13:32:34.805443 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:34.801973 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fgxqm\" (UniqueName: \"kubernetes.io/projected/8b6d7058-b5f5-4b35-a0ad-967187e9ba3b-kube-api-access-fgxqm\") pod \"e2e-api-a-dd2bbf-openshift-default-7fd7f76f49-gf5kt\" (UID: \"8b6d7058-b5f5-4b35-a0ad-967187e9ba3b\") " pod="openshift-ingress/e2e-api-a-dd2bbf-openshift-default-7fd7f76f49-gf5kt" Nov 03 13:32:34.805443 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:34.802135 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b6d7058-b5f5-4b35-a0ad-967187e9ba3b-istio-data\") pod \"e2e-api-a-dd2bbf-openshift-default-7fd7f76f49-gf5kt\" (UID: \"8b6d7058-b5f5-4b35-a0ad-967187e9ba3b\") " pod="openshift-ingress/e2e-api-a-dd2bbf-openshift-default-7fd7f76f49-gf5kt" Nov 03 13:32:34.805443 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:34.802206 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b6d7058-b5f5-4b35-a0ad-967187e9ba3b-istio-token\") pod \"e2e-api-a-dd2bbf-openshift-default-7fd7f76f49-gf5kt\" (UID: \"8b6d7058-b5f5-4b35-a0ad-967187e9ba3b\") " pod="openshift-ingress/e2e-api-a-dd2bbf-openshift-default-7fd7f76f49-gf5kt" Nov 03 13:32:34.805443 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:34.802324 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b6d7058-b5f5-4b35-a0ad-967187e9ba3b-credential-socket\") pod \"e2e-api-a-dd2bbf-openshift-default-7fd7f76f49-gf5kt\" (UID: \"8b6d7058-b5f5-4b35-a0ad-967187e9ba3b\") " pod="openshift-ingress/e2e-api-a-dd2bbf-openshift-default-7fd7f76f49-gf5kt" Nov 03 13:32:34.926309 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:34.912431 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b6d7058-b5f5-4b35-a0ad-967187e9ba3b-credential-socket\") pod \"e2e-api-a-dd2bbf-openshift-default-7fd7f76f49-gf5kt\" (UID: \"8b6d7058-b5f5-4b35-a0ad-967187e9ba3b\") " pod="openshift-ingress/e2e-api-a-dd2bbf-openshift-default-7fd7f76f49-gf5kt" Nov 03 13:32:34.935335 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:34.926699 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b6d7058-b5f5-4b35-a0ad-967187e9ba3b-istiod-ca-cert\") pod \"e2e-api-a-dd2bbf-openshift-default-7fd7f76f49-gf5kt\" (UID: \"8b6d7058-b5f5-4b35-a0ad-967187e9ba3b\") " pod="openshift-ingress/e2e-api-a-dd2bbf-openshift-default-7fd7f76f49-gf5kt" Nov 03 13:32:34.951304 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:34.935659 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b6d7058-b5f5-4b35-a0ad-967187e9ba3b-workload-socket\") pod \"e2e-api-a-dd2bbf-openshift-default-7fd7f76f49-gf5kt\" (UID: \"8b6d7058-b5f5-4b35-a0ad-967187e9ba3b\") " pod="openshift-ingress/e2e-api-a-dd2bbf-openshift-default-7fd7f76f49-gf5kt" Nov 03 13:32:34.951597 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:34.951565 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b6d7058-b5f5-4b35-a0ad-967187e9ba3b-workload-certs\") pod \"e2e-api-a-dd2bbf-openshift-default-7fd7f76f49-gf5kt\" (UID: \"8b6d7058-b5f5-4b35-a0ad-967187e9ba3b\") " pod="openshift-ingress/e2e-api-a-dd2bbf-openshift-default-7fd7f76f49-gf5kt" Nov 03 13:32:34.961397 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:34.951783 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b6d7058-b5f5-4b35-a0ad-967187e9ba3b-istio-envoy\") pod \"e2e-api-a-dd2bbf-openshift-default-7fd7f76f49-gf5kt\" (UID: \"8b6d7058-b5f5-4b35-a0ad-967187e9ba3b\") " pod="openshift-ingress/e2e-api-a-dd2bbf-openshift-default-7fd7f76f49-gf5kt" Nov 03 13:32:34.961740 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:34.947694 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b6d7058-b5f5-4b35-a0ad-967187e9ba3b-workload-socket\") pod \"e2e-api-a-dd2bbf-openshift-default-7fd7f76f49-gf5kt\" (UID: \"8b6d7058-b5f5-4b35-a0ad-967187e9ba3b\") " pod="openshift-ingress/e2e-api-a-dd2bbf-openshift-default-7fd7f76f49-gf5kt" Nov 03 13:32:34.967643 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:34.958719 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b6d7058-b5f5-4b35-a0ad-967187e9ba3b-workload-certs\") pod \"e2e-api-a-dd2bbf-openshift-default-7fd7f76f49-gf5kt\" (UID: \"8b6d7058-b5f5-4b35-a0ad-967187e9ba3b\") " pod="openshift-ingress/e2e-api-a-dd2bbf-openshift-default-7fd7f76f49-gf5kt" Nov 03 13:32:34.967643 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:34.921657 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b6d7058-b5f5-4b35-a0ad-967187e9ba3b-credential-socket\") pod \"e2e-api-a-dd2bbf-openshift-default-7fd7f76f49-gf5kt\" (UID: \"8b6d7058-b5f5-4b35-a0ad-967187e9ba3b\") " pod="openshift-ingress/e2e-api-a-dd2bbf-openshift-default-7fd7f76f49-gf5kt" Nov 03 13:32:34.967875 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:34.947418 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b6d7058-b5f5-4b35-a0ad-967187e9ba3b-istiod-ca-cert\") pod \"e2e-api-a-dd2bbf-openshift-default-7fd7f76f49-gf5kt\" (UID: \"8b6d7058-b5f5-4b35-a0ad-967187e9ba3b\") " pod="openshift-ingress/e2e-api-a-dd2bbf-openshift-default-7fd7f76f49-gf5kt" Nov 03 13:32:34.968256 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:34.968209 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b6d7058-b5f5-4b35-a0ad-967187e9ba3b-istio-podinfo\") pod \"e2e-api-a-dd2bbf-openshift-default-7fd7f76f49-gf5kt\" (UID: \"8b6d7058-b5f5-4b35-a0ad-967187e9ba3b\") " pod="openshift-ingress/e2e-api-a-dd2bbf-openshift-default-7fd7f76f49-gf5kt" Nov 03 13:32:34.968913 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:34.968890 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fgxqm\" (UniqueName: \"kubernetes.io/projected/8b6d7058-b5f5-4b35-a0ad-967187e9ba3b-kube-api-access-fgxqm\") pod \"e2e-api-a-dd2bbf-openshift-default-7fd7f76f49-gf5kt\" (UID: \"8b6d7058-b5f5-4b35-a0ad-967187e9ba3b\") " pod="openshift-ingress/e2e-api-a-dd2bbf-openshift-default-7fd7f76f49-gf5kt" Nov 03 13:32:34.969298 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:34.969248 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b6d7058-b5f5-4b35-a0ad-967187e9ba3b-istio-data\") pod \"e2e-api-a-dd2bbf-openshift-default-7fd7f76f49-gf5kt\" (UID: \"8b6d7058-b5f5-4b35-a0ad-967187e9ba3b\") " pod="openshift-ingress/e2e-api-a-dd2bbf-openshift-default-7fd7f76f49-gf5kt" Nov 03 13:32:34.969528 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:34.969502 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b6d7058-b5f5-4b35-a0ad-967187e9ba3b-istio-token\") pod \"e2e-api-a-dd2bbf-openshift-default-7fd7f76f49-gf5kt\" (UID: \"8b6d7058-b5f5-4b35-a0ad-967187e9ba3b\") " pod="openshift-ingress/e2e-api-a-dd2bbf-openshift-default-7fd7f76f49-gf5kt" Nov 03 13:32:34.970702 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:34.970665 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b6d7058-b5f5-4b35-a0ad-967187e9ba3b-istio-data\") pod \"e2e-api-a-dd2bbf-openshift-default-7fd7f76f49-gf5kt\" (UID: \"8b6d7058-b5f5-4b35-a0ad-967187e9ba3b\") " pod="openshift-ingress/e2e-api-a-dd2bbf-openshift-default-7fd7f76f49-gf5kt" Nov 03 13:32:34.971691 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:34.971667 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b6d7058-b5f5-4b35-a0ad-967187e9ba3b-istio-envoy\") pod \"e2e-api-a-dd2bbf-openshift-default-7fd7f76f49-gf5kt\" (UID: \"8b6d7058-b5f5-4b35-a0ad-967187e9ba3b\") " pod="openshift-ingress/e2e-api-a-dd2bbf-openshift-default-7fd7f76f49-gf5kt" Nov 03 13:32:34.983665 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:34.983628 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fgxqm\" (UniqueName: \"kubernetes.io/projected/8b6d7058-b5f5-4b35-a0ad-967187e9ba3b-kube-api-access-fgxqm\") pod \"e2e-api-a-dd2bbf-openshift-default-7fd7f76f49-gf5kt\" (UID: \"8b6d7058-b5f5-4b35-a0ad-967187e9ba3b\") " pod="openshift-ingress/e2e-api-a-dd2bbf-openshift-default-7fd7f76f49-gf5kt" Nov 03 13:32:34.987417 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:34.985341 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b6d7058-b5f5-4b35-a0ad-967187e9ba3b-istio-token\") pod \"e2e-api-a-dd2bbf-openshift-default-7fd7f76f49-gf5kt\" (UID: \"8b6d7058-b5f5-4b35-a0ad-967187e9ba3b\") " pod="openshift-ingress/e2e-api-a-dd2bbf-openshift-default-7fd7f76f49-gf5kt" Nov 03 13:32:35.018614 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:35.013136 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b6d7058-b5f5-4b35-a0ad-967187e9ba3b-istio-podinfo\") pod \"e2e-api-a-dd2bbf-openshift-default-7fd7f76f49-gf5kt\" (UID: \"8b6d7058-b5f5-4b35-a0ad-967187e9ba3b\") " pod="openshift-ingress/e2e-api-a-dd2bbf-openshift-default-7fd7f76f49-gf5kt" Nov 03 13:32:35.211194 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:35.211162 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-dd2bbf-openshift-default-7fd7f76f49-gf5kt" Nov 03 13:32:35.713135 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:35.713102 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-dd2bbf-openshift-default-7fd7f76f49-gf5kt"] Nov 03 13:32:35.715459 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:32:35.714497 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b6d7058_b5f5_4b35_a0ad_967187e9ba3b.slice/crio-d1b62e1e1b715516a5f50f87975e482e33021932574c00d999d2f4d717d00353 WatchSource:0}: Error finding container d1b62e1e1b715516a5f50f87975e482e33021932574c00d999d2f4d717d00353: Status 404 returned error can't find the container with id d1b62e1e1b715516a5f50f87975e482e33021932574c00d999d2f4d717d00353 Nov 03 13:32:35.717009 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:35.716977 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:32:35.717098 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:35.717049 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:32:35.717098 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:35.717079 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:32:37.052762 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:37.052714 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-dd2bbf-openshift-default-7fd7f76f49-gf5kt" event={"ID":"8b6d7058-b5f5-4b35-a0ad-967187e9ba3b","Type":"ContainerStarted","Data":"b52a553b51d07d9678f8fdc0f5d77dd37b70e785ca19068e555e14a4263fd9bf"} Nov 03 13:32:37.053420 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:37.053391 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-dd2bbf-openshift-default-7fd7f76f49-gf5kt" event={"ID":"8b6d7058-b5f5-4b35-a0ad-967187e9ba3b","Type":"ContainerStarted","Data":"d1b62e1e1b715516a5f50f87975e482e33021932574c00d999d2f4d717d00353"} Nov 03 13:32:37.144331 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:37.139063 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-dd2bbf-openshift-default-7fd7f76f49-gf5kt" podStartSLOduration=3.139036628 podStartE2EDuration="3.139036628s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:32:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:32:37.103074913 +0000 UTC m=+1437.111266218" watchObservedRunningTime="2026-11-03 13:32:37.139036628 +0000 UTC m=+1437.147227906" Nov 03 13:32:37.214240 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:37.214203 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-dd2bbf-openshift-default-7fd7f76f49-gf5kt" Nov 03 13:32:37.214744 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:37.214717 5024 patch_prober.go:29] interesting pod/e2e-api-a-dd2bbf-openshift-default-7fd7f76f49-gf5kt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body= Nov 03 13:32:37.214939 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:37.214911 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-dd2bbf-openshift-default-7fd7f76f49-gf5kt" podUID="8b6d7058-b5f5-4b35-a0ad-967187e9ba3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" Nov 03 13:32:37.493646 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:37.493600 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-08111b/premium-w1-08111b-kserve-57555d598-dx9g5" podUID="e45c569a-243f-49ab-8d74-dd08cbb6127c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.134:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 13:32:37.817361 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:37.815876 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d3ca48/premium-w3-d3ca48-kserve-75ffc959b-zvjbc" podUID="add1e8b9-e7fd-4e4d-8d10-f97d0e2612ba" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 13:32:37.963431 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:37.963381 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-08111b/simulator-w1-08111b-kserve-7f65c5867-btf5b" podUID="e8aba665-e4cc-4e67-8cdd-d09ef7542e15" containerName="main" probeResult="failure" output="Get \"https://10.132.0.133:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 13:32:38.221914 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:38.221868 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-dd2bbf-openshift-default-7fd7f76f49-gf5kt" Nov 03 13:32:39.215307 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:39.214507 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-dd2bbf-openshift-default-7fd7f76f49-gf5kt" Nov 03 13:32:39.242382 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:39.238903 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-dd2bbf-openshift-default-7fd7f76f49-gf5kt" Nov 03 13:32:39.952321 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:39.950346 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-08111b/simulator-w1-08111b-kserve-7f65c5867-btf5b" podUID="e8aba665-e4cc-4e67-8cdd-d09ef7542e15" containerName="main" probeResult="failure" output="Get \"https://10.132.0.133:8000/ready\": context deadline exceeded" Nov 03 13:32:40.144677 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:32:40.144633 5024 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d7mwz" containerName="extractor" Nov 03 13:32:40.677351 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:40.676521 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d3ca48/premium-w3-d3ca48-kserve-75ffc959b-zvjbc" podUID="add1e8b9-e7fd-4e4d-8d10-f97d0e2612ba" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 13:32:40.952492 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:40.952380 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-08111b/simulator-w1-08111b-kserve-7f65c5867-btf5b" podUID="e8aba665-e4cc-4e67-8cdd-d09ef7542e15" containerName="main" probeResult="failure" output="Get \"https://10.132.0.133:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 13:32:40.953593 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:40.953564 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-08111b/simulator-w1-08111b-kserve-7f65c5867-btf5b" Nov 03 13:32:41.169954 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:41.169845 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-08111b/simulator-w1-08111b-kserve-7f65c5867-btf5b" Nov 03 13:32:42.001309 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:41.985062 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-dd2bbf-74fb8cb9c9-95hxq"] Nov 03 13:32:42.306527 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:42.306433 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-dd2bbf-74fb8cb9c9-95hxq" Nov 03 13:32:42.315356 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:42.312959 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-dd2bbf\"" Nov 03 13:32:42.379958 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:42.379913 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-dd2bbf-74fb8cb9c9-95hxq"] Nov 03 13:32:42.380234 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:42.380219 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-dd2bbf-68fc6dd7d7-z55dw"] Nov 03 13:32:42.387968 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:42.387933 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8h2h\" (UniqueName: \"kubernetes.io/projected/50a9026b-b55f-4650-988a-755a758dca86-kube-api-access-q8h2h\") pod \"payload-processing-e2e-api-a-dd2bbf-74fb8cb9c9-95hxq\" (UID: \"50a9026b-b55f-4650-988a-755a758dca86\") " pod="openshift-ingress/payload-processing-e2e-api-a-dd2bbf-74fb8cb9c9-95hxq" Nov 03 13:32:42.388206 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:42.388186 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50a9026b-b55f-4650-988a-755a758dca86-plugins-config-volume\") pod \"payload-processing-e2e-api-a-dd2bbf-74fb8cb9c9-95hxq\" (UID: \"50a9026b-b55f-4650-988a-755a758dca86\") " pod="openshift-ingress/payload-processing-e2e-api-a-dd2bbf-74fb8cb9c9-95hxq" Nov 03 13:32:42.392370 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:42.392270 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-dd2bbf-68fc6dd7d7-z55dw"] Nov 03 13:32:42.392587 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:42.392346 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-dd2bbf-68fc6dd7d7-z55dw" Nov 03 13:32:42.404917 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:42.404885 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-dd2bbf-dockercfg-d4pvf\"" Nov 03 13:32:42.513228 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:42.500054 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8h2h\" (UniqueName: \"kubernetes.io/projected/50a9026b-b55f-4650-988a-755a758dca86-kube-api-access-q8h2h\") pod \"payload-processing-e2e-api-a-dd2bbf-74fb8cb9c9-95hxq\" (UID: \"50a9026b-b55f-4650-988a-755a758dca86\") " pod="openshift-ingress/payload-processing-e2e-api-a-dd2bbf-74fb8cb9c9-95hxq" Nov 03 13:32:42.513228 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:42.500202 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e553bdcf-e114-4046-8e10-4a2687d78a52-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-dd2bbf-68fc6dd7d7-z55dw\" (UID: \"e553bdcf-e114-4046-8e10-4a2687d78a52\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-dd2bbf-68fc6dd7d7-z55dw" Nov 03 13:32:42.513228 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:42.500277 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s25kn\" (UniqueName: \"kubernetes.io/projected/e553bdcf-e114-4046-8e10-4a2687d78a52-kube-api-access-s25kn\") pod \"payload-pre-processing-e2e-api-a-dd2bbf-68fc6dd7d7-z55dw\" (UID: \"e553bdcf-e114-4046-8e10-4a2687d78a52\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-dd2bbf-68fc6dd7d7-z55dw" Nov 03 13:32:42.513228 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:42.500415 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50a9026b-b55f-4650-988a-755a758dca86-plugins-config-volume\") pod \"payload-processing-e2e-api-a-dd2bbf-74fb8cb9c9-95hxq\" (UID: \"50a9026b-b55f-4650-988a-755a758dca86\") " pod="openshift-ingress/payload-processing-e2e-api-a-dd2bbf-74fb8cb9c9-95hxq" Nov 03 13:32:42.513228 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:42.501668 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50a9026b-b55f-4650-988a-755a758dca86-plugins-config-volume\") pod \"payload-processing-e2e-api-a-dd2bbf-74fb8cb9c9-95hxq\" (UID: \"50a9026b-b55f-4650-988a-755a758dca86\") " pod="openshift-ingress/payload-processing-e2e-api-a-dd2bbf-74fb8cb9c9-95hxq" Nov 03 13:32:42.559365 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:42.551933 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8h2h\" (UniqueName: \"kubernetes.io/projected/50a9026b-b55f-4650-988a-755a758dca86-kube-api-access-q8h2h\") pod \"payload-processing-e2e-api-a-dd2bbf-74fb8cb9c9-95hxq\" (UID: \"50a9026b-b55f-4650-988a-755a758dca86\") " pod="openshift-ingress/payload-processing-e2e-api-a-dd2bbf-74fb8cb9c9-95hxq" Nov 03 13:32:42.629200 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:42.620143 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e553bdcf-e114-4046-8e10-4a2687d78a52-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-dd2bbf-68fc6dd7d7-z55dw\" (UID: \"e553bdcf-e114-4046-8e10-4a2687d78a52\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-dd2bbf-68fc6dd7d7-z55dw" Nov 03 13:32:42.629200 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:42.620243 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s25kn\" (UniqueName: \"kubernetes.io/projected/e553bdcf-e114-4046-8e10-4a2687d78a52-kube-api-access-s25kn\") pod \"payload-pre-processing-e2e-api-a-dd2bbf-68fc6dd7d7-z55dw\" (UID: \"e553bdcf-e114-4046-8e10-4a2687d78a52\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-dd2bbf-68fc6dd7d7-z55dw" Nov 03 13:32:42.629200 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:42.621726 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e553bdcf-e114-4046-8e10-4a2687d78a52-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-dd2bbf-68fc6dd7d7-z55dw\" (UID: \"e553bdcf-e114-4046-8e10-4a2687d78a52\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-dd2bbf-68fc6dd7d7-z55dw" Nov 03 13:32:42.650187 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:42.650146 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s25kn\" (UniqueName: \"kubernetes.io/projected/e553bdcf-e114-4046-8e10-4a2687d78a52-kube-api-access-s25kn\") pod \"payload-pre-processing-e2e-api-a-dd2bbf-68fc6dd7d7-z55dw\" (UID: \"e553bdcf-e114-4046-8e10-4a2687d78a52\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-dd2bbf-68fc6dd7d7-z55dw" Nov 03 13:32:42.654509 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:42.651759 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-dd2bbf-74fb8cb9c9-95hxq" Nov 03 13:32:42.824376 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:42.824314 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-dd2bbf-68fc6dd7d7-z55dw" Nov 03 13:32:43.054333 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:43.054061 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-dd2bbf-74fb8cb9c9-95hxq"] Nov 03 13:32:43.058024 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:32:43.057996 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50a9026b_b55f_4650_988a_755a758dca86.slice/crio-a8d86ac2cad62f9016a50b3489e5af54cd2c98f0d501ead33dbec2a3625f853c WatchSource:0}: Error finding container a8d86ac2cad62f9016a50b3489e5af54cd2c98f0d501ead33dbec2a3625f853c: Status 404 returned error can't find the container with id a8d86ac2cad62f9016a50b3489e5af54cd2c98f0d501ead33dbec2a3625f853c Nov 03 13:32:43.164628 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:32:43.164595 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode553bdcf_e114_4046_8e10_4a2687d78a52.slice/crio-742758944e7cbbaf9c9a61f6373c95e1fa77612e66c0f1c68578ff45db932539 WatchSource:0}: Error finding container 742758944e7cbbaf9c9a61f6373c95e1fa77612e66c0f1c68578ff45db932539: Status 404 returned error can't find the container with id 742758944e7cbbaf9c9a61f6373c95e1fa77612e66c0f1c68578ff45db932539 Nov 03 13:32:43.165182 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:43.165134 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-dd2bbf-68fc6dd7d7-z55dw"] Nov 03 13:32:43.680025 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:43.679985 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-dd2bbf-68fc6dd7d7-z55dw" event={"ID":"e553bdcf-e114-4046-8e10-4a2687d78a52","Type":"ContainerStarted","Data":"fd9d2b375395c7849a5995911f32c3e63401eb36d89fe4146655c47e2a840f36"} Nov 03 13:32:43.680246 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:43.680048 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-dd2bbf-68fc6dd7d7-z55dw" event={"ID":"e553bdcf-e114-4046-8e10-4a2687d78a52","Type":"ContainerStarted","Data":"742758944e7cbbaf9c9a61f6373c95e1fa77612e66c0f1c68578ff45db932539"} Nov 03 13:32:43.680246 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:43.680076 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-dd2bbf-68fc6dd7d7-z55dw" Nov 03 13:32:43.682332 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:43.682273 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-dd2bbf-74fb8cb9c9-95hxq" event={"ID":"50a9026b-b55f-4650-988a-755a758dca86","Type":"ContainerStarted","Data":"c9aebf711b98782cfbb203dc381cc5769078119ced9b613f92b9e3a96d0983c0"} Nov 03 13:32:43.682476 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:43.682359 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-dd2bbf-74fb8cb9c9-95hxq" Nov 03 13:32:43.682476 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:43.682382 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-dd2bbf-74fb8cb9c9-95hxq" event={"ID":"50a9026b-b55f-4650-988a-755a758dca86","Type":"ContainerStarted","Data":"a8d86ac2cad62f9016a50b3489e5af54cd2c98f0d501ead33dbec2a3625f853c"} Nov 03 13:32:43.702171 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:43.702120 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-dd2bbf-68fc6dd7d7-z55dw" podStartSLOduration=2.702103583 podStartE2EDuration="2.702103583s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:32:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:32:43.699330306 +0000 UTC m=+1443.707521583" watchObservedRunningTime="2026-11-03 13:32:43.702103583 +0000 UTC m=+1443.710294859" Nov 03 13:32:43.720361 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:43.720252 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-dd2bbf-74fb8cb9c9-95hxq" podStartSLOduration=2.72023244 podStartE2EDuration="2.72023244s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:32:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:32:43.71861995 +0000 UTC m=+1443.726811227" watchObservedRunningTime="2026-11-03 13:32:43.72023244 +0000 UTC m=+1443.728423719" Nov 03 13:32:44.571982 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:44.571946 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-5546676475-98nhd" Nov 03 13:32:45.325127 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:45.325089 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-1ec4b892-openshift-default-6c8678f7c-prh98"] Nov 03 13:32:45.331577 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:45.331546 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-1ec4b892-openshift-default-6c8678f7c-prh98" Nov 03 13:32:45.336411 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:45.336380 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-1ec4b892-openshift-default-dockercfg-rm5mf\"" Nov 03 13:32:45.344268 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:45.344239 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-1ec4b892-openshift-default-6c8678f7c-prh98"] Nov 03 13:32:45.408695 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:45.408663 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zcplm\" (UniqueName: \"kubernetes.io/projected/752f4211-a41b-4b2e-b568-3b3bc1ce0989-kube-api-access-zcplm\") pod \"e2e-mt-1ec4b892-openshift-default-6c8678f7c-prh98\" (UID: \"752f4211-a41b-4b2e-b568-3b3bc1ce0989\") " pod="openshift-ingress/e2e-mt-1ec4b892-openshift-default-6c8678f7c-prh98" Nov 03 13:32:45.408885 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:45.408701 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/752f4211-a41b-4b2e-b568-3b3bc1ce0989-istio-data\") pod \"e2e-mt-1ec4b892-openshift-default-6c8678f7c-prh98\" (UID: \"752f4211-a41b-4b2e-b568-3b3bc1ce0989\") " pod="openshift-ingress/e2e-mt-1ec4b892-openshift-default-6c8678f7c-prh98" Nov 03 13:32:45.408885 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:45.408820 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/752f4211-a41b-4b2e-b568-3b3bc1ce0989-workload-certs\") pod \"e2e-mt-1ec4b892-openshift-default-6c8678f7c-prh98\" (UID: \"752f4211-a41b-4b2e-b568-3b3bc1ce0989\") " pod="openshift-ingress/e2e-mt-1ec4b892-openshift-default-6c8678f7c-prh98" Nov 03 13:32:45.408993 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:45.408912 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/752f4211-a41b-4b2e-b568-3b3bc1ce0989-istio-envoy\") pod \"e2e-mt-1ec4b892-openshift-default-6c8678f7c-prh98\" (UID: \"752f4211-a41b-4b2e-b568-3b3bc1ce0989\") " pod="openshift-ingress/e2e-mt-1ec4b892-openshift-default-6c8678f7c-prh98" Nov 03 13:32:45.408993 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:45.408960 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/752f4211-a41b-4b2e-b568-3b3bc1ce0989-istio-podinfo\") pod \"e2e-mt-1ec4b892-openshift-default-6c8678f7c-prh98\" (UID: \"752f4211-a41b-4b2e-b568-3b3bc1ce0989\") " pod="openshift-ingress/e2e-mt-1ec4b892-openshift-default-6c8678f7c-prh98" Nov 03 13:32:45.409092 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:45.408992 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/752f4211-a41b-4b2e-b568-3b3bc1ce0989-credential-socket\") pod \"e2e-mt-1ec4b892-openshift-default-6c8678f7c-prh98\" (UID: \"752f4211-a41b-4b2e-b568-3b3bc1ce0989\") " pod="openshift-ingress/e2e-mt-1ec4b892-openshift-default-6c8678f7c-prh98" Nov 03 13:32:45.409142 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:45.409110 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/752f4211-a41b-4b2e-b568-3b3bc1ce0989-istiod-ca-cert\") pod \"e2e-mt-1ec4b892-openshift-default-6c8678f7c-prh98\" (UID: \"752f4211-a41b-4b2e-b568-3b3bc1ce0989\") " pod="openshift-ingress/e2e-mt-1ec4b892-openshift-default-6c8678f7c-prh98" Nov 03 13:32:45.409201 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:45.409185 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/752f4211-a41b-4b2e-b568-3b3bc1ce0989-workload-socket\") pod \"e2e-mt-1ec4b892-openshift-default-6c8678f7c-prh98\" (UID: \"752f4211-a41b-4b2e-b568-3b3bc1ce0989\") " pod="openshift-ingress/e2e-mt-1ec4b892-openshift-default-6c8678f7c-prh98" Nov 03 13:32:45.409255 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:45.409222 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/752f4211-a41b-4b2e-b568-3b3bc1ce0989-istio-token\") pod \"e2e-mt-1ec4b892-openshift-default-6c8678f7c-prh98\" (UID: \"752f4211-a41b-4b2e-b568-3b3bc1ce0989\") " pod="openshift-ingress/e2e-mt-1ec4b892-openshift-default-6c8678f7c-prh98" Nov 03 13:32:45.532309 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:45.512507 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/752f4211-a41b-4b2e-b568-3b3bc1ce0989-istiod-ca-cert\") pod \"e2e-mt-1ec4b892-openshift-default-6c8678f7c-prh98\" (UID: \"752f4211-a41b-4b2e-b568-3b3bc1ce0989\") " pod="openshift-ingress/e2e-mt-1ec4b892-openshift-default-6c8678f7c-prh98" Nov 03 13:32:45.532669 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:45.532638 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/752f4211-a41b-4b2e-b568-3b3bc1ce0989-workload-socket\") pod \"e2e-mt-1ec4b892-openshift-default-6c8678f7c-prh98\" (UID: \"752f4211-a41b-4b2e-b568-3b3bc1ce0989\") " pod="openshift-ingress/e2e-mt-1ec4b892-openshift-default-6c8678f7c-prh98" Nov 03 13:32:45.532810 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:45.532795 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/752f4211-a41b-4b2e-b568-3b3bc1ce0989-istio-token\") pod \"e2e-mt-1ec4b892-openshift-default-6c8678f7c-prh98\" (UID: \"752f4211-a41b-4b2e-b568-3b3bc1ce0989\") " pod="openshift-ingress/e2e-mt-1ec4b892-openshift-default-6c8678f7c-prh98" Nov 03 13:32:45.533343 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:45.533315 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zcplm\" (UniqueName: \"kubernetes.io/projected/752f4211-a41b-4b2e-b568-3b3bc1ce0989-kube-api-access-zcplm\") pod \"e2e-mt-1ec4b892-openshift-default-6c8678f7c-prh98\" (UID: \"752f4211-a41b-4b2e-b568-3b3bc1ce0989\") " pod="openshift-ingress/e2e-mt-1ec4b892-openshift-default-6c8678f7c-prh98" Nov 03 13:32:45.534420 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:45.534394 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/752f4211-a41b-4b2e-b568-3b3bc1ce0989-istio-data\") pod \"e2e-mt-1ec4b892-openshift-default-6c8678f7c-prh98\" (UID: \"752f4211-a41b-4b2e-b568-3b3bc1ce0989\") " pod="openshift-ingress/e2e-mt-1ec4b892-openshift-default-6c8678f7c-prh98" Nov 03 13:32:45.534643 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:45.534625 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/752f4211-a41b-4b2e-b568-3b3bc1ce0989-workload-certs\") pod \"e2e-mt-1ec4b892-openshift-default-6c8678f7c-prh98\" (UID: \"752f4211-a41b-4b2e-b568-3b3bc1ce0989\") " pod="openshift-ingress/e2e-mt-1ec4b892-openshift-default-6c8678f7c-prh98" Nov 03 13:32:45.534756 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:45.534742 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/752f4211-a41b-4b2e-b568-3b3bc1ce0989-istio-envoy\") pod \"e2e-mt-1ec4b892-openshift-default-6c8678f7c-prh98\" (UID: \"752f4211-a41b-4b2e-b568-3b3bc1ce0989\") " pod="openshift-ingress/e2e-mt-1ec4b892-openshift-default-6c8678f7c-prh98" Nov 03 13:32:45.534850 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:45.534837 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/752f4211-a41b-4b2e-b568-3b3bc1ce0989-istio-podinfo\") pod \"e2e-mt-1ec4b892-openshift-default-6c8678f7c-prh98\" (UID: \"752f4211-a41b-4b2e-b568-3b3bc1ce0989\") " pod="openshift-ingress/e2e-mt-1ec4b892-openshift-default-6c8678f7c-prh98" Nov 03 13:32:45.534945 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:45.534933 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/752f4211-a41b-4b2e-b568-3b3bc1ce0989-credential-socket\") pod \"e2e-mt-1ec4b892-openshift-default-6c8678f7c-prh98\" (UID: \"752f4211-a41b-4b2e-b568-3b3bc1ce0989\") " pod="openshift-ingress/e2e-mt-1ec4b892-openshift-default-6c8678f7c-prh98" Nov 03 13:32:45.535492 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:45.535473 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/752f4211-a41b-4b2e-b568-3b3bc1ce0989-credential-socket\") pod \"e2e-mt-1ec4b892-openshift-default-6c8678f7c-prh98\" (UID: \"752f4211-a41b-4b2e-b568-3b3bc1ce0989\") " pod="openshift-ingress/e2e-mt-1ec4b892-openshift-default-6c8678f7c-prh98" Nov 03 13:32:45.535620 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:45.533815 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/752f4211-a41b-4b2e-b568-3b3bc1ce0989-workload-socket\") pod \"e2e-mt-1ec4b892-openshift-default-6c8678f7c-prh98\" (UID: \"752f4211-a41b-4b2e-b568-3b3bc1ce0989\") " pod="openshift-ingress/e2e-mt-1ec4b892-openshift-default-6c8678f7c-prh98" Nov 03 13:32:45.536171 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:45.536152 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/752f4211-a41b-4b2e-b568-3b3bc1ce0989-istio-data\") pod \"e2e-mt-1ec4b892-openshift-default-6c8678f7c-prh98\" (UID: \"752f4211-a41b-4b2e-b568-3b3bc1ce0989\") " pod="openshift-ingress/e2e-mt-1ec4b892-openshift-default-6c8678f7c-prh98" Nov 03 13:32:45.536302 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:45.529978 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/752f4211-a41b-4b2e-b568-3b3bc1ce0989-istiod-ca-cert\") pod \"e2e-mt-1ec4b892-openshift-default-6c8678f7c-prh98\" (UID: \"752f4211-a41b-4b2e-b568-3b3bc1ce0989\") " pod="openshift-ingress/e2e-mt-1ec4b892-openshift-default-6c8678f7c-prh98" Nov 03 13:32:45.536753 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:45.536733 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/752f4211-a41b-4b2e-b568-3b3bc1ce0989-workload-certs\") pod \"e2e-mt-1ec4b892-openshift-default-6c8678f7c-prh98\" (UID: \"752f4211-a41b-4b2e-b568-3b3bc1ce0989\") " pod="openshift-ingress/e2e-mt-1ec4b892-openshift-default-6c8678f7c-prh98" Nov 03 13:32:45.573347 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:45.570884 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/752f4211-a41b-4b2e-b568-3b3bc1ce0989-istio-envoy\") pod \"e2e-mt-1ec4b892-openshift-default-6c8678f7c-prh98\" (UID: \"752f4211-a41b-4b2e-b568-3b3bc1ce0989\") " pod="openshift-ingress/e2e-mt-1ec4b892-openshift-default-6c8678f7c-prh98" Nov 03 13:32:45.596373 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:45.593383 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/752f4211-a41b-4b2e-b568-3b3bc1ce0989-istio-podinfo\") pod \"e2e-mt-1ec4b892-openshift-default-6c8678f7c-prh98\" (UID: \"752f4211-a41b-4b2e-b568-3b3bc1ce0989\") " pod="openshift-ingress/e2e-mt-1ec4b892-openshift-default-6c8678f7c-prh98" Nov 03 13:32:45.627129 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:45.618689 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zcplm\" (UniqueName: \"kubernetes.io/projected/752f4211-a41b-4b2e-b568-3b3bc1ce0989-kube-api-access-zcplm\") pod \"e2e-mt-1ec4b892-openshift-default-6c8678f7c-prh98\" (UID: \"752f4211-a41b-4b2e-b568-3b3bc1ce0989\") " pod="openshift-ingress/e2e-mt-1ec4b892-openshift-default-6c8678f7c-prh98" Nov 03 13:32:45.627129 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:45.626900 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/752f4211-a41b-4b2e-b568-3b3bc1ce0989-istio-token\") pod \"e2e-mt-1ec4b892-openshift-default-6c8678f7c-prh98\" (UID: \"752f4211-a41b-4b2e-b568-3b3bc1ce0989\") " pod="openshift-ingress/e2e-mt-1ec4b892-openshift-default-6c8678f7c-prh98" Nov 03 13:32:45.679712 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:45.679678 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-1ec4b892-openshift-default-6c8678f7c-prh98" Nov 03 13:32:46.206736 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:46.206611 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-1ec4b892-openshift-default-6c8678f7c-prh98"] Nov 03 13:32:46.207137 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:32:46.207107 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod752f4211_a41b_4b2e_b568_3b3bc1ce0989.slice/crio-e7f9accae9d900ef81ca1b69ed184e3ea1b053524e94297d63d22fa9c68c6133 WatchSource:0}: Error finding container e7f9accae9d900ef81ca1b69ed184e3ea1b053524e94297d63d22fa9c68c6133: Status 404 returned error can't find the container with id e7f9accae9d900ef81ca1b69ed184e3ea1b053524e94297d63d22fa9c68c6133 Nov 03 13:32:46.210970 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:46.210919 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:32:46.211074 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:46.211018 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:32:46.211074 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:46.211066 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:32:46.878457 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:46.878411 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-1ec4b892-openshift-default-6c8678f7c-prh98" event={"ID":"752f4211-a41b-4b2e-b568-3b3bc1ce0989","Type":"ContainerStarted","Data":"7724369ad432b2c96a54f3a089844dd113f77e4ccf6f578ac8e8719894f4ee45"} Nov 03 13:32:46.878864 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:46.878474 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-1ec4b892-openshift-default-6c8678f7c-prh98" event={"ID":"752f4211-a41b-4b2e-b568-3b3bc1ce0989","Type":"ContainerStarted","Data":"e7f9accae9d900ef81ca1b69ed184e3ea1b053524e94297d63d22fa9c68c6133"} Nov 03 13:32:46.907874 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:46.907814 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-1ec4b892-openshift-default-6c8678f7c-prh98" podStartSLOduration=1.907799813 podStartE2EDuration="1.907799813s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:32:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:32:46.903571413 +0000 UTC m=+1446.911762694" watchObservedRunningTime="2026-11-03 13:32:46.907799813 +0000 UTC m=+1446.915991090" Nov 03 13:32:47.680148 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:47.680109 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-1ec4b892-openshift-default-6c8678f7c-prh98" Nov 03 13:32:47.692648 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:47.692616 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-1ec4b892-openshift-default-6c8678f7c-prh98" Nov 03 13:32:47.903581 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:47.903548 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-1ec4b892-openshift-default-6c8678f7c-prh98" Nov 03 13:32:47.904856 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:47.904830 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-1ec4b892-openshift-default-6c8678f7c-prh98" Nov 03 13:32:54.698423 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:54.698392 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-dd2bbf-68fc6dd7d7-z55dw" Nov 03 13:32:54.698974 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:32:54.698825 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-dd2bbf-74fb8cb9c9-95hxq" Nov 03 13:33:01.292312 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:01.280077 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-7554n79pc"] Nov 03 13:33:04.490308 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:04.489656 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-08111b/premium-w1-08111b-kserve-57555d598-dx9g5" podUID="e45c569a-243f-49ab-8d74-dd08cbb6127c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.134:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 13:33:04.691502 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:04.691441 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d3ca48/premium-w3-d3ca48-kserve-75ffc959b-zvjbc" podUID="add1e8b9-e7fd-4e4d-8d10-f97d0e2612ba" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded" Nov 03 13:33:05.694310 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:05.693380 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d3ca48/premium-w3-d3ca48-kserve-75ffc959b-zvjbc" podUID="add1e8b9-e7fd-4e4d-8d10-f97d0e2612ba" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 13:33:05.694310 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:05.693900 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-d3ca48/premium-w3-d3ca48-kserve-75ffc959b-zvjbc" Nov 03 13:33:05.730406 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:05.730372 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-d3ca48/premium-w3-d3ca48-kserve-75ffc959b-zvjbc" Nov 03 13:33:06.224537 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:06.224492 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-08111b/simulator-w1-08111b-kserve-7f65c5867-btf5b" podUID="e8aba665-e4cc-4e67-8cdd-d09ef7542e15" containerName="main" probeResult="failure" output="Get \"https://10.132.0.133:8000/ready\": context deadline exceeded" Nov 03 13:33:06.368319 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:06.356920 5024 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-7554n79pc" podUID="aaad66b5-4b6d-4f05-a2f9-a8d34e6036a4" containerName="payload-processing" containerID="cri-o://f2e2cf6705e996b618b47c9fc81e82626dc9852d81cfaace9136ae8f0b093ace" gracePeriod=25 Nov 03 13:33:06.492763 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:06.492667 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-08111b/premium-w1-08111b-kserve-57555d598-dx9g5" podUID="e45c569a-243f-49ab-8d74-dd08cbb6127c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.134:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 13:33:06.823360 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:06.815882 5024 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-h2ns8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 13:33:06.823360 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:06.815952 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-h2ns8" podUID="155ba865-dcfd-4203-82b9-5399ca58d988" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 13:33:06.840442 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:06.840393 5024 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-d3ca48/premium-w3-d3ca48-kserve-75ffc959b-zvjbc" podUID="add1e8b9-e7fd-4e4d-8d10-f97d0e2612ba" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 13:33:07.518559 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:07.518511 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-08111b/premium-w1-08111b-kserve-57555d598-dx9g5" podUID="e45c569a-243f-49ab-8d74-dd08cbb6127c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.134:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 13:33:07.549162 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:07.549125 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-08111b/premium-w1-08111b-kserve-57555d598-dx9g5" Nov 03 13:33:07.786483 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:07.784164 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d3ca48/premium-w3-d3ca48-kserve-75ffc959b-zvjbc" podUID="add1e8b9-e7fd-4e4d-8d10-f97d0e2612ba" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded" Nov 03 13:33:08.199957 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:08.185487 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-08111b/simulator-w1-08111b-kserve-7f65c5867-btf5b" podUID="e8aba665-e4cc-4e67-8cdd-d09ef7542e15" containerName="main" probeResult="failure" output="Get \"https://10.132.0.133:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 13:33:08.405393 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:08.405347 5024 generic.go:356] "Generic (PLEG): container finished" podID="aaad66b5-4b6d-4f05-a2f9-a8d34e6036a4" containerID="f2e2cf6705e996b618b47c9fc81e82626dc9852d81cfaace9136ae8f0b093ace" exitCode=0 Nov 03 13:33:08.405887 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:08.405860 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-7554n79pc" event={"ID":"aaad66b5-4b6d-4f05-a2f9-a8d34e6036a4","Type":"ContainerDied","Data":"f2e2cf6705e996b618b47c9fc81e82626dc9852d81cfaace9136ae8f0b093ace"} Nov 03 13:33:08.852308 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:08.844589 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-08111b/premium-w1-08111b-kserve-57555d598-dx9g5" podUID="e45c569a-243f-49ab-8d74-dd08cbb6127c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.134:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 13:33:09.401015 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:09.400960 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-08111b/simulator-w1-08111b-kserve-7f65c5867-btf5b" podUID="e8aba665-e4cc-4e67-8cdd-d09ef7542e15" containerName="main" probeResult="failure" output="Get \"https://10.132.0.133:8000/ready\": context deadline exceeded" Nov 03 13:33:09.402582 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:09.402556 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-08111b/simulator-w1-08111b-kserve-7f65c5867-btf5b" Nov 03 13:33:09.510577 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:09.508227 5024 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-nr5hn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 13:33:09.527778 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:09.527729 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-xpkhb" podUID="b0c8b366-13df-493b-a519-10797d27e14c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.39:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 13:33:09.528507 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:09.528466 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-q6cdk" podUID="510862f1-3997-4a93-b4a9-75e3855154dc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.40:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 13:33:09.550856 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:09.550806 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-nr5hn" podUID="3cd0d23e-9823-4988-ac79-0820dcc4c028" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 13:33:09.930127 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:09.901456 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-08111b/premium-w1-08111b-kserve-57555d598-dx9g5" Nov 03 13:33:10.077017 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:33:09.958720 5024 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d7mwz" containerName="extractor" Nov 03 13:33:10.077017 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:09.958820 5024 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8h566d" podUID="6428a275-67b4-43e6-be9c-a16728516866" containerName="main" probeResult="failure" output="Get \"https://10.132.0.41:8000/health\": context deadline exceeded" Nov 03 13:33:10.421579 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:10.421438 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-08111b/simulator-w1-08111b-kserve-7f65c5867-btf5b" podUID="e8aba665-e4cc-4e67-8cdd-d09ef7542e15" containerName="main" probeResult="failure" output="Get \"https://10.132.0.133:8000/ready\": context deadline exceeded" Nov 03 13:33:10.615690 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:10.615589 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-08111b/simulator-w1-08111b-kserve-7f65c5867-btf5b" Nov 03 13:33:10.640878 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:10.640844 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-7554n79pc" Nov 03 13:33:10.763317 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:10.740410 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d3ca48/premium-w3-d3ca48-kserve-75ffc959b-zvjbc" podUID="add1e8b9-e7fd-4e4d-8d10-f97d0e2612ba" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded" Nov 03 13:33:10.796124 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:10.795236 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aaad66b5-4b6d-4f05-a2f9-a8d34e6036a4-provider-credentials\" (UniqueName: \"kubernetes.io/projected/aaad66b5-4b6d-4f05-a2f9-a8d34e6036a4-provider-credentials\") pod \"aaad66b5-4b6d-4f05-a2f9-a8d34e6036a4\" (UID: \"aaad66b5-4b6d-4f05-a2f9-a8d34e6036a4\") " Nov 03 13:33:10.796124 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:10.795307 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aaad66b5-4b6d-4f05-a2f9-a8d34e6036a4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aaad66b5-4b6d-4f05-a2f9-a8d34e6036a4-plugins-config-volume\") pod \"aaad66b5-4b6d-4f05-a2f9-a8d34e6036a4\" (UID: \"aaad66b5-4b6d-4f05-a2f9-a8d34e6036a4\") " Nov 03 13:33:10.796124 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:10.795405 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aaad66b5-4b6d-4f05-a2f9-a8d34e6036a4-kube-api-access-2hzmw\" (UniqueName: \"kubernetes.io/projected/aaad66b5-4b6d-4f05-a2f9-a8d34e6036a4-kube-api-access-2hzmw\") pod \"aaad66b5-4b6d-4f05-a2f9-a8d34e6036a4\" (UID: \"aaad66b5-4b6d-4f05-a2f9-a8d34e6036a4\") " Nov 03 13:33:10.796124 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:10.795629 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aaad66b5-4b6d-4f05-a2f9-a8d34e6036a4-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/aaad66b5-4b6d-4f05-a2f9-a8d34e6036a4-routing-overlay-volume\") pod \"aaad66b5-4b6d-4f05-a2f9-a8d34e6036a4\" (UID: \"aaad66b5-4b6d-4f05-a2f9-a8d34e6036a4\") " Nov 03 13:33:10.856916 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:10.856881 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aaad66b5-4b6d-4f05-a2f9-a8d34e6036a4-plugins-config-volume" pod "aaad66b5-4b6d-4f05-a2f9-a8d34e6036a4" (UID: "aaad66b5-4b6d-4f05-a2f9-a8d34e6036a4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:33:10.868410 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:10.862661 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aaad66b5-4b6d-4f05-a2f9-a8d34e6036a4-routing-overlay-volume" pod "aaad66b5-4b6d-4f05-a2f9-a8d34e6036a4" (UID: "aaad66b5-4b6d-4f05-a2f9-a8d34e6036a4"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:33:10.984308 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:10.956548 5024 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/aaad66b5-4b6d-4f05-a2f9-a8d34e6036a4-routing-overlay-volume\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:33:10.984308 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:10.956585 5024 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aaad66b5-4b6d-4f05-a2f9-a8d34e6036a4-plugins-config-volume\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:33:11.148799 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:11.148634 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aaad66b5-4b6d-4f05-a2f9-a8d34e6036a4-kube-api-access-2hzmw" pod "aaad66b5-4b6d-4f05-a2f9-a8d34e6036a4" (UID: "aaad66b5-4b6d-4f05-a2f9-a8d34e6036a4"). InnerVolumeSpecName "kube-api-access-2hzmw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:33:11.149008 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:11.148818 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aaad66b5-4b6d-4f05-a2f9-a8d34e6036a4-provider-credentials" pod "aaad66b5-4b6d-4f05-a2f9-a8d34e6036a4" (UID: "aaad66b5-4b6d-4f05-a2f9-a8d34e6036a4"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:33:11.239106 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:11.238203 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2hzmw\" (UniqueName: \"kubernetes.io/projected/aaad66b5-4b6d-4f05-a2f9-a8d34e6036a4-kube-api-access-2hzmw\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:33:11.239106 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:11.238234 5024 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/aaad66b5-4b6d-4f05-a2f9-a8d34e6036a4-provider-credentials\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:33:11.601366 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:11.601232 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-418af8-55df5c84df-vjqhm"] Nov 03 13:33:11.622258 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:11.622227 5024 cpu_manager.go:414] "containerMap: removing container" podUID="aaad66b5-4b6d-4f05-a2f9-a8d34e6036a4" containerName="payload-processing" Nov 03 13:33:11.622258 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:11.622253 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aaad66b5-4b6d-4f05-a2f9-a8d34e6036a4" containerName="payload-processing" Nov 03 13:33:11.622522 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:11.622453 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="aaad66b5-4b6d-4f05-a2f9-a8d34e6036a4" containerName="payload-processing" Nov 03 13:33:11.628342 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:11.628312 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-418af8-55df5c84df-vjqhm"] Nov 03 13:33:11.628486 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:11.628440 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-418af8-55df5c84df-vjqhm" Nov 03 13:33:11.636106 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:11.636067 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-7554n79pc" Nov 03 13:33:11.636298 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:11.636245 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-7554n79pc" event={"ID":"aaad66b5-4b6d-4f05-a2f9-a8d34e6036a4","Type":"ContainerDied","Data":"133deb9ea6077ec3de907940382721cd53ca2a47ba3c60045098c1db4ae3e470"} Nov 03 13:33:11.636383 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:11.636332 5024 scope.go:122] "RemoveContainer" containerID="f2e2cf6705e996b618b47c9fc81e82626dc9852d81cfaace9136ae8f0b093ace" Nov 03 13:33:11.654968 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:11.654917 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-418af8\"" Nov 03 13:33:11.655138 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:11.655022 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-418af8-dockercfg-9ppxq\"" Nov 03 13:33:11.724472 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:11.724428 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-7554n79pc"] Nov 03 13:33:11.833309 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:11.755376 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jt57d\" (UniqueName: \"kubernetes.io/projected/1f4853bc-d6b2-40dd-b779-ddb415176c1a-kube-api-access-jt57d\") pod \"payload-processing-e2e-api-b-418af8-55df5c84df-vjqhm\" (UID: \"1f4853bc-d6b2-40dd-b779-ddb415176c1a\") " pod="openshift-ingress/payload-processing-e2e-api-b-418af8-55df5c84df-vjqhm" Nov 03 13:33:11.833870 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:11.833839 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f4853bc-d6b2-40dd-b779-ddb415176c1a-plugins-config-volume\") pod \"payload-processing-e2e-api-b-418af8-55df5c84df-vjqhm\" (UID: \"1f4853bc-d6b2-40dd-b779-ddb415176c1a\") " pod="openshift-ingress/payload-processing-e2e-api-b-418af8-55df5c84df-vjqhm" Nov 03 13:33:11.836023 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:11.835981 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-a62d22/payload-processing-external-model-e2e-extmod-a-a62d22-7554n79pc"] Nov 03 13:33:11.937435 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:11.937388 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jt57d\" (UniqueName: \"kubernetes.io/projected/1f4853bc-d6b2-40dd-b779-ddb415176c1a-kube-api-access-jt57d\") pod \"payload-processing-e2e-api-b-418af8-55df5c84df-vjqhm\" (UID: \"1f4853bc-d6b2-40dd-b779-ddb415176c1a\") " pod="openshift-ingress/payload-processing-e2e-api-b-418af8-55df5c84df-vjqhm" Nov 03 13:33:11.963603 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:11.963490 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f4853bc-d6b2-40dd-b779-ddb415176c1a-plugins-config-volume\") pod \"payload-processing-e2e-api-b-418af8-55df5c84df-vjqhm\" (UID: \"1f4853bc-d6b2-40dd-b779-ddb415176c1a\") " pod="openshift-ingress/payload-processing-e2e-api-b-418af8-55df5c84df-vjqhm" Nov 03 13:33:11.967717 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:11.965300 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f4853bc-d6b2-40dd-b779-ddb415176c1a-plugins-config-volume\") pod \"payload-processing-e2e-api-b-418af8-55df5c84df-vjqhm\" (UID: \"1f4853bc-d6b2-40dd-b779-ddb415176c1a\") " pod="openshift-ingress/payload-processing-e2e-api-b-418af8-55df5c84df-vjqhm" Nov 03 13:33:12.149307 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:12.122235 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jt57d\" (UniqueName: \"kubernetes.io/projected/1f4853bc-d6b2-40dd-b779-ddb415176c1a-kube-api-access-jt57d\") pod \"payload-processing-e2e-api-b-418af8-55df5c84df-vjqhm\" (UID: \"1f4853bc-d6b2-40dd-b779-ddb415176c1a\") " pod="openshift-ingress/payload-processing-e2e-api-b-418af8-55df5c84df-vjqhm" Nov 03 13:33:12.264765 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:12.264680 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aaad66b5-4b6d-4f05-a2f9-a8d34e6036a4" path="/var/lib/kubelet/pods/aaad66b5-4b6d-4f05-a2f9-a8d34e6036a4/volumes" Nov 03 13:33:12.265782 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:12.265758 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-418af8-55df5c84df-vjqhm" Nov 03 13:33:12.667101 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:12.667061 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-418af8-55df5c84df-vjqhm"] Nov 03 13:33:13.669003 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:13.668958 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-418af8-55df5c84df-vjqhm" event={"ID":"1f4853bc-d6b2-40dd-b779-ddb415176c1a","Type":"ContainerStarted","Data":"4e2c6b771e344aeda062b8846a308ab6631a210cd71f874fe821abe192d7be3d"} Nov 03 13:33:13.669474 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:13.669034 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-418af8-55df5c84df-vjqhm" event={"ID":"1f4853bc-d6b2-40dd-b779-ddb415176c1a","Type":"ContainerStarted","Data":"4e005f906d60298e3879b88a2a89f45337d826848c932c10e4765f20cd7dff1d"} Nov 03 13:33:13.669474 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:13.669067 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-418af8-55df5c84df-vjqhm" Nov 03 13:33:13.689694 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:13.689621 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-418af8-55df5c84df-vjqhm" podStartSLOduration=2.689599417 podStartE2EDuration="2.689599417s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:33:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:33:13.687747886 +0000 UTC m=+1473.695939176" watchObservedRunningTime="2026-11-03 13:33:13.689599417 +0000 UTC m=+1473.697790695" Nov 03 13:33:18.825357 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:18.823633 5024 generic.go:356] "Generic (PLEG): container finished" podID="8b6d7058-b5f5-4b35-a0ad-967187e9ba3b" containerID="b52a553b51d07d9678f8fdc0f5d77dd37b70e785ca19068e555e14a4263fd9bf" exitCode=0 Nov 03 13:33:18.825357 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:18.823710 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-dd2bbf-openshift-default-7fd7f76f49-gf5kt" event={"ID":"8b6d7058-b5f5-4b35-a0ad-967187e9ba3b","Type":"ContainerDied","Data":"b52a553b51d07d9678f8fdc0f5d77dd37b70e785ca19068e555e14a4263fd9bf"} Nov 03 13:33:18.825357 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:18.824362 5024 scope.go:122] "RemoveContainer" containerID="b52a553b51d07d9678f8fdc0f5d77dd37b70e785ca19068e555e14a4263fd9bf" Nov 03 13:33:18.826717 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:18.826658 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:33:18.826859 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:18.826772 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:33:18.826859 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:18.826823 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:33:18.857803 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:18.857770 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-4m57d"] Nov 03 13:33:18.860326 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:18.858030 5024 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-4m57d" podUID="2ca8b6af-59bf-4d09-ba65-9d79a4e2faba" containerName="istio-proxy" containerID="cri-o://f32398773d95c537adc89160133a4f7cc75a42791b497690b585e6fde7e0a3ad" gracePeriod=30 Nov 03 13:33:19.852146 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:19.852102 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-dd2bbf-openshift-default-7fd7f76f49-gf5kt" event={"ID":"8b6d7058-b5f5-4b35-a0ad-967187e9ba3b","Type":"ContainerStarted","Data":"d722e94bf1775221ae03d4eca896479d933d868d6f033c2f6fa7a7e0e48fc120"} Nov 03 13:33:20.211813 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:20.211722 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-dd2bbf-openshift-default-7fd7f76f49-gf5kt" Nov 03 13:33:20.213076 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:20.213045 5024 patch_prober.go:29] interesting pod/e2e-api-a-dd2bbf-openshift-default-7fd7f76f49-gf5kt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body= Nov 03 13:33:20.213199 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:20.213099 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-dd2bbf-openshift-default-7fd7f76f49-gf5kt" podUID="8b6d7058-b5f5-4b35-a0ad-967187e9ba3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" Nov 03 13:33:20.714638 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:20.714597 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-418af8-9dd6d7ffd-6wx9j"] Nov 03 13:33:20.719821 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:20.719792 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-418af8-9dd6d7ffd-6wx9j" Nov 03 13:33:20.732746 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:20.732708 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-418af8-9dd6d7ffd-6wx9j"] Nov 03 13:33:20.736170 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:20.736142 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-418af8-serving-cert\"" Nov 03 13:33:20.813740 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:20.813687 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z9s9x\" (UniqueName: \"kubernetes.io/projected/7adbc4a3-4489-45fd-a5d6-1b83a240af2a-kube-api-access-z9s9x\") pod \"maas-api-e2e-api-b-418af8-9dd6d7ffd-6wx9j\" (UID: \"7adbc4a3-4489-45fd-a5d6-1b83a240af2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-418af8-9dd6d7ffd-6wx9j" Nov 03 13:33:20.813943 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:20.813777 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7adbc4a3-4489-45fd-a5d6-1b83a240af2a-metrics-certs\") pod \"maas-api-e2e-api-b-418af8-9dd6d7ffd-6wx9j\" (UID: \"7adbc4a3-4489-45fd-a5d6-1b83a240af2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-418af8-9dd6d7ffd-6wx9j" Nov 03 13:33:20.814052 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:20.814032 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7adbc4a3-4489-45fd-a5d6-1b83a240af2a-maas-api-tls\") pod \"maas-api-e2e-api-b-418af8-9dd6d7ffd-6wx9j\" (UID: \"7adbc4a3-4489-45fd-a5d6-1b83a240af2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-418af8-9dd6d7ffd-6wx9j" Nov 03 13:33:20.814142 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:20.814126 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7adbc4a3-4489-45fd-a5d6-1b83a240af2a-openshift-service-ca\") pod \"maas-api-e2e-api-b-418af8-9dd6d7ffd-6wx9j\" (UID: \"7adbc4a3-4489-45fd-a5d6-1b83a240af2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-418af8-9dd6d7ffd-6wx9j" Nov 03 13:33:20.932638 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:20.931212 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7adbc4a3-4489-45fd-a5d6-1b83a240af2a-maas-api-tls\") pod \"maas-api-e2e-api-b-418af8-9dd6d7ffd-6wx9j\" (UID: \"7adbc4a3-4489-45fd-a5d6-1b83a240af2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-418af8-9dd6d7ffd-6wx9j" Nov 03 13:33:20.932638 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:20.931351 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7adbc4a3-4489-45fd-a5d6-1b83a240af2a-openshift-service-ca\") pod \"maas-api-e2e-api-b-418af8-9dd6d7ffd-6wx9j\" (UID: \"7adbc4a3-4489-45fd-a5d6-1b83a240af2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-418af8-9dd6d7ffd-6wx9j" Nov 03 13:33:20.932638 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:20.931513 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z9s9x\" (UniqueName: \"kubernetes.io/projected/7adbc4a3-4489-45fd-a5d6-1b83a240af2a-kube-api-access-z9s9x\") pod \"maas-api-e2e-api-b-418af8-9dd6d7ffd-6wx9j\" (UID: \"7adbc4a3-4489-45fd-a5d6-1b83a240af2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-418af8-9dd6d7ffd-6wx9j" Nov 03 13:33:20.932638 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:20.931586 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7adbc4a3-4489-45fd-a5d6-1b83a240af2a-metrics-certs\") pod \"maas-api-e2e-api-b-418af8-9dd6d7ffd-6wx9j\" (UID: \"7adbc4a3-4489-45fd-a5d6-1b83a240af2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-418af8-9dd6d7ffd-6wx9j" Nov 03 13:33:20.999154 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:20.963806 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7adbc4a3-4489-45fd-a5d6-1b83a240af2a-metrics-certs\") pod \"maas-api-e2e-api-b-418af8-9dd6d7ffd-6wx9j\" (UID: \"7adbc4a3-4489-45fd-a5d6-1b83a240af2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-418af8-9dd6d7ffd-6wx9j" Nov 03 13:33:21.035757 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:20.988236 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7adbc4a3-4489-45fd-a5d6-1b83a240af2a-openshift-service-ca\") pod \"maas-api-e2e-api-b-418af8-9dd6d7ffd-6wx9j\" (UID: \"7adbc4a3-4489-45fd-a5d6-1b83a240af2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-418af8-9dd6d7ffd-6wx9j" Nov 03 13:33:21.049543 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:21.049506 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7adbc4a3-4489-45fd-a5d6-1b83a240af2a-maas-api-tls\") pod \"maas-api-e2e-api-b-418af8-9dd6d7ffd-6wx9j\" (UID: \"7adbc4a3-4489-45fd-a5d6-1b83a240af2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-418af8-9dd6d7ffd-6wx9j" Nov 03 13:33:21.053310 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:21.053025 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z9s9x\" (UniqueName: \"kubernetes.io/projected/7adbc4a3-4489-45fd-a5d6-1b83a240af2a-kube-api-access-z9s9x\") pod \"maas-api-e2e-api-b-418af8-9dd6d7ffd-6wx9j\" (UID: \"7adbc4a3-4489-45fd-a5d6-1b83a240af2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-418af8-9dd6d7ffd-6wx9j" Nov 03 13:33:21.212122 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:21.212078 5024 patch_prober.go:29] interesting pod/e2e-api-a-dd2bbf-openshift-default-7fd7f76f49-gf5kt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body= Nov 03 13:33:21.212358 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:21.212136 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-dd2bbf-openshift-default-7fd7f76f49-gf5kt" podUID="8b6d7058-b5f5-4b35-a0ad-967187e9ba3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" Nov 03 13:33:21.382313 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:21.380924 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-418af8-9dd6d7ffd-6wx9j" Nov 03 13:33:22.110420 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:22.110393 5024 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 13:33:22.115741 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:22.115714 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-418af8-9dd6d7ffd-6wx9j"] Nov 03 13:33:22.211807 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:22.211766 5024 patch_prober.go:29] interesting pod/e2e-api-a-dd2bbf-openshift-default-7fd7f76f49-gf5kt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body= Nov 03 13:33:22.212013 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:22.211832 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-dd2bbf-openshift-default-7fd7f76f49-gf5kt" podUID="8b6d7058-b5f5-4b35-a0ad-967187e9ba3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" Nov 03 13:33:22.613334 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:22.613233 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-1ec4b892/e2e-lifecycle-model-1ec4b892-kserve-5b96cbdb9c-rl92k"] Nov 03 13:33:22.618204 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:22.618180 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-1ec4b892/e2e-lifecycle-model-1ec4b892-kserve-5b96cbdb9c-rl92k" Nov 03 13:33:22.626807 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:22.626774 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-1ec4b892/e2e-lifecycle-model-1ec4b892-kserve-5b96cbdb9c-rl92k"] Nov 03 13:33:22.629933 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:22.629907 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-1ec4b892\"/\"kube-root-ca.crt\"" Nov 03 13:33:22.630066 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:22.629996 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-1ec4b892\"/\"openshift-service-ca.crt\"" Nov 03 13:33:22.630066 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:22.630043 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-1ec4b892\"/\"e2e-lifecycle-model-1ec4b892-kserve-self-signed-certs\"" Nov 03 13:33:22.630200 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:22.630161 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-1ec4b892\"/\"default-dockercfg-lltq2\"" Nov 03 13:33:22.845315 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:22.833902 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b1a1aac6-a092-4640-b64d-6b62b12163d2-dshm\") pod \"e2e-lifecycle-model-1ec4b892-kserve-5b96cbdb9c-rl92k\" (UID: \"b1a1aac6-a092-4640-b64d-6b62b12163d2\") " pod="ai-tenant-e2e-mt-1ec4b892/e2e-lifecycle-model-1ec4b892-kserve-5b96cbdb9c-rl92k" Nov 03 13:33:22.845315 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:22.834016 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b1a1aac6-a092-4640-b64d-6b62b12163d2-tls-certs\") pod \"e2e-lifecycle-model-1ec4b892-kserve-5b96cbdb9c-rl92k\" (UID: \"b1a1aac6-a092-4640-b64d-6b62b12163d2\") " pod="ai-tenant-e2e-mt-1ec4b892/e2e-lifecycle-model-1ec4b892-kserve-5b96cbdb9c-rl92k" Nov 03 13:33:22.845315 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:22.834472 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1a1aac6-a092-4640-b64d-6b62b12163d2-tmp-dir\") pod \"e2e-lifecycle-model-1ec4b892-kserve-5b96cbdb9c-rl92k\" (UID: \"b1a1aac6-a092-4640-b64d-6b62b12163d2\") " pod="ai-tenant-e2e-mt-1ec4b892/e2e-lifecycle-model-1ec4b892-kserve-5b96cbdb9c-rl92k" Nov 03 13:33:22.845315 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:22.834537 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b1a1aac6-a092-4640-b64d-6b62b12163d2-model-cache\") pod \"e2e-lifecycle-model-1ec4b892-kserve-5b96cbdb9c-rl92k\" (UID: \"b1a1aac6-a092-4640-b64d-6b62b12163d2\") " pod="ai-tenant-e2e-mt-1ec4b892/e2e-lifecycle-model-1ec4b892-kserve-5b96cbdb9c-rl92k" Nov 03 13:33:22.845315 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:22.834567 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zkwg8\" (UniqueName: \"kubernetes.io/projected/b1a1aac6-a092-4640-b64d-6b62b12163d2-kube-api-access-zkwg8\") pod \"e2e-lifecycle-model-1ec4b892-kserve-5b96cbdb9c-rl92k\" (UID: \"b1a1aac6-a092-4640-b64d-6b62b12163d2\") " pod="ai-tenant-e2e-mt-1ec4b892/e2e-lifecycle-model-1ec4b892-kserve-5b96cbdb9c-rl92k" Nov 03 13:33:22.845315 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:22.834679 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b1a1aac6-a092-4640-b64d-6b62b12163d2-home\") pod \"e2e-lifecycle-model-1ec4b892-kserve-5b96cbdb9c-rl92k\" (UID: \"b1a1aac6-a092-4640-b64d-6b62b12163d2\") " pod="ai-tenant-e2e-mt-1ec4b892/e2e-lifecycle-model-1ec4b892-kserve-5b96cbdb9c-rl92k" Nov 03 13:33:22.950355 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:22.943859 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1a1aac6-a092-4640-b64d-6b62b12163d2-tmp-dir\") pod \"e2e-lifecycle-model-1ec4b892-kserve-5b96cbdb9c-rl92k\" (UID: \"b1a1aac6-a092-4640-b64d-6b62b12163d2\") " pod="ai-tenant-e2e-mt-1ec4b892/e2e-lifecycle-model-1ec4b892-kserve-5b96cbdb9c-rl92k" Nov 03 13:33:22.950355 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:22.943917 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b1a1aac6-a092-4640-b64d-6b62b12163d2-model-cache\") pod \"e2e-lifecycle-model-1ec4b892-kserve-5b96cbdb9c-rl92k\" (UID: \"b1a1aac6-a092-4640-b64d-6b62b12163d2\") " pod="ai-tenant-e2e-mt-1ec4b892/e2e-lifecycle-model-1ec4b892-kserve-5b96cbdb9c-rl92k" Nov 03 13:33:22.950355 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:22.943945 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zkwg8\" (UniqueName: \"kubernetes.io/projected/b1a1aac6-a092-4640-b64d-6b62b12163d2-kube-api-access-zkwg8\") pod \"e2e-lifecycle-model-1ec4b892-kserve-5b96cbdb9c-rl92k\" (UID: \"b1a1aac6-a092-4640-b64d-6b62b12163d2\") " pod="ai-tenant-e2e-mt-1ec4b892/e2e-lifecycle-model-1ec4b892-kserve-5b96cbdb9c-rl92k" Nov 03 13:33:22.950355 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:22.944029 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b1a1aac6-a092-4640-b64d-6b62b12163d2-home\") pod \"e2e-lifecycle-model-1ec4b892-kserve-5b96cbdb9c-rl92k\" (UID: \"b1a1aac6-a092-4640-b64d-6b62b12163d2\") " pod="ai-tenant-e2e-mt-1ec4b892/e2e-lifecycle-model-1ec4b892-kserve-5b96cbdb9c-rl92k" Nov 03 13:33:22.950355 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:22.944148 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b1a1aac6-a092-4640-b64d-6b62b12163d2-dshm\") pod \"e2e-lifecycle-model-1ec4b892-kserve-5b96cbdb9c-rl92k\" (UID: \"b1a1aac6-a092-4640-b64d-6b62b12163d2\") " pod="ai-tenant-e2e-mt-1ec4b892/e2e-lifecycle-model-1ec4b892-kserve-5b96cbdb9c-rl92k" Nov 03 13:33:22.950355 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:22.944212 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b1a1aac6-a092-4640-b64d-6b62b12163d2-tls-certs\") pod \"e2e-lifecycle-model-1ec4b892-kserve-5b96cbdb9c-rl92k\" (UID: \"b1a1aac6-a092-4640-b64d-6b62b12163d2\") " pod="ai-tenant-e2e-mt-1ec4b892/e2e-lifecycle-model-1ec4b892-kserve-5b96cbdb9c-rl92k" Nov 03 13:33:22.951771 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:22.951737 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1a1aac6-a092-4640-b64d-6b62b12163d2-tmp-dir\") pod \"e2e-lifecycle-model-1ec4b892-kserve-5b96cbdb9c-rl92k\" (UID: \"b1a1aac6-a092-4640-b64d-6b62b12163d2\") " pod="ai-tenant-e2e-mt-1ec4b892/e2e-lifecycle-model-1ec4b892-kserve-5b96cbdb9c-rl92k" Nov 03 13:33:22.952190 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:22.952168 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b1a1aac6-a092-4640-b64d-6b62b12163d2-model-cache\") pod \"e2e-lifecycle-model-1ec4b892-kserve-5b96cbdb9c-rl92k\" (UID: \"b1a1aac6-a092-4640-b64d-6b62b12163d2\") " pod="ai-tenant-e2e-mt-1ec4b892/e2e-lifecycle-model-1ec4b892-kserve-5b96cbdb9c-rl92k" Nov 03 13:33:22.952568 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:22.952550 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b1a1aac6-a092-4640-b64d-6b62b12163d2-home\") pod \"e2e-lifecycle-model-1ec4b892-kserve-5b96cbdb9c-rl92k\" (UID: \"b1a1aac6-a092-4640-b64d-6b62b12163d2\") " pod="ai-tenant-e2e-mt-1ec4b892/e2e-lifecycle-model-1ec4b892-kserve-5b96cbdb9c-rl92k" Nov 03 13:33:22.965451 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:22.962728 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b1a1aac6-a092-4640-b64d-6b62b12163d2-tls-certs\") pod \"e2e-lifecycle-model-1ec4b892-kserve-5b96cbdb9c-rl92k\" (UID: \"b1a1aac6-a092-4640-b64d-6b62b12163d2\") " pod="ai-tenant-e2e-mt-1ec4b892/e2e-lifecycle-model-1ec4b892-kserve-5b96cbdb9c-rl92k" Nov 03 13:33:22.992871 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:22.992834 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b1a1aac6-a092-4640-b64d-6b62b12163d2-dshm\") pod \"e2e-lifecycle-model-1ec4b892-kserve-5b96cbdb9c-rl92k\" (UID: \"b1a1aac6-a092-4640-b64d-6b62b12163d2\") " pod="ai-tenant-e2e-mt-1ec4b892/e2e-lifecycle-model-1ec4b892-kserve-5b96cbdb9c-rl92k" Nov 03 13:33:23.009665 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:23.009631 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zkwg8\" (UniqueName: \"kubernetes.io/projected/b1a1aac6-a092-4640-b64d-6b62b12163d2-kube-api-access-zkwg8\") pod \"e2e-lifecycle-model-1ec4b892-kserve-5b96cbdb9c-rl92k\" (UID: \"b1a1aac6-a092-4640-b64d-6b62b12163d2\") " pod="ai-tenant-e2e-mt-1ec4b892/e2e-lifecycle-model-1ec4b892-kserve-5b96cbdb9c-rl92k" Nov 03 13:33:23.222472 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:23.222379 5024 patch_prober.go:29] interesting pod/e2e-api-a-dd2bbf-openshift-default-7fd7f76f49-gf5kt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body= Nov 03 13:33:23.223052 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:23.223017 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-dd2bbf-openshift-default-7fd7f76f49-gf5kt" podUID="8b6d7058-b5f5-4b35-a0ad-967187e9ba3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" Nov 03 13:33:23.244631 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:23.244575 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-418af8-9dd6d7ffd-6wx9j" event={"ID":"7adbc4a3-4489-45fd-a5d6-1b83a240af2a","Type":"ContainerStarted","Data":"807432c1dd6bb4de2cb84e8918ea89b64ca2d9ac99296645a7732dd017236f7b"} Nov 03 13:33:23.257514 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:23.256824 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-418af8-9dd6d7ffd-6wx9j" Nov 03 13:33:23.257514 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:23.256874 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-418af8-9dd6d7ffd-6wx9j" event={"ID":"7adbc4a3-4489-45fd-a5d6-1b83a240af2a","Type":"ContainerStarted","Data":"67175150240a69e4b7242461c1563aee9f05dbc1bdae53f749b2670675590ffe"} Nov 03 13:33:23.257874 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:23.257815 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-1ec4b892/e2e-lifecycle-model-1ec4b892-kserve-5b96cbdb9c-rl92k" Nov 03 13:33:23.359004 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:23.338746 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-418af8-9dd6d7ffd-6wx9j" podStartSLOduration=2.826841633 podStartE2EDuration="3.338723357s" totalImagesPullingTime="511.881724ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:33:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:33:23.330651822 +0000 UTC m=+1483.338843100" watchObservedRunningTime="2026-11-03 13:33:23.338723357 +0000 UTC m=+1483.346914912" Nov 03 13:33:24.035473 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:24.031822 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-1ec4b892/e2e-lifecycle-model-1ec4b892-kserve-5b96cbdb9c-rl92k"] Nov 03 13:33:24.214380 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:24.214347 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-dd2bbf-openshift-default-7fd7f76f49-gf5kt" Nov 03 13:33:24.216346 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:24.216319 5024 patch_prober.go:29] interesting pod/e2e-api-a-dd2bbf-openshift-default-7fd7f76f49-gf5kt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body= Nov 03 13:33:24.216563 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:24.216541 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-dd2bbf-openshift-default-7fd7f76f49-gf5kt" podUID="8b6d7058-b5f5-4b35-a0ad-967187e9ba3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" Nov 03 13:33:24.259867 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:24.259759 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-1ec4b892/e2e-lifecycle-model-1ec4b892-kserve-5b96cbdb9c-rl92k" event={"ID":"b1a1aac6-a092-4640-b64d-6b62b12163d2","Type":"ContainerStarted","Data":"ed935b179655f78e8113be171468f27d20e1cde0a0327b370d200890690bf32d"} Nov 03 13:33:24.259867 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:24.259823 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-1ec4b892/e2e-lifecycle-model-1ec4b892-kserve-5b96cbdb9c-rl92k" event={"ID":"b1a1aac6-a092-4640-b64d-6b62b12163d2","Type":"ContainerStarted","Data":"aaa648165d1f2e58d5ef68c14793faafa0b30a0dc92aa64836d7d7f63943f9e9"} Nov 03 13:33:24.261715 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:24.261683 5024 generic.go:356] "Generic (PLEG): container finished" podID="2ca8b6af-59bf-4d09-ba65-9d79a4e2faba" containerID="f32398773d95c537adc89160133a4f7cc75a42791b497690b585e6fde7e0a3ad" exitCode=0 Nov 03 13:33:24.262428 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:24.262405 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-4m57d" event={"ID":"2ca8b6af-59bf-4d09-ba65-9d79a4e2faba","Type":"ContainerDied","Data":"f32398773d95c537adc89160133a4f7cc75a42791b497690b585e6fde7e0a3ad"} Nov 03 13:33:24.266779 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:24.266757 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-4m57d" Nov 03 13:33:24.328677 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:24.328241 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-mt-1ec4b892/e2e-lifecycle-model-1ec4b892-kserve-5b96cbdb9c-rl92k" podStartSLOduration=2.328191939 podStartE2EDuration="2.328191939s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:33:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:33:24.304990227 +0000 UTC m=+1484.313181504" watchObservedRunningTime="2026-11-03 13:33:24.328191939 +0000 UTC m=+1484.336383218" Nov 03 13:33:24.365111 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:24.365076 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ca8b6af-59bf-4d09-ba65-9d79a4e2faba-kube-api-access-np2bp\" (UniqueName: \"kubernetes.io/projected/2ca8b6af-59bf-4d09-ba65-9d79a4e2faba-kube-api-access-np2bp\") pod \"2ca8b6af-59bf-4d09-ba65-9d79a4e2faba\" (UID: \"2ca8b6af-59bf-4d09-ba65-9d79a4e2faba\") " Nov 03 13:33:24.365250 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:24.365174 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ca8b6af-59bf-4d09-ba65-9d79a4e2faba-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ca8b6af-59bf-4d09-ba65-9d79a4e2faba-credential-socket\") pod \"2ca8b6af-59bf-4d09-ba65-9d79a4e2faba\" (UID: \"2ca8b6af-59bf-4d09-ba65-9d79a4e2faba\") " Nov 03 13:33:24.365250 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:24.365210 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ca8b6af-59bf-4d09-ba65-9d79a4e2faba-istio-token\" (UniqueName: \"kubernetes.io/projected/2ca8b6af-59bf-4d09-ba65-9d79a4e2faba-istio-token\") pod \"2ca8b6af-59bf-4d09-ba65-9d79a4e2faba\" (UID: \"2ca8b6af-59bf-4d09-ba65-9d79a4e2faba\") " Nov 03 13:33:24.365250 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:24.365233 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ca8b6af-59bf-4d09-ba65-9d79a4e2faba-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ca8b6af-59bf-4d09-ba65-9d79a4e2faba-istio-envoy\") pod \"2ca8b6af-59bf-4d09-ba65-9d79a4e2faba\" (UID: \"2ca8b6af-59bf-4d09-ba65-9d79a4e2faba\") " Nov 03 13:33:24.365487 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:24.365442 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ca8b6af-59bf-4d09-ba65-9d79a4e2faba-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ca8b6af-59bf-4d09-ba65-9d79a4e2faba-workload-certs\") pod \"2ca8b6af-59bf-4d09-ba65-9d79a4e2faba\" (UID: \"2ca8b6af-59bf-4d09-ba65-9d79a4e2faba\") " Nov 03 13:33:24.365549 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:24.365503 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ca8b6af-59bf-4d09-ba65-9d79a4e2faba-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ca8b6af-59bf-4d09-ba65-9d79a4e2faba-workload-socket\") pod \"2ca8b6af-59bf-4d09-ba65-9d79a4e2faba\" (UID: \"2ca8b6af-59bf-4d09-ba65-9d79a4e2faba\") " Nov 03 13:33:24.365620 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:24.365603 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2ca8b6af-59bf-4d09-ba65-9d79a4e2faba-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ca8b6af-59bf-4d09-ba65-9d79a4e2faba-istiod-ca-cert\") pod \"2ca8b6af-59bf-4d09-ba65-9d79a4e2faba\" (UID: \"2ca8b6af-59bf-4d09-ba65-9d79a4e2faba\") " Nov 03 13:33:24.365729 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:24.365713 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ca8b6af-59bf-4d09-ba65-9d79a4e2faba-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ca8b6af-59bf-4d09-ba65-9d79a4e2faba-istio-data\") pod \"2ca8b6af-59bf-4d09-ba65-9d79a4e2faba\" (UID: \"2ca8b6af-59bf-4d09-ba65-9d79a4e2faba\") " Nov 03 13:33:24.365792 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:24.365747 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ca8b6af-59bf-4d09-ba65-9d79a4e2faba-workload-certs" pod "2ca8b6af-59bf-4d09-ba65-9d79a4e2faba" (UID: "2ca8b6af-59bf-4d09-ba65-9d79a4e2faba"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:33:24.365853 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:24.365838 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2ca8b6af-59bf-4d09-ba65-9d79a4e2faba-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ca8b6af-59bf-4d09-ba65-9d79a4e2faba-istio-podinfo\") pod \"2ca8b6af-59bf-4d09-ba65-9d79a4e2faba\" (UID: \"2ca8b6af-59bf-4d09-ba65-9d79a4e2faba\") " Nov 03 13:33:24.366013 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:24.365990 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ca8b6af-59bf-4d09-ba65-9d79a4e2faba-istio-data" pod "2ca8b6af-59bf-4d09-ba65-9d79a4e2faba" (UID: "2ca8b6af-59bf-4d09-ba65-9d79a4e2faba"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:33:24.366095 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:24.366009 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ca8b6af-59bf-4d09-ba65-9d79a4e2faba-workload-socket" pod "2ca8b6af-59bf-4d09-ba65-9d79a4e2faba" (UID: "2ca8b6af-59bf-4d09-ba65-9d79a4e2faba"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:33:24.366357 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:24.366258 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ca8b6af-59bf-4d09-ba65-9d79a4e2faba-credential-socket" pod "2ca8b6af-59bf-4d09-ba65-9d79a4e2faba" (UID: "2ca8b6af-59bf-4d09-ba65-9d79a4e2faba"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:33:24.366357 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:24.366177 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ca8b6af-59bf-4d09-ba65-9d79a4e2faba-istiod-ca-cert" pod "2ca8b6af-59bf-4d09-ba65-9d79a4e2faba" (UID: "2ca8b6af-59bf-4d09-ba65-9d79a4e2faba"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:33:24.367016 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:24.366988 5024 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ca8b6af-59bf-4d09-ba65-9d79a4e2faba-istio-data\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:33:24.367112 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:24.367020 5024 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ca8b6af-59bf-4d09-ba65-9d79a4e2faba-credential-socket\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:33:24.367112 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:24.367035 5024 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ca8b6af-59bf-4d09-ba65-9d79a4e2faba-workload-certs\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:33:24.367112 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:24.367048 5024 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ca8b6af-59bf-4d09-ba65-9d79a4e2faba-workload-socket\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:33:24.367112 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:24.367065 5024 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ca8b6af-59bf-4d09-ba65-9d79a4e2faba-istiod-ca-cert\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:33:24.368971 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:24.368941 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ca8b6af-59bf-4d09-ba65-9d79a4e2faba-istio-envoy" pod "2ca8b6af-59bf-4d09-ba65-9d79a4e2faba" (UID: "2ca8b6af-59bf-4d09-ba65-9d79a4e2faba"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:33:24.369250 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:24.369225 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ca8b6af-59bf-4d09-ba65-9d79a4e2faba-istio-token" pod "2ca8b6af-59bf-4d09-ba65-9d79a4e2faba" (UID: "2ca8b6af-59bf-4d09-ba65-9d79a4e2faba"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:33:24.369475 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:24.369451 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ca8b6af-59bf-4d09-ba65-9d79a4e2faba-kube-api-access-np2bp" pod "2ca8b6af-59bf-4d09-ba65-9d79a4e2faba" (UID: "2ca8b6af-59bf-4d09-ba65-9d79a4e2faba"). InnerVolumeSpecName "kube-api-access-np2bp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:33:24.369595 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:24.369577 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2ca8b6af-59bf-4d09-ba65-9d79a4e2faba-istio-podinfo" pod "2ca8b6af-59bf-4d09-ba65-9d79a4e2faba" (UID: "2ca8b6af-59bf-4d09-ba65-9d79a4e2faba"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 13:33:24.502341 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:24.501257 5024 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ca8b6af-59bf-4d09-ba65-9d79a4e2faba-istio-podinfo\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:33:24.502341 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:24.501372 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-np2bp\" (UniqueName: \"kubernetes.io/projected/2ca8b6af-59bf-4d09-ba65-9d79a4e2faba-kube-api-access-np2bp\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:33:24.502341 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:24.501394 5024 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ca8b6af-59bf-4d09-ba65-9d79a4e2faba-istio-token\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:33:24.502341 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:24.501409 5024 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ca8b6af-59bf-4d09-ba65-9d79a4e2faba-istio-envoy\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:33:24.824368 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:24.823578 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-418af8-55df5c84df-vjqhm" Nov 03 13:33:25.212463 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:25.212424 5024 patch_prober.go:29] interesting pod/e2e-api-a-dd2bbf-openshift-default-7fd7f76f49-gf5kt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body= Nov 03 13:33:25.212651 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:25.212489 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-dd2bbf-openshift-default-7fd7f76f49-gf5kt" podUID="8b6d7058-b5f5-4b35-a0ad-967187e9ba3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" Nov 03 13:33:25.259235 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:25.259193 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-1ec4b892/e2e-lifecycle-model-1ec4b892-kserve-5b96cbdb9c-rl92k" Nov 03 13:33:25.301323 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:25.299902 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-4m57d" Nov 03 13:33:25.301809 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:25.301486 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-4m57d" event={"ID":"2ca8b6af-59bf-4d09-ba65-9d79a4e2faba","Type":"ContainerDied","Data":"55fa9f6c43cf17c6b841292678701d5ed1c6ab7327f27d67de9530af3c2f3145"} Nov 03 13:33:25.301809 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:25.301561 5024 scope.go:122] "RemoveContainer" containerID="f32398773d95c537adc89160133a4f7cc75a42791b497690b585e6fde7e0a3ad" Nov 03 13:33:25.331076 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:25.331043 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-4m57d"] Nov 03 13:33:25.339224 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:25.339193 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-4m57d"] Nov 03 13:33:26.113781 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:26.113739 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2ca8b6af-59bf-4d09-ba65-9d79a4e2faba" path="/var/lib/kubelet/pods/2ca8b6af-59bf-4d09-ba65-9d79a4e2faba/volumes" Nov 03 13:33:26.282077 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:26.281646 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-dd2bbf-openshift-default-7fd7f76f49-gf5kt" Nov 03 13:33:26.307310 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:26.287986 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-dd2bbf-openshift-default-7fd7f76f49-gf5kt" Nov 03 13:33:29.268872 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:29.268838 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-418af8-9dd6d7ffd-6wx9j" Nov 03 13:33:33.261127 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:33.261094 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-1ec4b892/e2e-lifecycle-model-1ec4b892-kserve-5b96cbdb9c-rl92k" Nov 03 13:33:33.440324 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:33.438422 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-1ec4b892/e2e-lifecycle-model-1ec4b892-kserve-5b96cbdb9c-rl92k" Nov 03 13:33:33.525985 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:33.525894 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-1ec4b892/e2e-lifecycle-model-1ec4b892-kserve-5b96cbdb9c-rl92k" Nov 03 13:33:37.420570 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:37.420535 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-nqkls"] Nov 03 13:33:37.437749 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:37.437718 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-rzhn5"] Nov 03 13:33:37.438190 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:37.438145 5024 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-rzhn5" podUID="1727674a-caa9-4d85-8ee8-45bee8538927" containerName="istio-proxy" containerID="cri-o://6f03585f11e71d5baf89675f2619debaaa20d61dc8cc67e809ba057de9da739a" gracePeriod=30 Nov 03 13:33:37.439120 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:37.438781 5024 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-nqkls" podUID="19ea0757-f048-4f73-9091-cf3a0d86dd12" containerName="istio-proxy" containerID="cri-o://87f6449ca7b71baa86b1ff7cabe3d3f44555837eb4a4a31e5028ddf20e66573a" gracePeriod=30 Nov 03 13:33:37.669075 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:37.669034 5024 patch_prober.go:29] interesting pod/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-nqkls container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 13:33:37.669251 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:37.669106 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-nqkls" podUID="19ea0757-f048-4f73-9091-cf3a0d86dd12" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 13:33:37.969157 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:37.967871 5024 patch_prober.go:29] interesting pod/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-rzhn5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 13:33:37.969157 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:37.968093 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-rzhn5" podUID="1727674a-caa9-4d85-8ee8-45bee8538927" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 13:33:39.534891 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:33:39.534609 5024 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d7mwz" containerName="extractor" Nov 03 13:33:40.947568 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:40.947536 5024 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6htht_cd91de08-084e-4667-b13f-2e07b3cff624/console-operator/1.log" Nov 03 13:33:40.948708 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:40.948685 5024 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6htht_cd91de08-084e-4667-b13f-2e07b3cff624/console-operator/1.log" Nov 03 13:33:40.959873 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:40.959844 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-8lgm7"] Nov 03 13:33:40.960070 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:40.960048 5024 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-8lgm7" podUID="c8657989-e2c5-4dfc-b32a-3491ad189395" containerName="istio-proxy" containerID="cri-o://7d1b86e4edd6ae09cc252a82929f5f7c328be74f5ff0057cbaa26ff45c752cb4" gracePeriod=30 Nov 03 13:33:42.869928 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:42.844349 5024 generic.go:356] "Generic (PLEG): container finished" podID="1727674a-caa9-4d85-8ee8-45bee8538927" containerID="6f03585f11e71d5baf89675f2619debaaa20d61dc8cc67e809ba057de9da739a" exitCode=0 Nov 03 13:33:42.869928 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:42.844612 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-rzhn5" event={"ID":"1727674a-caa9-4d85-8ee8-45bee8538927","Type":"ContainerDied","Data":"6f03585f11e71d5baf89675f2619debaaa20d61dc8cc67e809ba057de9da739a"} Nov 03 13:33:42.869928 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:42.846743 5024 generic.go:356] "Generic (PLEG): container finished" podID="19ea0757-f048-4f73-9091-cf3a0d86dd12" containerID="87f6449ca7b71baa86b1ff7cabe3d3f44555837eb4a4a31e5028ddf20e66573a" exitCode=0 Nov 03 13:33:42.869928 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:42.846842 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-nqkls" event={"ID":"19ea0757-f048-4f73-9091-cf3a0d86dd12","Type":"ContainerDied","Data":"87f6449ca7b71baa86b1ff7cabe3d3f44555837eb4a4a31e5028ddf20e66573a"} Nov 03 13:33:42.899276 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:42.899240 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-nqkls" Nov 03 13:33:43.020345 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:43.017213 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19ea0757-f048-4f73-9091-cf3a0d86dd12-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19ea0757-f048-4f73-9091-cf3a0d86dd12-workload-socket\") pod \"19ea0757-f048-4f73-9091-cf3a0d86dd12\" (UID: \"19ea0757-f048-4f73-9091-cf3a0d86dd12\") " Nov 03 13:33:43.020345 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:43.017323 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19ea0757-f048-4f73-9091-cf3a0d86dd12-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19ea0757-f048-4f73-9091-cf3a0d86dd12-istio-envoy\") pod \"19ea0757-f048-4f73-9091-cf3a0d86dd12\" (UID: \"19ea0757-f048-4f73-9091-cf3a0d86dd12\") " Nov 03 13:33:43.020345 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:43.017386 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/19ea0757-f048-4f73-9091-cf3a0d86dd12-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19ea0757-f048-4f73-9091-cf3a0d86dd12-istio-podinfo\") pod \"19ea0757-f048-4f73-9091-cf3a0d86dd12\" (UID: \"19ea0757-f048-4f73-9091-cf3a0d86dd12\") " Nov 03 13:33:43.020345 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:43.017458 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/19ea0757-f048-4f73-9091-cf3a0d86dd12-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19ea0757-f048-4f73-9091-cf3a0d86dd12-istiod-ca-cert\") pod \"19ea0757-f048-4f73-9091-cf3a0d86dd12\" (UID: \"19ea0757-f048-4f73-9091-cf3a0d86dd12\") " Nov 03 13:33:43.020345 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:43.017486 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19ea0757-f048-4f73-9091-cf3a0d86dd12-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19ea0757-f048-4f73-9091-cf3a0d86dd12-workload-certs\") pod \"19ea0757-f048-4f73-9091-cf3a0d86dd12\" (UID: \"19ea0757-f048-4f73-9091-cf3a0d86dd12\") " Nov 03 13:33:43.020345 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:43.017626 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19ea0757-f048-4f73-9091-cf3a0d86dd12-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19ea0757-f048-4f73-9091-cf3a0d86dd12-istio-data\") pod \"19ea0757-f048-4f73-9091-cf3a0d86dd12\" (UID: \"19ea0757-f048-4f73-9091-cf3a0d86dd12\") " Nov 03 13:33:43.020345 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:43.017660 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19ea0757-f048-4f73-9091-cf3a0d86dd12-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19ea0757-f048-4f73-9091-cf3a0d86dd12-credential-socket\") pod \"19ea0757-f048-4f73-9091-cf3a0d86dd12\" (UID: \"19ea0757-f048-4f73-9091-cf3a0d86dd12\") " Nov 03 13:33:43.020345 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:43.017758 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19ea0757-f048-4f73-9091-cf3a0d86dd12-kube-api-access-zjdtq\" (UniqueName: \"kubernetes.io/projected/19ea0757-f048-4f73-9091-cf3a0d86dd12-kube-api-access-zjdtq\") pod \"19ea0757-f048-4f73-9091-cf3a0d86dd12\" (UID: \"19ea0757-f048-4f73-9091-cf3a0d86dd12\") " Nov 03 13:33:43.020345 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:43.017847 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19ea0757-f048-4f73-9091-cf3a0d86dd12-istio-token\" (UniqueName: \"kubernetes.io/projected/19ea0757-f048-4f73-9091-cf3a0d86dd12-istio-token\") pod \"19ea0757-f048-4f73-9091-cf3a0d86dd12\" (UID: \"19ea0757-f048-4f73-9091-cf3a0d86dd12\") " Nov 03 13:33:43.020345 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:43.019265 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19ea0757-f048-4f73-9091-cf3a0d86dd12-istiod-ca-cert" pod "19ea0757-f048-4f73-9091-cf3a0d86dd12" (UID: "19ea0757-f048-4f73-9091-cf3a0d86dd12"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:33:43.020345 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:43.019503 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19ea0757-f048-4f73-9091-cf3a0d86dd12-workload-socket" pod "19ea0757-f048-4f73-9091-cf3a0d86dd12" (UID: "19ea0757-f048-4f73-9091-cf3a0d86dd12"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:33:43.020345 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:43.020340 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19ea0757-f048-4f73-9091-cf3a0d86dd12-istio-data" pod "19ea0757-f048-4f73-9091-cf3a0d86dd12" (UID: "19ea0757-f048-4f73-9091-cf3a0d86dd12"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:33:43.021103 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:43.020493 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19ea0757-f048-4f73-9091-cf3a0d86dd12-workload-certs" pod "19ea0757-f048-4f73-9091-cf3a0d86dd12" (UID: "19ea0757-f048-4f73-9091-cf3a0d86dd12"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:33:43.023363 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:43.022100 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19ea0757-f048-4f73-9091-cf3a0d86dd12-credential-socket" pod "19ea0757-f048-4f73-9091-cf3a0d86dd12" (UID: "19ea0757-f048-4f73-9091-cf3a0d86dd12"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:33:43.023889 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:43.023754 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19ea0757-f048-4f73-9091-cf3a0d86dd12-istio-envoy" pod "19ea0757-f048-4f73-9091-cf3a0d86dd12" (UID: "19ea0757-f048-4f73-9091-cf3a0d86dd12"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:33:43.023889 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:43.023814 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19ea0757-f048-4f73-9091-cf3a0d86dd12-istio-token" pod "19ea0757-f048-4f73-9091-cf3a0d86dd12" (UID: "19ea0757-f048-4f73-9091-cf3a0d86dd12"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:33:43.024593 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:43.024566 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19ea0757-f048-4f73-9091-cf3a0d86dd12-kube-api-access-zjdtq" pod "19ea0757-f048-4f73-9091-cf3a0d86dd12" (UID: "19ea0757-f048-4f73-9091-cf3a0d86dd12"). InnerVolumeSpecName "kube-api-access-zjdtq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:33:43.024753 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:43.024729 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/19ea0757-f048-4f73-9091-cf3a0d86dd12-istio-podinfo" pod "19ea0757-f048-4f73-9091-cf3a0d86dd12" (UID: "19ea0757-f048-4f73-9091-cf3a0d86dd12"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 13:33:43.123602 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:43.120312 5024 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19ea0757-f048-4f73-9091-cf3a0d86dd12-istiod-ca-cert\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:33:43.123602 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:43.120347 5024 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19ea0757-f048-4f73-9091-cf3a0d86dd12-workload-certs\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:33:43.123602 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:43.120363 5024 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19ea0757-f048-4f73-9091-cf3a0d86dd12-istio-data\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:33:43.123602 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:43.120377 5024 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19ea0757-f048-4f73-9091-cf3a0d86dd12-credential-socket\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:33:43.123602 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:43.120393 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zjdtq\" (UniqueName: \"kubernetes.io/projected/19ea0757-f048-4f73-9091-cf3a0d86dd12-kube-api-access-zjdtq\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:33:43.123602 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:43.120408 5024 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19ea0757-f048-4f73-9091-cf3a0d86dd12-istio-token\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:33:43.123602 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:43.120424 5024 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19ea0757-f048-4f73-9091-cf3a0d86dd12-workload-socket\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:33:43.123602 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:43.120441 5024 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19ea0757-f048-4f73-9091-cf3a0d86dd12-istio-envoy\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:33:43.123602 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:43.120456 5024 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19ea0757-f048-4f73-9091-cf3a0d86dd12-istio-podinfo\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:33:43.175048 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:43.175015 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-rzhn5" Nov 03 13:33:43.223594 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:43.221941 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1727674a-caa9-4d85-8ee8-45bee8538927-istio-token\" (UniqueName: \"kubernetes.io/projected/1727674a-caa9-4d85-8ee8-45bee8538927-istio-token\") pod \"1727674a-caa9-4d85-8ee8-45bee8538927\" (UID: \"1727674a-caa9-4d85-8ee8-45bee8538927\") " Nov 03 13:33:43.223594 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:43.222016 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1727674a-caa9-4d85-8ee8-45bee8538927-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1727674a-caa9-4d85-8ee8-45bee8538927-istiod-ca-cert\") pod \"1727674a-caa9-4d85-8ee8-45bee8538927\" (UID: \"1727674a-caa9-4d85-8ee8-45bee8538927\") " Nov 03 13:33:43.223594 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:43.222118 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1727674a-caa9-4d85-8ee8-45bee8538927-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1727674a-caa9-4d85-8ee8-45bee8538927-workload-socket\") pod \"1727674a-caa9-4d85-8ee8-45bee8538927\" (UID: \"1727674a-caa9-4d85-8ee8-45bee8538927\") " Nov 03 13:33:43.223594 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:43.222257 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1727674a-caa9-4d85-8ee8-45bee8538927-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1727674a-caa9-4d85-8ee8-45bee8538927-workload-certs\") pod \"1727674a-caa9-4d85-8ee8-45bee8538927\" (UID: \"1727674a-caa9-4d85-8ee8-45bee8538927\") " Nov 03 13:33:43.223594 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:43.222445 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1727674a-caa9-4d85-8ee8-45bee8538927-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1727674a-caa9-4d85-8ee8-45bee8538927-istio-envoy\") pod \"1727674a-caa9-4d85-8ee8-45bee8538927\" (UID: \"1727674a-caa9-4d85-8ee8-45bee8538927\") " Nov 03 13:33:43.223594 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:43.222607 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1727674a-caa9-4d85-8ee8-45bee8538927-kube-api-access-8qgrx\" (UniqueName: \"kubernetes.io/projected/1727674a-caa9-4d85-8ee8-45bee8538927-kube-api-access-8qgrx\") pod \"1727674a-caa9-4d85-8ee8-45bee8538927\" (UID: \"1727674a-caa9-4d85-8ee8-45bee8538927\") " Nov 03 13:33:43.223594 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:43.222675 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1727674a-caa9-4d85-8ee8-45bee8538927-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1727674a-caa9-4d85-8ee8-45bee8538927-istio-podinfo\") pod \"1727674a-caa9-4d85-8ee8-45bee8538927\" (UID: \"1727674a-caa9-4d85-8ee8-45bee8538927\") " Nov 03 13:33:43.223594 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:43.222700 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1727674a-caa9-4d85-8ee8-45bee8538927-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1727674a-caa9-4d85-8ee8-45bee8538927-istio-data\") pod \"1727674a-caa9-4d85-8ee8-45bee8538927\" (UID: \"1727674a-caa9-4d85-8ee8-45bee8538927\") " Nov 03 13:33:43.224204 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:43.224046 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1727674a-caa9-4d85-8ee8-45bee8538927-istiod-ca-cert" pod "1727674a-caa9-4d85-8ee8-45bee8538927" (UID: "1727674a-caa9-4d85-8ee8-45bee8538927"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:33:43.224388 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:43.224333 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1727674a-caa9-4d85-8ee8-45bee8538927-workload-socket" pod "1727674a-caa9-4d85-8ee8-45bee8538927" (UID: "1727674a-caa9-4d85-8ee8-45bee8538927"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:33:43.224785 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:43.224582 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1727674a-caa9-4d85-8ee8-45bee8538927-workload-certs" pod "1727674a-caa9-4d85-8ee8-45bee8538927" (UID: "1727674a-caa9-4d85-8ee8-45bee8538927"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:33:43.227055 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:43.226932 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1727674a-caa9-4d85-8ee8-45bee8538927-istio-data" pod "1727674a-caa9-4d85-8ee8-45bee8538927" (UID: "1727674a-caa9-4d85-8ee8-45bee8538927"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:33:43.227888 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:43.227861 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1727674a-caa9-4d85-8ee8-45bee8538927-istio-envoy" pod "1727674a-caa9-4d85-8ee8-45bee8538927" (UID: "1727674a-caa9-4d85-8ee8-45bee8538927"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:33:43.229457 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:43.229435 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1727674a-caa9-4d85-8ee8-45bee8538927-kube-api-access-8qgrx" pod "1727674a-caa9-4d85-8ee8-45bee8538927" (UID: "1727674a-caa9-4d85-8ee8-45bee8538927"). InnerVolumeSpecName "kube-api-access-8qgrx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:33:43.229831 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:43.229801 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1727674a-caa9-4d85-8ee8-45bee8538927-istio-token" pod "1727674a-caa9-4d85-8ee8-45bee8538927" (UID: "1727674a-caa9-4d85-8ee8-45bee8538927"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:33:43.230018 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:43.229989 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1727674a-caa9-4d85-8ee8-45bee8538927-istio-podinfo" pod "1727674a-caa9-4d85-8ee8-45bee8538927" (UID: "1727674a-caa9-4d85-8ee8-45bee8538927"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 13:33:43.324275 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:43.324236 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1727674a-caa9-4d85-8ee8-45bee8538927-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1727674a-caa9-4d85-8ee8-45bee8538927-credential-socket\") pod \"1727674a-caa9-4d85-8ee8-45bee8538927\" (UID: \"1727674a-caa9-4d85-8ee8-45bee8538927\") " Nov 03 13:33:43.324553 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:43.324527 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1727674a-caa9-4d85-8ee8-45bee8538927-credential-socket" pod "1727674a-caa9-4d85-8ee8-45bee8538927" (UID: "1727674a-caa9-4d85-8ee8-45bee8538927"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:33:43.324953 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:43.324934 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8qgrx\" (UniqueName: \"kubernetes.io/projected/1727674a-caa9-4d85-8ee8-45bee8538927-kube-api-access-8qgrx\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:33:43.324996 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:43.324957 5024 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1727674a-caa9-4d85-8ee8-45bee8538927-istio-podinfo\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:33:43.324996 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:43.324966 5024 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1727674a-caa9-4d85-8ee8-45bee8538927-istio-data\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:33:43.324996 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:43.324979 5024 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1727674a-caa9-4d85-8ee8-45bee8538927-credential-socket\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:33:43.324996 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:43.324987 5024 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1727674a-caa9-4d85-8ee8-45bee8538927-istio-token\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:33:43.325111 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:43.324995 5024 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1727674a-caa9-4d85-8ee8-45bee8538927-istiod-ca-cert\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:33:43.325111 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:43.325005 5024 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1727674a-caa9-4d85-8ee8-45bee8538927-workload-socket\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:33:43.325111 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:43.325013 5024 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1727674a-caa9-4d85-8ee8-45bee8538927-workload-certs\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:33:43.325111 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:43.325021 5024 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1727674a-caa9-4d85-8ee8-45bee8538927-istio-envoy\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:33:43.856424 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:43.856384 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-nqkls" Nov 03 13:33:43.856424 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:43.856399 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-nqkls" event={"ID":"19ea0757-f048-4f73-9091-cf3a0d86dd12","Type":"ContainerDied","Data":"036f17e36691ddd7eff3b486f13084f9cbb83c3a5dccc390c854aa0419ea03e5"} Nov 03 13:33:43.856671 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:43.856503 5024 scope.go:122] "RemoveContainer" containerID="87f6449ca7b71baa86b1ff7cabe3d3f44555837eb4a4a31e5028ddf20e66573a" Nov 03 13:33:43.858092 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:43.858068 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-rzhn5" event={"ID":"1727674a-caa9-4d85-8ee8-45bee8538927","Type":"ContainerDied","Data":"671164c163498b87c4f866190db94eaed327154dd2bad4be4d4e75107711d25b"} Nov 03 13:33:43.858180 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:43.858159 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-rzhn5" Nov 03 13:33:43.869765 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:43.869739 5024 scope.go:122] "RemoveContainer" containerID="6f03585f11e71d5baf89675f2619debaaa20d61dc8cc67e809ba057de9da739a" Nov 03 13:33:43.894340 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:43.894280 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-nqkls"] Nov 03 13:33:43.900719 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:43.900687 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-nqkls"] Nov 03 13:33:43.913943 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:43.913917 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-rzhn5"] Nov 03 13:33:43.919964 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:43.919939 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-rzhn5"] Nov 03 13:33:44.143977 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:44.143943 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1727674a-caa9-4d85-8ee8-45bee8538927" path="/var/lib/kubelet/pods/1727674a-caa9-4d85-8ee8-45bee8538927/volumes" Nov 03 13:33:44.144544 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:44.144525 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="19ea0757-f048-4f73-9091-cf3a0d86dd12" path="/var/lib/kubelet/pods/19ea0757-f048-4f73-9091-cf3a0d86dd12/volumes" Nov 03 13:33:45.948358 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:45.937067 5024 generic.go:356] "Generic (PLEG): container finished" podID="752f4211-a41b-4b2e-b568-3b3bc1ce0989" containerID="7724369ad432b2c96a54f3a089844dd113f77e4ccf6f578ac8e8719894f4ee45" exitCode=0 Nov 03 13:33:45.948358 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:45.937368 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-1ec4b892-openshift-default-6c8678f7c-prh98" event={"ID":"752f4211-a41b-4b2e-b568-3b3bc1ce0989","Type":"ContainerDied","Data":"7724369ad432b2c96a54f3a089844dd113f77e4ccf6f578ac8e8719894f4ee45"} Nov 03 13:33:45.948358 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:45.937911 5024 scope.go:122] "RemoveContainer" containerID="7724369ad432b2c96a54f3a089844dd113f77e4ccf6f578ac8e8719894f4ee45" Nov 03 13:33:45.957275 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:45.957229 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:33:45.958166 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:45.958128 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:33:45.970891 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:45.970852 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:33:46.153303 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:46.153242 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-dd2bbf-74fb8cb9c9-95hxq"] Nov 03 13:33:46.234104 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:46.203399 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-dd2bbf-68fc6dd7d7-z55dw"] Nov 03 13:33:46.864378 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:46.864350 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-8lgm7" Nov 03 13:33:46.965306 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:46.958239 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-1ec4b892-openshift-default-6c8678f7c-prh98" event={"ID":"752f4211-a41b-4b2e-b568-3b3bc1ce0989","Type":"ContainerStarted","Data":"856e798c4f8001afe2ebf9e60f45eab903bdd2fd33cfeffdaa7f430dd00e4129"} Nov 03 13:33:46.965306 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:46.961953 5024 generic.go:356] "Generic (PLEG): container finished" podID="c8657989-e2c5-4dfc-b32a-3491ad189395" containerID="7d1b86e4edd6ae09cc252a82929f5f7c328be74f5ff0057cbaa26ff45c752cb4" exitCode=0 Nov 03 13:33:46.965306 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:46.962072 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-8lgm7" event={"ID":"c8657989-e2c5-4dfc-b32a-3491ad189395","Type":"ContainerDied","Data":"7d1b86e4edd6ae09cc252a82929f5f7c328be74f5ff0057cbaa26ff45c752cb4"} Nov 03 13:33:46.965306 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:46.962116 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-8lgm7" event={"ID":"c8657989-e2c5-4dfc-b32a-3491ad189395","Type":"ContainerDied","Data":"7c6f219cd131a6784235566b21782bd96fe4486658a0a423ecbaa57a0c6346e2"} Nov 03 13:33:46.965306 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:46.962143 5024 scope.go:122] "RemoveContainer" containerID="7d1b86e4edd6ae09cc252a82929f5f7c328be74f5ff0057cbaa26ff45c752cb4" Nov 03 13:33:46.965306 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:46.962249 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-8lgm7" Nov 03 13:33:47.063832 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:47.061445 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c8657989-e2c5-4dfc-b32a-3491ad189395-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8657989-e2c5-4dfc-b32a-3491ad189395-istio-podinfo\") pod \"c8657989-e2c5-4dfc-b32a-3491ad189395\" (UID: \"c8657989-e2c5-4dfc-b32a-3491ad189395\") " Nov 03 13:33:47.063832 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:47.061515 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8657989-e2c5-4dfc-b32a-3491ad189395-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8657989-e2c5-4dfc-b32a-3491ad189395-credential-socket\") pod \"c8657989-e2c5-4dfc-b32a-3491ad189395\" (UID: \"c8657989-e2c5-4dfc-b32a-3491ad189395\") " Nov 03 13:33:47.063832 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:47.061578 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8657989-e2c5-4dfc-b32a-3491ad189395-istio-token\" (UniqueName: \"kubernetes.io/projected/c8657989-e2c5-4dfc-b32a-3491ad189395-istio-token\") pod \"c8657989-e2c5-4dfc-b32a-3491ad189395\" (UID: \"c8657989-e2c5-4dfc-b32a-3491ad189395\") " Nov 03 13:33:47.063832 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:47.061625 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8657989-e2c5-4dfc-b32a-3491ad189395-kube-api-access-jk96l\" (UniqueName: \"kubernetes.io/projected/c8657989-e2c5-4dfc-b32a-3491ad189395-kube-api-access-jk96l\") pod \"c8657989-e2c5-4dfc-b32a-3491ad189395\" (UID: \"c8657989-e2c5-4dfc-b32a-3491ad189395\") " Nov 03 13:33:47.063832 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:47.061732 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8657989-e2c5-4dfc-b32a-3491ad189395-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8657989-e2c5-4dfc-b32a-3491ad189395-istio-data\") pod \"c8657989-e2c5-4dfc-b32a-3491ad189395\" (UID: \"c8657989-e2c5-4dfc-b32a-3491ad189395\") " Nov 03 13:33:47.063832 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:47.061832 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8657989-e2c5-4dfc-b32a-3491ad189395-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8657989-e2c5-4dfc-b32a-3491ad189395-istio-envoy\") pod \"c8657989-e2c5-4dfc-b32a-3491ad189395\" (UID: \"c8657989-e2c5-4dfc-b32a-3491ad189395\") " Nov 03 13:33:47.063832 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:47.061884 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8657989-e2c5-4dfc-b32a-3491ad189395-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8657989-e2c5-4dfc-b32a-3491ad189395-workload-certs\") pod \"c8657989-e2c5-4dfc-b32a-3491ad189395\" (UID: \"c8657989-e2c5-4dfc-b32a-3491ad189395\") " Nov 03 13:33:47.063832 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:47.061937 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c8657989-e2c5-4dfc-b32a-3491ad189395-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8657989-e2c5-4dfc-b32a-3491ad189395-istiod-ca-cert\") pod \"c8657989-e2c5-4dfc-b32a-3491ad189395\" (UID: \"c8657989-e2c5-4dfc-b32a-3491ad189395\") " Nov 03 13:33:47.063832 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:47.061972 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8657989-e2c5-4dfc-b32a-3491ad189395-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8657989-e2c5-4dfc-b32a-3491ad189395-workload-socket\") pod \"c8657989-e2c5-4dfc-b32a-3491ad189395\" (UID: \"c8657989-e2c5-4dfc-b32a-3491ad189395\") " Nov 03 13:33:47.064830 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:47.064585 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8657989-e2c5-4dfc-b32a-3491ad189395-workload-socket" pod "c8657989-e2c5-4dfc-b32a-3491ad189395" (UID: "c8657989-e2c5-4dfc-b32a-3491ad189395"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:33:47.065160 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:47.064985 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8657989-e2c5-4dfc-b32a-3491ad189395-workload-certs" pod "c8657989-e2c5-4dfc-b32a-3491ad189395" (UID: "c8657989-e2c5-4dfc-b32a-3491ad189395"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:33:47.067365 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:47.065402 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8657989-e2c5-4dfc-b32a-3491ad189395-istiod-ca-cert" pod "c8657989-e2c5-4dfc-b32a-3491ad189395" (UID: "c8657989-e2c5-4dfc-b32a-3491ad189395"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:33:47.067365 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:47.065536 5024 scope.go:122] "RemoveContainer" containerID="7d1b86e4edd6ae09cc252a82929f5f7c328be74f5ff0057cbaa26ff45c752cb4" Nov 03 13:33:47.067365 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:47.066741 5024 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8657989-e2c5-4dfc-b32a-3491ad189395-workload-certs\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:33:47.067365 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:47.066762 5024 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8657989-e2c5-4dfc-b32a-3491ad189395-istiod-ca-cert\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:33:47.067365 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:47.066777 5024 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8657989-e2c5-4dfc-b32a-3491ad189395-workload-socket\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:33:47.068633 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:47.068363 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8657989-e2c5-4dfc-b32a-3491ad189395-credential-socket" pod "c8657989-e2c5-4dfc-b32a-3491ad189395" (UID: "c8657989-e2c5-4dfc-b32a-3491ad189395"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:33:47.069015 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:47.068996 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8657989-e2c5-4dfc-b32a-3491ad189395-istio-data" pod "c8657989-e2c5-4dfc-b32a-3491ad189395" (UID: "c8657989-e2c5-4dfc-b32a-3491ad189395"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:33:47.095902 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:33:47.078433 5024 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d1b86e4edd6ae09cc252a82929f5f7c328be74f5ff0057cbaa26ff45c752cb4\": container with ID starting with 7d1b86e4edd6ae09cc252a82929f5f7c328be74f5ff0057cbaa26ff45c752cb4 not found: ID does not exist" containerID="7d1b86e4edd6ae09cc252a82929f5f7c328be74f5ff0057cbaa26ff45c752cb4" Nov 03 13:33:47.095902 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:47.078510 5024 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d1b86e4edd6ae09cc252a82929f5f7c328be74f5ff0057cbaa26ff45c752cb4"} err="failed to get container status \"7d1b86e4edd6ae09cc252a82929f5f7c328be74f5ff0057cbaa26ff45c752cb4\": rpc error: code = NotFound desc = could not find container \"7d1b86e4edd6ae09cc252a82929f5f7c328be74f5ff0057cbaa26ff45c752cb4\": container with ID starting with 7d1b86e4edd6ae09cc252a82929f5f7c328be74f5ff0057cbaa26ff45c752cb4 not found: ID does not exist" Nov 03 13:33:47.095902 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:47.092898 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8657989-e2c5-4dfc-b32a-3491ad189395-istio-token" pod "c8657989-e2c5-4dfc-b32a-3491ad189395" (UID: "c8657989-e2c5-4dfc-b32a-3491ad189395"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:33:47.096431 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:47.096246 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c8657989-e2c5-4dfc-b32a-3491ad189395-istio-podinfo" pod "c8657989-e2c5-4dfc-b32a-3491ad189395" (UID: "c8657989-e2c5-4dfc-b32a-3491ad189395"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 13:33:47.096690 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:47.096660 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8657989-e2c5-4dfc-b32a-3491ad189395-istio-envoy" pod "c8657989-e2c5-4dfc-b32a-3491ad189395" (UID: "c8657989-e2c5-4dfc-b32a-3491ad189395"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:33:47.097827 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:47.097792 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8657989-e2c5-4dfc-b32a-3491ad189395-kube-api-access-jk96l" pod "c8657989-e2c5-4dfc-b32a-3491ad189395" (UID: "c8657989-e2c5-4dfc-b32a-3491ad189395"). InnerVolumeSpecName "kube-api-access-jk96l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:33:47.183563 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:47.183523 5024 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8657989-e2c5-4dfc-b32a-3491ad189395-istio-data\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:33:47.183563 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:47.183564 5024 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8657989-e2c5-4dfc-b32a-3491ad189395-istio-envoy\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:33:47.183815 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:47.183581 5024 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8657989-e2c5-4dfc-b32a-3491ad189395-istio-podinfo\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:33:47.183815 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:47.183604 5024 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8657989-e2c5-4dfc-b32a-3491ad189395-credential-socket\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:33:47.183815 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:47.183616 5024 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8657989-e2c5-4dfc-b32a-3491ad189395-istio-token\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:33:47.183815 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:47.183629 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jk96l\" (UniqueName: \"kubernetes.io/projected/c8657989-e2c5-4dfc-b32a-3491ad189395-kube-api-access-jk96l\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:33:47.292651 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:47.292618 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-8lgm7"] Nov 03 13:33:47.297265 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:47.297233 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-8lgm7"] Nov 03 13:33:47.684699 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:47.684667 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-1ec4b892-openshift-default-6c8678f7c-prh98" Nov 03 13:33:47.685087 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:47.685064 5024 patch_prober.go:29] interesting pod/e2e-mt-1ec4b892-openshift-default-6c8678f7c-prh98 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body= Nov 03 13:33:47.685250 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:47.685223 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-1ec4b892-openshift-default-6c8678f7c-prh98" podUID="752f4211-a41b-4b2e-b568-3b3bc1ce0989" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" Nov 03 13:33:47.903431 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:47.903390 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-1ec4b892-openshift-default-6c8678f7c-prh98" Nov 03 13:33:48.155838 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:48.155800 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c8657989-e2c5-4dfc-b32a-3491ad189395" path="/var/lib/kubelet/pods/c8657989-e2c5-4dfc-b32a-3491ad189395/volumes" Nov 03 13:33:48.688307 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:48.687375 5024 patch_prober.go:29] interesting pod/e2e-mt-1ec4b892-openshift-default-6c8678f7c-prh98 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body= Nov 03 13:33:48.688307 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:48.687440 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-1ec4b892-openshift-default-6c8678f7c-prh98" podUID="752f4211-a41b-4b2e-b568-3b3bc1ce0989" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" Nov 03 13:33:49.684370 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:49.684335 5024 patch_prober.go:29] interesting pod/e2e-mt-1ec4b892-openshift-default-6c8678f7c-prh98 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body= Nov 03 13:33:49.684910 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:49.684882 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-1ec4b892-openshift-default-6c8678f7c-prh98" podUID="752f4211-a41b-4b2e-b568-3b3bc1ce0989" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" Nov 03 13:33:50.698308 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:50.681752 5024 patch_prober.go:29] interesting pod/e2e-mt-1ec4b892-openshift-default-6c8678f7c-prh98 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body= Nov 03 13:33:50.698308 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:50.681832 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-1ec4b892-openshift-default-6c8678f7c-prh98" podUID="752f4211-a41b-4b2e-b568-3b3bc1ce0989" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" Nov 03 13:33:51.155864 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:51.155788 5024 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-dd2bbf-74fb8cb9c9-95hxq" podUID="50a9026b-b55f-4650-988a-755a758dca86" containerName="payload-processing" containerID="cri-o://c9aebf711b98782cfbb203dc381cc5769078119ced9b613f92b9e3a96d0983c0" gracePeriod=25 Nov 03 13:33:51.204586 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:51.204541 5024 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-dd2bbf-68fc6dd7d7-z55dw" podUID="e553bdcf-e114-4046-8e10-4a2687d78a52" containerName="payload-pre-processing" containerID="cri-o://fd9d2b375395c7849a5995911f32c3e63401eb36d89fe4146655c47e2a840f36" gracePeriod=25 Nov 03 13:33:51.702559 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:51.702517 5024 patch_prober.go:29] interesting pod/e2e-mt-1ec4b892-openshift-default-6c8678f7c-prh98 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body= Nov 03 13:33:51.702997 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:51.702573 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-1ec4b892-openshift-default-6c8678f7c-prh98" podUID="752f4211-a41b-4b2e-b568-3b3bc1ce0989" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" Nov 03 13:33:51.703250 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:51.703220 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-dd2bbf-74fb8cb9c9-95hxq" Nov 03 13:33:51.759628 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:51.759594 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/50a9026b-b55f-4650-988a-755a758dca86-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50a9026b-b55f-4650-988a-755a758dca86-plugins-config-volume\") pod \"50a9026b-b55f-4650-988a-755a758dca86\" (UID: \"50a9026b-b55f-4650-988a-755a758dca86\") " Nov 03 13:33:51.759806 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:51.759680 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50a9026b-b55f-4650-988a-755a758dca86-kube-api-access-q8h2h\" (UniqueName: \"kubernetes.io/projected/50a9026b-b55f-4650-988a-755a758dca86-kube-api-access-q8h2h\") pod \"50a9026b-b55f-4650-988a-755a758dca86\" (UID: \"50a9026b-b55f-4650-988a-755a758dca86\") " Nov 03 13:33:51.760797 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:51.760734 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50a9026b-b55f-4650-988a-755a758dca86-plugins-config-volume" pod "50a9026b-b55f-4650-988a-755a758dca86" (UID: "50a9026b-b55f-4650-988a-755a758dca86"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:33:51.762828 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:51.762800 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50a9026b-b55f-4650-988a-755a758dca86-kube-api-access-q8h2h" pod "50a9026b-b55f-4650-988a-755a758dca86" (UID: "50a9026b-b55f-4650-988a-755a758dca86"). InnerVolumeSpecName "kube-api-access-q8h2h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:33:51.822021 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:51.821990 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-dd2bbf-68fc6dd7d7-z55dw" Nov 03 13:33:51.861012 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:51.860981 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e553bdcf-e114-4046-8e10-4a2687d78a52-kube-api-access-s25kn\" (UniqueName: \"kubernetes.io/projected/e553bdcf-e114-4046-8e10-4a2687d78a52-kube-api-access-s25kn\") pod \"e553bdcf-e114-4046-8e10-4a2687d78a52\" (UID: \"e553bdcf-e114-4046-8e10-4a2687d78a52\") " Nov 03 13:33:51.861190 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:51.861031 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e553bdcf-e114-4046-8e10-4a2687d78a52-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e553bdcf-e114-4046-8e10-4a2687d78a52-plugins-config-volume\") pod \"e553bdcf-e114-4046-8e10-4a2687d78a52\" (UID: \"e553bdcf-e114-4046-8e10-4a2687d78a52\") " Nov 03 13:33:51.861505 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:51.861470 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e553bdcf-e114-4046-8e10-4a2687d78a52-plugins-config-volume" pod "e553bdcf-e114-4046-8e10-4a2687d78a52" (UID: "e553bdcf-e114-4046-8e10-4a2687d78a52"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:33:51.861929 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:51.861771 5024 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e553bdcf-e114-4046-8e10-4a2687d78a52-plugins-config-volume\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:33:51.861929 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:51.861800 5024 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50a9026b-b55f-4650-988a-755a758dca86-plugins-config-volume\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:33:51.861929 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:51.861816 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q8h2h\" (UniqueName: \"kubernetes.io/projected/50a9026b-b55f-4650-988a-755a758dca86-kube-api-access-q8h2h\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:33:51.883376 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:51.883337 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e553bdcf-e114-4046-8e10-4a2687d78a52-kube-api-access-s25kn" pod "e553bdcf-e114-4046-8e10-4a2687d78a52" (UID: "e553bdcf-e114-4046-8e10-4a2687d78a52"). InnerVolumeSpecName "kube-api-access-s25kn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:33:51.963052 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:51.962970 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s25kn\" (UniqueName: \"kubernetes.io/projected/e553bdcf-e114-4046-8e10-4a2687d78a52-kube-api-access-s25kn\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:33:52.043278 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:52.043243 5024 generic.go:356] "Generic (PLEG): container finished" podID="e553bdcf-e114-4046-8e10-4a2687d78a52" containerID="fd9d2b375395c7849a5995911f32c3e63401eb36d89fe4146655c47e2a840f36" exitCode=0 Nov 03 13:33:52.043491 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:52.043315 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-dd2bbf-68fc6dd7d7-z55dw" event={"ID":"e553bdcf-e114-4046-8e10-4a2687d78a52","Type":"ContainerDied","Data":"fd9d2b375395c7849a5995911f32c3e63401eb36d89fe4146655c47e2a840f36"} Nov 03 13:33:52.043491 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:52.043322 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-dd2bbf-68fc6dd7d7-z55dw" Nov 03 13:33:52.043491 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:52.043409 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-dd2bbf-68fc6dd7d7-z55dw" event={"ID":"e553bdcf-e114-4046-8e10-4a2687d78a52","Type":"ContainerDied","Data":"742758944e7cbbaf9c9a61f6373c95e1fa77612e66c0f1c68578ff45db932539"} Nov 03 13:33:52.043491 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:52.043443 5024 scope.go:122] "RemoveContainer" containerID="fd9d2b375395c7849a5995911f32c3e63401eb36d89fe4146655c47e2a840f36" Nov 03 13:33:52.044963 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:52.044935 5024 generic.go:356] "Generic (PLEG): container finished" podID="50a9026b-b55f-4650-988a-755a758dca86" containerID="c9aebf711b98782cfbb203dc381cc5769078119ced9b613f92b9e3a96d0983c0" exitCode=0 Nov 03 13:33:52.045074 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:52.044994 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-dd2bbf-74fb8cb9c9-95hxq" Nov 03 13:33:52.045074 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:52.045034 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-dd2bbf-74fb8cb9c9-95hxq" event={"ID":"50a9026b-b55f-4650-988a-755a758dca86","Type":"ContainerDied","Data":"c9aebf711b98782cfbb203dc381cc5769078119ced9b613f92b9e3a96d0983c0"} Nov 03 13:33:52.045186 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:52.045081 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-dd2bbf-74fb8cb9c9-95hxq" event={"ID":"50a9026b-b55f-4650-988a-755a758dca86","Type":"ContainerDied","Data":"a8d86ac2cad62f9016a50b3489e5af54cd2c98f0d501ead33dbec2a3625f853c"} Nov 03 13:33:52.056728 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:52.056709 5024 scope.go:122] "RemoveContainer" containerID="fd9d2b375395c7849a5995911f32c3e63401eb36d89fe4146655c47e2a840f36" Nov 03 13:33:52.057054 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:33:52.057021 5024 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd9d2b375395c7849a5995911f32c3e63401eb36d89fe4146655c47e2a840f36\": container with ID starting with fd9d2b375395c7849a5995911f32c3e63401eb36d89fe4146655c47e2a840f36 not found: ID does not exist" containerID="fd9d2b375395c7849a5995911f32c3e63401eb36d89fe4146655c47e2a840f36" Nov 03 13:33:52.057142 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:52.057067 5024 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd9d2b375395c7849a5995911f32c3e63401eb36d89fe4146655c47e2a840f36"} err="failed to get container status \"fd9d2b375395c7849a5995911f32c3e63401eb36d89fe4146655c47e2a840f36\": rpc error: code = NotFound desc = could not find container \"fd9d2b375395c7849a5995911f32c3e63401eb36d89fe4146655c47e2a840f36\": container with ID starting with fd9d2b375395c7849a5995911f32c3e63401eb36d89fe4146655c47e2a840f36 not found: ID does not exist" Nov 03 13:33:52.057142 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:52.057091 5024 scope.go:122] "RemoveContainer" containerID="c9aebf711b98782cfbb203dc381cc5769078119ced9b613f92b9e3a96d0983c0" Nov 03 13:33:52.068530 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:52.068507 5024 scope.go:122] "RemoveContainer" containerID="c9aebf711b98782cfbb203dc381cc5769078119ced9b613f92b9e3a96d0983c0" Nov 03 13:33:52.068813 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:33:52.068792 5024 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c9aebf711b98782cfbb203dc381cc5769078119ced9b613f92b9e3a96d0983c0\": container with ID starting with c9aebf711b98782cfbb203dc381cc5769078119ced9b613f92b9e3a96d0983c0 not found: ID does not exist" containerID="c9aebf711b98782cfbb203dc381cc5769078119ced9b613f92b9e3a96d0983c0" Nov 03 13:33:52.068903 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:52.068819 5024 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9aebf711b98782cfbb203dc381cc5769078119ced9b613f92b9e3a96d0983c0"} err="failed to get container status \"c9aebf711b98782cfbb203dc381cc5769078119ced9b613f92b9e3a96d0983c0\": rpc error: code = NotFound desc = could not find container \"c9aebf711b98782cfbb203dc381cc5769078119ced9b613f92b9e3a96d0983c0\": container with ID starting with c9aebf711b98782cfbb203dc381cc5769078119ced9b613f92b9e3a96d0983c0 not found: ID does not exist" Nov 03 13:33:52.117781 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:52.117746 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-dd2bbf-68fc6dd7d7-z55dw"] Nov 03 13:33:52.117947 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:52.117795 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-dd2bbf-68fc6dd7d7-z55dw"] Nov 03 13:33:52.117947 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:52.117831 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-dd2bbf-74fb8cb9c9-95hxq"] Nov 03 13:33:52.118925 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:52.118900 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-dd2bbf-74fb8cb9c9-95hxq"] Nov 03 13:33:52.691308 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:52.690653 5024 patch_prober.go:29] interesting pod/e2e-mt-1ec4b892-openshift-default-6c8678f7c-prh98 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body= Nov 03 13:33:52.691308 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:52.690728 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-1ec4b892-openshift-default-6c8678f7c-prh98" podUID="752f4211-a41b-4b2e-b568-3b3bc1ce0989" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" Nov 03 13:33:53.742188 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:53.742147 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-1ec4b892-openshift-default-6c8678f7c-prh98" Nov 03 13:33:53.745088 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:53.745059 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-1ec4b892-openshift-default-6c8678f7c-prh98" Nov 03 13:33:54.113823 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:54.113786 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="50a9026b-b55f-4650-988a-755a758dca86" path="/var/lib/kubelet/pods/50a9026b-b55f-4650-988a-755a758dca86/volumes" Nov 03 13:33:54.114577 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:33:54.114551 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e553bdcf-e114-4046-8e10-4a2687d78a52" path="/var/lib/kubelet/pods/e553bdcf-e114-4046-8e10-4a2687d78a52/volumes" Nov 03 13:34:01.096109 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:34:01.096065 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-dd2bbf-openshift-default-7fd7f76f49-gf5kt"] Nov 03 13:34:01.096685 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:34:01.096360 5024 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-dd2bbf-openshift-default-7fd7f76f49-gf5kt" podUID="8b6d7058-b5f5-4b35-a0ad-967187e9ba3b" containerName="istio-proxy" containerID="cri-o://d722e94bf1775221ae03d4eca896479d933d868d6f033c2f6fa7a7e0e48fc120" gracePeriod=30 Nov 03 13:34:06.454862 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:34:06.454769 5024 generic.go:356] "Generic (PLEG): container finished" podID="8b6d7058-b5f5-4b35-a0ad-967187e9ba3b" containerID="d722e94bf1775221ae03d4eca896479d933d868d6f033c2f6fa7a7e0e48fc120" exitCode=0 Nov 03 13:34:06.455394 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:34:06.455366 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-dd2bbf-openshift-default-7fd7f76f49-gf5kt" event={"ID":"8b6d7058-b5f5-4b35-a0ad-967187e9ba3b","Type":"ContainerDied","Data":"d722e94bf1775221ae03d4eca896479d933d868d6f033c2f6fa7a7e0e48fc120"} Nov 03 13:34:06.455572 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:34:06.455526 5024 scope.go:122] "RemoveContainer" containerID="b52a553b51d07d9678f8fdc0f5d77dd37b70e785ca19068e555e14a4263fd9bf" Nov 03 13:34:06.531519 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:34:06.531489 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-dd2bbf-openshift-default-7fd7f76f49-gf5kt" Nov 03 13:34:06.607626 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:34:06.607578 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b6d7058-b5f5-4b35-a0ad-967187e9ba3b-kube-api-access-fgxqm\" (UniqueName: \"kubernetes.io/projected/8b6d7058-b5f5-4b35-a0ad-967187e9ba3b-kube-api-access-fgxqm\") pod \"8b6d7058-b5f5-4b35-a0ad-967187e9ba3b\" (UID: \"8b6d7058-b5f5-4b35-a0ad-967187e9ba3b\") " Nov 03 13:34:06.607804 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:34:06.607650 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b6d7058-b5f5-4b35-a0ad-967187e9ba3b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b6d7058-b5f5-4b35-a0ad-967187e9ba3b-istio-data\") pod \"8b6d7058-b5f5-4b35-a0ad-967187e9ba3b\" (UID: \"8b6d7058-b5f5-4b35-a0ad-967187e9ba3b\") " Nov 03 13:34:06.607804 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:34:06.607758 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8b6d7058-b5f5-4b35-a0ad-967187e9ba3b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b6d7058-b5f5-4b35-a0ad-967187e9ba3b-istiod-ca-cert\") pod \"8b6d7058-b5f5-4b35-a0ad-967187e9ba3b\" (UID: \"8b6d7058-b5f5-4b35-a0ad-967187e9ba3b\") " Nov 03 13:34:06.607804 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:34:06.607788 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b6d7058-b5f5-4b35-a0ad-967187e9ba3b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b6d7058-b5f5-4b35-a0ad-967187e9ba3b-credential-socket\") pod \"8b6d7058-b5f5-4b35-a0ad-967187e9ba3b\" (UID: \"8b6d7058-b5f5-4b35-a0ad-967187e9ba3b\") " Nov 03 13:34:06.607912 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:34:06.607823 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8b6d7058-b5f5-4b35-a0ad-967187e9ba3b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b6d7058-b5f5-4b35-a0ad-967187e9ba3b-istio-podinfo\") pod \"8b6d7058-b5f5-4b35-a0ad-967187e9ba3b\" (UID: \"8b6d7058-b5f5-4b35-a0ad-967187e9ba3b\") " Nov 03 13:34:06.607976 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:34:06.607942 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b6d7058-b5f5-4b35-a0ad-967187e9ba3b-istio-data" pod "8b6d7058-b5f5-4b35-a0ad-967187e9ba3b" (UID: "8b6d7058-b5f5-4b35-a0ad-967187e9ba3b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:34:06.608015 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:34:06.607960 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b6d7058-b5f5-4b35-a0ad-967187e9ba3b-istio-token\" (UniqueName: \"kubernetes.io/projected/8b6d7058-b5f5-4b35-a0ad-967187e9ba3b-istio-token\") pod \"8b6d7058-b5f5-4b35-a0ad-967187e9ba3b\" (UID: \"8b6d7058-b5f5-4b35-a0ad-967187e9ba3b\") " Nov 03 13:34:06.608086 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:34:06.608066 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b6d7058-b5f5-4b35-a0ad-967187e9ba3b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b6d7058-b5f5-4b35-a0ad-967187e9ba3b-istio-envoy\") pod \"8b6d7058-b5f5-4b35-a0ad-967187e9ba3b\" (UID: \"8b6d7058-b5f5-4b35-a0ad-967187e9ba3b\") " Nov 03 13:34:06.608997 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:34:06.608277 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b6d7058-b5f5-4b35-a0ad-967187e9ba3b-credential-socket" pod "8b6d7058-b5f5-4b35-a0ad-967187e9ba3b" (UID: "8b6d7058-b5f5-4b35-a0ad-967187e9ba3b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:34:06.608997 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:34:06.608399 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b6d7058-b5f5-4b35-a0ad-967187e9ba3b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b6d7058-b5f5-4b35-a0ad-967187e9ba3b-workload-certs\") pod \"8b6d7058-b5f5-4b35-a0ad-967187e9ba3b\" (UID: \"8b6d7058-b5f5-4b35-a0ad-967187e9ba3b\") " Nov 03 13:34:06.608997 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:34:06.608444 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b6d7058-b5f5-4b35-a0ad-967187e9ba3b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b6d7058-b5f5-4b35-a0ad-967187e9ba3b-workload-socket\") pod \"8b6d7058-b5f5-4b35-a0ad-967187e9ba3b\" (UID: \"8b6d7058-b5f5-4b35-a0ad-967187e9ba3b\") " Nov 03 13:34:06.608997 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:34:06.608506 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b6d7058-b5f5-4b35-a0ad-967187e9ba3b-istiod-ca-cert" pod "8b6d7058-b5f5-4b35-a0ad-967187e9ba3b" (UID: "8b6d7058-b5f5-4b35-a0ad-967187e9ba3b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:34:06.608997 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:34:06.608753 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b6d7058-b5f5-4b35-a0ad-967187e9ba3b-workload-certs" pod "8b6d7058-b5f5-4b35-a0ad-967187e9ba3b" (UID: "8b6d7058-b5f5-4b35-a0ad-967187e9ba3b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:34:06.608997 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:34:06.608965 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b6d7058-b5f5-4b35-a0ad-967187e9ba3b-workload-socket" pod "8b6d7058-b5f5-4b35-a0ad-967187e9ba3b" (UID: "8b6d7058-b5f5-4b35-a0ad-967187e9ba3b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:34:06.609476 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:34:06.609367 5024 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b6d7058-b5f5-4b35-a0ad-967187e9ba3b-workload-certs\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:34:06.609476 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:34:06.609388 5024 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b6d7058-b5f5-4b35-a0ad-967187e9ba3b-workload-socket\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:34:06.609476 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:34:06.609404 5024 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b6d7058-b5f5-4b35-a0ad-967187e9ba3b-istio-data\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:34:06.609476 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:34:06.609419 5024 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b6d7058-b5f5-4b35-a0ad-967187e9ba3b-istiod-ca-cert\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:34:06.609476 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:34:06.609434 5024 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b6d7058-b5f5-4b35-a0ad-967187e9ba3b-credential-socket\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:34:06.610368 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:34:06.610342 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b6d7058-b5f5-4b35-a0ad-967187e9ba3b-istio-token" pod "8b6d7058-b5f5-4b35-a0ad-967187e9ba3b" (UID: "8b6d7058-b5f5-4b35-a0ad-967187e9ba3b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:34:06.610483 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:34:06.610454 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8b6d7058-b5f5-4b35-a0ad-967187e9ba3b-istio-podinfo" pod "8b6d7058-b5f5-4b35-a0ad-967187e9ba3b" (UID: "8b6d7058-b5f5-4b35-a0ad-967187e9ba3b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 13:34:06.610572 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:34:06.610555 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b6d7058-b5f5-4b35-a0ad-967187e9ba3b-kube-api-access-fgxqm" pod "8b6d7058-b5f5-4b35-a0ad-967187e9ba3b" (UID: "8b6d7058-b5f5-4b35-a0ad-967187e9ba3b"). InnerVolumeSpecName "kube-api-access-fgxqm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:34:06.614613 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:34:06.614576 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b6d7058-b5f5-4b35-a0ad-967187e9ba3b-istio-envoy" pod "8b6d7058-b5f5-4b35-a0ad-967187e9ba3b" (UID: "8b6d7058-b5f5-4b35-a0ad-967187e9ba3b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:34:06.711238 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:34:06.711141 5024 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b6d7058-b5f5-4b35-a0ad-967187e9ba3b-istio-token\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:34:06.711238 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:34:06.711176 5024 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b6d7058-b5f5-4b35-a0ad-967187e9ba3b-istio-envoy\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:34:06.711238 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:34:06.711192 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fgxqm\" (UniqueName: \"kubernetes.io/projected/8b6d7058-b5f5-4b35-a0ad-967187e9ba3b-kube-api-access-fgxqm\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:34:06.711238 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:34:06.711208 5024 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b6d7058-b5f5-4b35-a0ad-967187e9ba3b-istio-podinfo\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:34:07.037361 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:34:07.031130 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-418af8-55df5c84df-vjqhm"] Nov 03 13:34:07.510323 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:34:07.506855 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-dd2bbf-openshift-default-7fd7f76f49-gf5kt" event={"ID":"8b6d7058-b5f5-4b35-a0ad-967187e9ba3b","Type":"ContainerDied","Data":"d1b62e1e1b715516a5f50f87975e482e33021932574c00d999d2f4d717d00353"} Nov 03 13:34:07.510323 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:34:07.506931 5024 scope.go:122] "RemoveContainer" containerID="d722e94bf1775221ae03d4eca896479d933d868d6f033c2f6fa7a7e0e48fc120" Nov 03 13:34:07.510323 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:34:07.507055 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-dd2bbf-openshift-default-7fd7f76f49-gf5kt" Nov 03 13:34:07.766563 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:34:07.766478 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-dd2bbf-openshift-default-7fd7f76f49-gf5kt"] Nov 03 13:34:07.773152 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:34:07.773109 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-dd2bbf-openshift-default-7fd7f76f49-gf5kt"] Nov 03 13:34:08.111164 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:34:08.111131 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b6d7058-b5f5-4b35-a0ad-967187e9ba3b" path="/var/lib/kubelet/pods/8b6d7058-b5f5-4b35-a0ad-967187e9ba3b/volumes" Nov 03 13:34:09.516463 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:34:09.516417 5024 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d7mwz" containerName="extractor" Nov 03 13:34:09.672533 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:34:09.672487 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-418af8-9dd6d7ffd-6wx9j"] Nov 03 13:34:09.672790 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:34:09.672744 5024 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-418af8-9dd6d7ffd-6wx9j" podUID="7adbc4a3-4489-45fd-a5d6-1b83a240af2a" containerName="maas-api" containerID="cri-o://807432c1dd6bb4de2cb84e8918ea89b64ca2d9ac99296645a7732dd017236f7b" gracePeriod=30 Nov 03 13:34:10.393170 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:34:10.393086 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-418af8-9dd6d7ffd-6wx9j" Nov 03 13:34:10.543389 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:34:10.543344 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7adbc4a3-4489-45fd-a5d6-1b83a240af2a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7adbc4a3-4489-45fd-a5d6-1b83a240af2a-metrics-certs\") pod \"7adbc4a3-4489-45fd-a5d6-1b83a240af2a\" (UID: \"7adbc4a3-4489-45fd-a5d6-1b83a240af2a\") " Nov 03 13:34:10.543809 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:34:10.543483 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7adbc4a3-4489-45fd-a5d6-1b83a240af2a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7adbc4a3-4489-45fd-a5d6-1b83a240af2a-maas-api-tls\") pod \"7adbc4a3-4489-45fd-a5d6-1b83a240af2a\" (UID: \"7adbc4a3-4489-45fd-a5d6-1b83a240af2a\") " Nov 03 13:34:10.543809 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:34:10.543509 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7adbc4a3-4489-45fd-a5d6-1b83a240af2a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7adbc4a3-4489-45fd-a5d6-1b83a240af2a-openshift-service-ca\") pod \"7adbc4a3-4489-45fd-a5d6-1b83a240af2a\" (UID: \"7adbc4a3-4489-45fd-a5d6-1b83a240af2a\") " Nov 03 13:34:10.543809 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:34:10.543527 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7adbc4a3-4489-45fd-a5d6-1b83a240af2a-kube-api-access-z9s9x\" (UniqueName: \"kubernetes.io/projected/7adbc4a3-4489-45fd-a5d6-1b83a240af2a-kube-api-access-z9s9x\") pod \"7adbc4a3-4489-45fd-a5d6-1b83a240af2a\" (UID: \"7adbc4a3-4489-45fd-a5d6-1b83a240af2a\") " Nov 03 13:34:10.544698 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:34:10.544663 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7adbc4a3-4489-45fd-a5d6-1b83a240af2a-openshift-service-ca" pod "7adbc4a3-4489-45fd-a5d6-1b83a240af2a" (UID: "7adbc4a3-4489-45fd-a5d6-1b83a240af2a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:34:10.546820 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:34:10.546788 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7adbc4a3-4489-45fd-a5d6-1b83a240af2a-kube-api-access-z9s9x" pod "7adbc4a3-4489-45fd-a5d6-1b83a240af2a" (UID: "7adbc4a3-4489-45fd-a5d6-1b83a240af2a"). InnerVolumeSpecName "kube-api-access-z9s9x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:34:10.546930 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:34:10.546823 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7adbc4a3-4489-45fd-a5d6-1b83a240af2a-metrics-certs" pod "7adbc4a3-4489-45fd-a5d6-1b83a240af2a" (UID: "7adbc4a3-4489-45fd-a5d6-1b83a240af2a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 13:34:10.546930 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:34:10.546914 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7adbc4a3-4489-45fd-a5d6-1b83a240af2a-maas-api-tls" pod "7adbc4a3-4489-45fd-a5d6-1b83a240af2a" (UID: "7adbc4a3-4489-45fd-a5d6-1b83a240af2a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 13:34:10.615117 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:34:10.615081 5024 generic.go:356] "Generic (PLEG): container finished" podID="7adbc4a3-4489-45fd-a5d6-1b83a240af2a" containerID="807432c1dd6bb4de2cb84e8918ea89b64ca2d9ac99296645a7732dd017236f7b" exitCode=0 Nov 03 13:34:10.615344 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:34:10.615147 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-418af8-9dd6d7ffd-6wx9j" Nov 03 13:34:10.615344 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:34:10.615172 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-418af8-9dd6d7ffd-6wx9j" event={"ID":"7adbc4a3-4489-45fd-a5d6-1b83a240af2a","Type":"ContainerDied","Data":"807432c1dd6bb4de2cb84e8918ea89b64ca2d9ac99296645a7732dd017236f7b"} Nov 03 13:34:10.615344 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:34:10.615246 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-418af8-9dd6d7ffd-6wx9j" event={"ID":"7adbc4a3-4489-45fd-a5d6-1b83a240af2a","Type":"ContainerDied","Data":"67175150240a69e4b7242461c1563aee9f05dbc1bdae53f749b2670675590ffe"} Nov 03 13:34:10.615344 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:34:10.615321 5024 scope.go:122] "RemoveContainer" containerID="807432c1dd6bb4de2cb84e8918ea89b64ca2d9ac99296645a7732dd017236f7b" Nov 03 13:34:10.627545 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:34:10.627522 5024 scope.go:122] "RemoveContainer" containerID="807432c1dd6bb4de2cb84e8918ea89b64ca2d9ac99296645a7732dd017236f7b" Nov 03 13:34:10.627886 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:34:10.627864 5024 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"807432c1dd6bb4de2cb84e8918ea89b64ca2d9ac99296645a7732dd017236f7b\": container with ID starting with 807432c1dd6bb4de2cb84e8918ea89b64ca2d9ac99296645a7732dd017236f7b not found: ID does not exist" containerID="807432c1dd6bb4de2cb84e8918ea89b64ca2d9ac99296645a7732dd017236f7b" Nov 03 13:34:10.627958 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:34:10.627896 5024 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"807432c1dd6bb4de2cb84e8918ea89b64ca2d9ac99296645a7732dd017236f7b"} err="failed to get container status \"807432c1dd6bb4de2cb84e8918ea89b64ca2d9ac99296645a7732dd017236f7b\": rpc error: code = NotFound desc = could not find container \"807432c1dd6bb4de2cb84e8918ea89b64ca2d9ac99296645a7732dd017236f7b\": container with ID starting with 807432c1dd6bb4de2cb84e8918ea89b64ca2d9ac99296645a7732dd017236f7b not found: ID does not exist" Nov 03 13:34:10.642726 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:34:10.642682 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-418af8-9dd6d7ffd-6wx9j"] Nov 03 13:34:10.646130 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:34:10.646049 5024 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7adbc4a3-4489-45fd-a5d6-1b83a240af2a-metrics-certs\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:34:10.646130 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:34:10.646083 5024 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7adbc4a3-4489-45fd-a5d6-1b83a240af2a-maas-api-tls\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:34:10.646130 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:34:10.646099 5024 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7adbc4a3-4489-45fd-a5d6-1b83a240af2a-openshift-service-ca\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:34:10.646130 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:34:10.646114 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z9s9x\" (UniqueName: \"kubernetes.io/projected/7adbc4a3-4489-45fd-a5d6-1b83a240af2a-kube-api-access-z9s9x\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:34:10.646701 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:34:10.646671 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-418af8-9dd6d7ffd-6wx9j"] Nov 03 13:34:12.032185 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:34:12.032108 5024 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-418af8-55df5c84df-vjqhm" podUID="1f4853bc-d6b2-40dd-b779-ddb415176c1a" containerName="payload-processing" containerID="cri-o://4e2c6b771e344aeda062b8846a308ab6631a210cd71f874fe821abe192d7be3d" gracePeriod=25 Nov 03 13:34:12.111988 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:34:12.111943 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7adbc4a3-4489-45fd-a5d6-1b83a240af2a" path="/var/lib/kubelet/pods/7adbc4a3-4489-45fd-a5d6-1b83a240af2a/volumes" Nov 03 13:34:12.420353 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:34:12.420274 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-418af8-55df5c84df-vjqhm" Nov 03 13:34:12.613631 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:34:12.613511 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f4853bc-d6b2-40dd-b779-ddb415176c1a-kube-api-access-jt57d\" (UniqueName: \"kubernetes.io/projected/1f4853bc-d6b2-40dd-b779-ddb415176c1a-kube-api-access-jt57d\") pod \"1f4853bc-d6b2-40dd-b779-ddb415176c1a\" (UID: \"1f4853bc-d6b2-40dd-b779-ddb415176c1a\") " Nov 03 13:34:12.613920 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:34:12.613743 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f4853bc-d6b2-40dd-b779-ddb415176c1a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f4853bc-d6b2-40dd-b779-ddb415176c1a-plugins-config-volume\") pod \"1f4853bc-d6b2-40dd-b779-ddb415176c1a\" (UID: \"1f4853bc-d6b2-40dd-b779-ddb415176c1a\") " Nov 03 13:34:12.614470 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:34:12.614430 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f4853bc-d6b2-40dd-b779-ddb415176c1a-plugins-config-volume" pod "1f4853bc-d6b2-40dd-b779-ddb415176c1a" (UID: "1f4853bc-d6b2-40dd-b779-ddb415176c1a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:34:12.614775 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:34:12.614723 5024 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f4853bc-d6b2-40dd-b779-ddb415176c1a-plugins-config-volume\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:34:12.616527 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:34:12.616301 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f4853bc-d6b2-40dd-b779-ddb415176c1a-kube-api-access-jt57d" pod "1f4853bc-d6b2-40dd-b779-ddb415176c1a" (UID: "1f4853bc-d6b2-40dd-b779-ddb415176c1a"). InnerVolumeSpecName "kube-api-access-jt57d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:34:12.635869 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:34:12.635834 5024 generic.go:356] "Generic (PLEG): container finished" podID="1f4853bc-d6b2-40dd-b779-ddb415176c1a" containerID="4e2c6b771e344aeda062b8846a308ab6631a210cd71f874fe821abe192d7be3d" exitCode=0 Nov 03 13:34:12.636027 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:34:12.635880 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-418af8-55df5c84df-vjqhm" Nov 03 13:34:12.636027 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:34:12.635915 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-418af8-55df5c84df-vjqhm" event={"ID":"1f4853bc-d6b2-40dd-b779-ddb415176c1a","Type":"ContainerDied","Data":"4e2c6b771e344aeda062b8846a308ab6631a210cd71f874fe821abe192d7be3d"} Nov 03 13:34:12.636027 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:34:12.635971 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-418af8-55df5c84df-vjqhm" event={"ID":"1f4853bc-d6b2-40dd-b779-ddb415176c1a","Type":"ContainerDied","Data":"4e005f906d60298e3879b88a2a89f45337d826848c932c10e4765f20cd7dff1d"} Nov 03 13:34:12.636027 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:34:12.635996 5024 scope.go:122] "RemoveContainer" containerID="4e2c6b771e344aeda062b8846a308ab6631a210cd71f874fe821abe192d7be3d" Nov 03 13:34:12.647313 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:34:12.647278 5024 scope.go:122] "RemoveContainer" containerID="4e2c6b771e344aeda062b8846a308ab6631a210cd71f874fe821abe192d7be3d" Nov 03 13:34:12.647629 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:34:12.647612 5024 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e2c6b771e344aeda062b8846a308ab6631a210cd71f874fe821abe192d7be3d\": container with ID starting with 4e2c6b771e344aeda062b8846a308ab6631a210cd71f874fe821abe192d7be3d not found: ID does not exist" containerID="4e2c6b771e344aeda062b8846a308ab6631a210cd71f874fe821abe192d7be3d" Nov 03 13:34:12.647665 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:34:12.647638 5024 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e2c6b771e344aeda062b8846a308ab6631a210cd71f874fe821abe192d7be3d"} err="failed to get container status \"4e2c6b771e344aeda062b8846a308ab6631a210cd71f874fe821abe192d7be3d\": rpc error: code = NotFound desc = could not find container \"4e2c6b771e344aeda062b8846a308ab6631a210cd71f874fe821abe192d7be3d\": container with ID starting with 4e2c6b771e344aeda062b8846a308ab6631a210cd71f874fe821abe192d7be3d not found: ID does not exist" Nov 03 13:34:12.667564 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:34:12.667521 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-418af8-55df5c84df-vjqhm"] Nov 03 13:34:12.672237 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:34:12.672206 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-418af8-55df5c84df-vjqhm"] Nov 03 13:34:12.726344 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:34:12.716876 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jt57d\" (UniqueName: \"kubernetes.io/projected/1f4853bc-d6b2-40dd-b779-ddb415176c1a-kube-api-access-jt57d\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:34:14.111387 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:34:14.111347 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f4853bc-d6b2-40dd-b779-ddb415176c1a" path="/var/lib/kubelet/pods/1f4853bc-d6b2-40dd-b779-ddb415176c1a/volumes" Nov 03 13:34:17.606714 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:34:17.606665 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-2cwvx"] Nov 03 13:34:17.607523 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:34:17.607501 5024 cpu_manager.go:414] "containerMap: removing container" podUID="c8657989-e2c5-4dfc-b32a-3491ad189395" containerName="istio-proxy" Nov 03 13:34:17.607523 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:34:17.607522 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8657989-e2c5-4dfc-b32a-3491ad189395" containerName="istio-proxy" Nov 03 13:34:17.607688 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:34:17.607533 5024 cpu_manager.go:414] "containerMap: removing container" podUID="50a9026b-b55f-4650-988a-755a758dca86" containerName="payload-processing" Nov 03 13:34:17.607688 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:34:17.607538 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50a9026b-b55f-4650-988a-755a758dca86" containerName="payload-processing" Nov 03 13:34:17.607688 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:34:17.607547 5024 cpu_manager.go:414] "containerMap: removing container" podUID="1727674a-caa9-4d85-8ee8-45bee8538927" containerName="istio-proxy" Nov 03 13:34:17.607688 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:34:17.607553 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1727674a-caa9-4d85-8ee8-45bee8538927" containerName="istio-proxy" Nov 03 13:34:17.607688 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:34:17.607581 5024 cpu_manager.go:414] "containerMap: removing container" podUID="8b6d7058-b5f5-4b35-a0ad-967187e9ba3b" containerName="istio-proxy" Nov 03 13:34:17.607688 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:34:17.607586 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b6d7058-b5f5-4b35-a0ad-967187e9ba3b" containerName="istio-proxy" Nov 03 13:34:17.607688 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:34:17.607597 5024 cpu_manager.go:414] "containerMap: removing container" podUID="19ea0757-f048-4f73-9091-cf3a0d86dd12" containerName="istio-proxy" Nov 03 13:34:17.607688 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:34:17.607602 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19ea0757-f048-4f73-9091-cf3a0d86dd12" containerName="istio-proxy" Nov 03 13:34:17.607688 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:34:17.607623 5024 cpu_manager.go:414] "containerMap: removing container" podUID="7adbc4a3-4489-45fd-a5d6-1b83a240af2a" containerName="maas-api" Nov 03 13:34:17.607688 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:34:17.607628 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7adbc4a3-4489-45fd-a5d6-1b83a240af2a" containerName="maas-api" Nov 03 13:34:17.607688 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:34:17.607638 5024 cpu_manager.go:414] "containerMap: removing container" podUID="2ca8b6af-59bf-4d09-ba65-9d79a4e2faba" containerName="istio-proxy" Nov 03 13:34:17.607688 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:34:17.607642 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ca8b6af-59bf-4d09-ba65-9d79a4e2faba" containerName="istio-proxy" Nov 03 13:34:17.607688 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:34:17.607653 5024 cpu_manager.go:414] "containerMap: removing container" podUID="8b6d7058-b5f5-4b35-a0ad-967187e9ba3b" containerName="istio-proxy" Nov 03 13:34:17.607688 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:34:17.607658 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b6d7058-b5f5-4b35-a0ad-967187e9ba3b" containerName="istio-proxy" Nov 03 13:34:17.607688 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:34:17.607663 5024 cpu_manager.go:414] "containerMap: removing container" podUID="e553bdcf-e114-4046-8e10-4a2687d78a52" containerName="payload-pre-processing" Nov 03 13:34:17.607688 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:34:17.607668 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e553bdcf-e114-4046-8e10-4a2687d78a52" containerName="payload-pre-processing" Nov 03 13:34:17.607688 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:34:17.607679 5024 cpu_manager.go:414] "containerMap: removing container" podUID="1f4853bc-d6b2-40dd-b779-ddb415176c1a" containerName="payload-processing" Nov 03 13:34:17.607688 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:34:17.607684 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f4853bc-d6b2-40dd-b779-ddb415176c1a" containerName="payload-processing" Nov 03 13:34:17.608638 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:34:17.607802 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b6d7058-b5f5-4b35-a0ad-967187e9ba3b" containerName="istio-proxy" Nov 03 13:34:17.608638 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:34:17.607812 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="19ea0757-f048-4f73-9091-cf3a0d86dd12" containerName="istio-proxy" Nov 03 13:34:17.608638 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:34:17.607823 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="7adbc4a3-4489-45fd-a5d6-1b83a240af2a" containerName="maas-api" Nov 03 13:34:17.608638 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:34:17.607828 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ca8b6af-59bf-4d09-ba65-9d79a4e2faba" containerName="istio-proxy" Nov 03 13:34:17.608638 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:34:17.607835 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b6d7058-b5f5-4b35-a0ad-967187e9ba3b" containerName="istio-proxy" Nov 03 13:34:17.608638 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:34:17.607842 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="e553bdcf-e114-4046-8e10-4a2687d78a52" containerName="payload-pre-processing" Nov 03 13:34:17.608638 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:34:17.607850 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f4853bc-d6b2-40dd-b779-ddb415176c1a" containerName="payload-processing" Nov 03 13:34:17.608638 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:34:17.607856 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8657989-e2c5-4dfc-b32a-3491ad189395" containerName="istio-proxy" Nov 03 13:34:17.608638 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:34:17.607861 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="50a9026b-b55f-4650-988a-755a758dca86" containerName="payload-processing" Nov 03 13:34:17.608638 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:34:17.607866 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="1727674a-caa9-4d85-8ee8-45bee8538927" containerName="istio-proxy" Nov 03 13:34:17.625452 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:34:17.625390 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-2cwvx" Nov 03 13:34:17.631586 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:34:17.631547 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-2cwvx"] Nov 03 13:34:17.649540 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:34:17.649504 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-a724a6-openshift-default-dockercfg-l5hxx\"" Nov 03 13:34:17.737356 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:34:17.724375 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xsqv9\" (UniqueName: \"kubernetes.io/projected/7d6b62e0-65bf-4766-93f4-b065c82304fb-kube-api-access-xsqv9\") pod \"e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-2cwvx\" (UID: \"7d6b62e0-65bf-4766-93f4-b065c82304fb\") " pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-2cwvx" Nov 03 13:34:17.737356 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:34:17.724496 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d6b62e0-65bf-4766-93f4-b065c82304fb-workload-socket\") pod \"e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-2cwvx\" (UID: \"7d6b62e0-65bf-4766-93f4-b065c82304fb\") " pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-2cwvx" Nov 03 13:34:17.737356 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:34:17.724608 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d6b62e0-65bf-4766-93f4-b065c82304fb-istiod-ca-cert\") pod \"e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-2cwvx\" (UID: \"7d6b62e0-65bf-4766-93f4-b065c82304fb\") " pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-2cwvx" Nov 03 13:34:17.737356 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:34:17.724636 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d6b62e0-65bf-4766-93f4-b065c82304fb-istio-token\") pod \"e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-2cwvx\" (UID: \"7d6b62e0-65bf-4766-93f4-b065c82304fb\") " pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-2cwvx" Nov 03 13:34:17.737356 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:34:17.724728 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d6b62e0-65bf-4766-93f4-b065c82304fb-istio-podinfo\") pod \"e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-2cwvx\" (UID: \"7d6b62e0-65bf-4766-93f4-b065c82304fb\") " pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-2cwvx" Nov 03 13:34:17.737356 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:34:17.724801 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d6b62e0-65bf-4766-93f4-b065c82304fb-istio-envoy\") pod \"e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-2cwvx\" (UID: \"7d6b62e0-65bf-4766-93f4-b065c82304fb\") " pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-2cwvx" Nov 03 13:34:17.737356 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:34:17.724835 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d6b62e0-65bf-4766-93f4-b065c82304fb-workload-certs\") pod \"e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-2cwvx\" (UID: \"7d6b62e0-65bf-4766-93f4-b065c82304fb\") " pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-2cwvx" Nov 03 13:34:17.737356 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:34:17.724883 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d6b62e0-65bf-4766-93f4-b065c82304fb-credential-socket\") pod \"e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-2cwvx\" (UID: \"7d6b62e0-65bf-4766-93f4-b065c82304fb\") " pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-2cwvx" Nov 03 13:34:17.737356 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:34:17.724967 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d6b62e0-65bf-4766-93f4-b065c82304fb-istio-data\") pod \"e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-2cwvx\" (UID: \"7d6b62e0-65bf-4766-93f4-b065c82304fb\") " pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-2cwvx" Nov 03 13:34:17.846365 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:34:17.832383 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d6b62e0-65bf-4766-93f4-b065c82304fb-workload-socket\") pod \"e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-2cwvx\" (UID: \"7d6b62e0-65bf-4766-93f4-b065c82304fb\") " pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-2cwvx" Nov 03 13:34:17.846365 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:34:17.832517 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d6b62e0-65bf-4766-93f4-b065c82304fb-istiod-ca-cert\") pod \"e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-2cwvx\" (UID: \"7d6b62e0-65bf-4766-93f4-b065c82304fb\") " pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-2cwvx" Nov 03 13:34:17.846365 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:34:17.832545 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d6b62e0-65bf-4766-93f4-b065c82304fb-istio-token\") pod \"e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-2cwvx\" (UID: \"7d6b62e0-65bf-4766-93f4-b065c82304fb\") " pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-2cwvx" Nov 03 13:34:17.846365 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:34:17.832635 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d6b62e0-65bf-4766-93f4-b065c82304fb-istio-podinfo\") pod \"e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-2cwvx\" (UID: \"7d6b62e0-65bf-4766-93f4-b065c82304fb\") " pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-2cwvx" Nov 03 13:34:17.846365 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:34:17.832710 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d6b62e0-65bf-4766-93f4-b065c82304fb-istio-envoy\") pod \"e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-2cwvx\" (UID: \"7d6b62e0-65bf-4766-93f4-b065c82304fb\") " pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-2cwvx" Nov 03 13:34:17.846365 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:34:17.832748 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d6b62e0-65bf-4766-93f4-b065c82304fb-workload-certs\") pod \"e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-2cwvx\" (UID: \"7d6b62e0-65bf-4766-93f4-b065c82304fb\") " pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-2cwvx" Nov 03 13:34:17.846365 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:34:17.832792 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d6b62e0-65bf-4766-93f4-b065c82304fb-credential-socket\") pod \"e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-2cwvx\" (UID: \"7d6b62e0-65bf-4766-93f4-b065c82304fb\") " pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-2cwvx" Nov 03 13:34:17.846365 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:34:17.832870 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d6b62e0-65bf-4766-93f4-b065c82304fb-istio-data\") pod \"e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-2cwvx\" (UID: \"7d6b62e0-65bf-4766-93f4-b065c82304fb\") " pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-2cwvx" Nov 03 13:34:17.846365 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:34:17.832963 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xsqv9\" (UniqueName: \"kubernetes.io/projected/7d6b62e0-65bf-4766-93f4-b065c82304fb-kube-api-access-xsqv9\") pod \"e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-2cwvx\" (UID: \"7d6b62e0-65bf-4766-93f4-b065c82304fb\") " pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-2cwvx" Nov 03 13:34:17.846365 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:34:17.833738 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d6b62e0-65bf-4766-93f4-b065c82304fb-workload-socket\") pod \"e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-2cwvx\" (UID: \"7d6b62e0-65bf-4766-93f4-b065c82304fb\") " pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-2cwvx" Nov 03 13:34:17.846365 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:34:17.834412 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d6b62e0-65bf-4766-93f4-b065c82304fb-istiod-ca-cert\") pod \"e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-2cwvx\" (UID: \"7d6b62e0-65bf-4766-93f4-b065c82304fb\") " pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-2cwvx" Nov 03 13:34:17.848155 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:34:17.848117 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d6b62e0-65bf-4766-93f4-b065c82304fb-workload-certs\") pod \"e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-2cwvx\" (UID: \"7d6b62e0-65bf-4766-93f4-b065c82304fb\") " pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-2cwvx" Nov 03 13:34:17.858780 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:34:17.858697 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d6b62e0-65bf-4766-93f4-b065c82304fb-credential-socket\") pod \"e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-2cwvx\" (UID: \"7d6b62e0-65bf-4766-93f4-b065c82304fb\") " pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-2cwvx" Nov 03 13:34:17.888427 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:34:17.878642 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d6b62e0-65bf-4766-93f4-b065c82304fb-istio-data\") pod \"e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-2cwvx\" (UID: \"7d6b62e0-65bf-4766-93f4-b065c82304fb\") " pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-2cwvx" Nov 03 13:34:17.914310 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:34:17.913242 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d6b62e0-65bf-4766-93f4-b065c82304fb-istio-token\") pod \"e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-2cwvx\" (UID: \"7d6b62e0-65bf-4766-93f4-b065c82304fb\") " pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-2cwvx" Nov 03 13:34:17.927306 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:34:17.920730 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d6b62e0-65bf-4766-93f4-b065c82304fb-istio-envoy\") pod \"e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-2cwvx\" (UID: \"7d6b62e0-65bf-4766-93f4-b065c82304fb\") " pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-2cwvx" Nov 03 13:34:17.941307 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:34:17.933079 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xsqv9\" (UniqueName: \"kubernetes.io/projected/7d6b62e0-65bf-4766-93f4-b065c82304fb-kube-api-access-xsqv9\") pod \"e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-2cwvx\" (UID: \"7d6b62e0-65bf-4766-93f4-b065c82304fb\") " pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-2cwvx" Nov 03 13:34:17.965503 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:34:17.962991 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d6b62e0-65bf-4766-93f4-b065c82304fb-istio-podinfo\") pod \"e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-2cwvx\" (UID: \"7d6b62e0-65bf-4766-93f4-b065c82304fb\") " pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-2cwvx" Nov 03 13:34:18.254696 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:34:18.254595 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-2cwvx" Nov 03 13:34:18.894442 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:34:18.894340 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-2cwvx"] Nov 03 13:34:19.028307 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:34:19.026858 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:34:19.028307 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:34:19.026965 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:34:19.028307 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:34:19.027015 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:34:20.139906 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:34:20.139870 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-2cwvx" event={"ID":"7d6b62e0-65bf-4766-93f4-b065c82304fb","Type":"ContainerStarted","Data":"3eed62c34bb403ff209a40eeeb098257d766a8681440034ee7d547e8b364ce98"} Nov 03 13:34:20.140343 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:34:20.139935 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-2cwvx" event={"ID":"7d6b62e0-65bf-4766-93f4-b065c82304fb","Type":"ContainerStarted","Data":"bd4e9e461757e70a821cf35531e7dc146453cf960245647af328754bfbe705e7"} Nov 03 13:34:20.167012 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:34:20.166955 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-2cwvx" podStartSLOduration=3.166940547 podStartE2EDuration="3.166940547s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:34:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:34:20.165027109 +0000 UTC m=+1540.173218389" watchObservedRunningTime="2026-11-03 13:34:20.166940547 +0000 UTC m=+1540.175131824" Nov 03 13:34:20.255672 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:34:20.255625 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-2cwvx" Nov 03 13:34:20.261636 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:34:20.261605 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-2cwvx" Nov 03 13:34:21.149108 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:34:21.149074 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-2cwvx" Nov 03 13:34:21.150109 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:34:21.150085 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-2cwvx" Nov 03 13:34:26.042831 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:34:26.042789 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-wkz2r"] Nov 03 13:34:26.043300 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:34:26.042996 5024 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-wkz2r" podUID="c49c4c7b-3182-4699-898a-5407d9a3fea1" containerName="istio-proxy" containerID="cri-o://a83ddae672e505fbb2fd4641a2fc8481b8c2a939713bcb369cde5647ec1eaae5" gracePeriod=30 Nov 03 13:34:31.325797 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:34:31.325329 5024 generic.go:356] "Generic (PLEG): container finished" podID="c49c4c7b-3182-4699-898a-5407d9a3fea1" containerID="a83ddae672e505fbb2fd4641a2fc8481b8c2a939713bcb369cde5647ec1eaae5" exitCode=0 Nov 03 13:34:31.325797 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:34:31.325582 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-wkz2r" event={"ID":"c49c4c7b-3182-4699-898a-5407d9a3fea1","Type":"ContainerDied","Data":"a83ddae672e505fbb2fd4641a2fc8481b8c2a939713bcb369cde5647ec1eaae5"} Nov 03 13:34:31.325797 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:34:31.325631 5024 scope.go:122] "RemoveContainer" containerID="fed93113530f4bcfe3c23ec37252c5c1ca8e3331bd12050e56155ad271ef7d79" Nov 03 13:34:31.425550 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:34:31.425519 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-wkz2r" Nov 03 13:34:31.457340 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:34:31.457300 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c49c4c7b-3182-4699-898a-5407d9a3fea1-kube-api-access-7fp4d\" (UniqueName: \"kubernetes.io/projected/c49c4c7b-3182-4699-898a-5407d9a3fea1-kube-api-access-7fp4d\") pod \"c49c4c7b-3182-4699-898a-5407d9a3fea1\" (UID: \"c49c4c7b-3182-4699-898a-5407d9a3fea1\") " Nov 03 13:34:31.457583 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:34:31.457411 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c49c4c7b-3182-4699-898a-5407d9a3fea1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c49c4c7b-3182-4699-898a-5407d9a3fea1-istio-data\") pod \"c49c4c7b-3182-4699-898a-5407d9a3fea1\" (UID: \"c49c4c7b-3182-4699-898a-5407d9a3fea1\") " Nov 03 13:34:31.457797 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:34:31.457754 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c49c4c7b-3182-4699-898a-5407d9a3fea1-istio-data" pod "c49c4c7b-3182-4699-898a-5407d9a3fea1" (UID: "c49c4c7b-3182-4699-898a-5407d9a3fea1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:34:31.458027 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:34:31.458009 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c49c4c7b-3182-4699-898a-5407d9a3fea1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c49c4c7b-3182-4699-898a-5407d9a3fea1-istio-envoy\") pod \"c49c4c7b-3182-4699-898a-5407d9a3fea1\" (UID: \"c49c4c7b-3182-4699-898a-5407d9a3fea1\") " Nov 03 13:34:31.458157 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:34:31.458136 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c49c4c7b-3182-4699-898a-5407d9a3fea1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c49c4c7b-3182-4699-898a-5407d9a3fea1-workload-certs\") pod \"c49c4c7b-3182-4699-898a-5407d9a3fea1\" (UID: \"c49c4c7b-3182-4699-898a-5407d9a3fea1\") " Nov 03 13:34:31.458278 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:34:31.458208 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c49c4c7b-3182-4699-898a-5407d9a3fea1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c49c4c7b-3182-4699-898a-5407d9a3fea1-workload-socket\") pod \"c49c4c7b-3182-4699-898a-5407d9a3fea1\" (UID: \"c49c4c7b-3182-4699-898a-5407d9a3fea1\") " Nov 03 13:34:31.458389 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:34:31.458373 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c49c4c7b-3182-4699-898a-5407d9a3fea1-istio-token\" (UniqueName: \"kubernetes.io/projected/c49c4c7b-3182-4699-898a-5407d9a3fea1-istio-token\") pod \"c49c4c7b-3182-4699-898a-5407d9a3fea1\" (UID: \"c49c4c7b-3182-4699-898a-5407d9a3fea1\") " Nov 03 13:34:31.458444 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:34:31.458430 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c49c4c7b-3182-4699-898a-5407d9a3fea1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c49c4c7b-3182-4699-898a-5407d9a3fea1-istio-podinfo\") pod \"c49c4c7b-3182-4699-898a-5407d9a3fea1\" (UID: \"c49c4c7b-3182-4699-898a-5407d9a3fea1\") " Nov 03 13:34:31.458770 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:34:31.458749 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c49c4c7b-3182-4699-898a-5407d9a3fea1-workload-certs" pod "c49c4c7b-3182-4699-898a-5407d9a3fea1" (UID: "c49c4c7b-3182-4699-898a-5407d9a3fea1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:34:31.459226 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:34:31.459187 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c49c4c7b-3182-4699-898a-5407d9a3fea1-workload-socket" pod "c49c4c7b-3182-4699-898a-5407d9a3fea1" (UID: "c49c4c7b-3182-4699-898a-5407d9a3fea1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:34:31.459537 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:34:31.459511 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c49c4c7b-3182-4699-898a-5407d9a3fea1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c49c4c7b-3182-4699-898a-5407d9a3fea1-istiod-ca-cert\") pod \"c49c4c7b-3182-4699-898a-5407d9a3fea1\" (UID: \"c49c4c7b-3182-4699-898a-5407d9a3fea1\") " Nov 03 13:34:31.459646 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:34:31.459635 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c49c4c7b-3182-4699-898a-5407d9a3fea1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c49c4c7b-3182-4699-898a-5407d9a3fea1-credential-socket\") pod \"c49c4c7b-3182-4699-898a-5407d9a3fea1\" (UID: \"c49c4c7b-3182-4699-898a-5407d9a3fea1\") " Nov 03 13:34:31.460588 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:34:31.460563 5024 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c49c4c7b-3182-4699-898a-5407d9a3fea1-istio-data\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:34:31.460686 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:34:31.460593 5024 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c49c4c7b-3182-4699-898a-5407d9a3fea1-workload-certs\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:34:31.460686 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:34:31.460611 5024 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c49c4c7b-3182-4699-898a-5407d9a3fea1-workload-socket\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:34:31.460916 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:34:31.460850 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c49c4c7b-3182-4699-898a-5407d9a3fea1-credential-socket" pod "c49c4c7b-3182-4699-898a-5407d9a3fea1" (UID: "c49c4c7b-3182-4699-898a-5407d9a3fea1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:34:31.461648 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:34:31.461138 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c49c4c7b-3182-4699-898a-5407d9a3fea1-istiod-ca-cert" pod "c49c4c7b-3182-4699-898a-5407d9a3fea1" (UID: "c49c4c7b-3182-4699-898a-5407d9a3fea1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:34:31.461648 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:34:31.461277 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c49c4c7b-3182-4699-898a-5407d9a3fea1-kube-api-access-7fp4d" pod "c49c4c7b-3182-4699-898a-5407d9a3fea1" (UID: "c49c4c7b-3182-4699-898a-5407d9a3fea1"). InnerVolumeSpecName "kube-api-access-7fp4d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:34:31.461648 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:34:31.461570 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c49c4c7b-3182-4699-898a-5407d9a3fea1-istio-token" pod "c49c4c7b-3182-4699-898a-5407d9a3fea1" (UID: "c49c4c7b-3182-4699-898a-5407d9a3fea1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:34:31.462061 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:34:31.462035 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c49c4c7b-3182-4699-898a-5407d9a3fea1-istio-podinfo" pod "c49c4c7b-3182-4699-898a-5407d9a3fea1" (UID: "c49c4c7b-3182-4699-898a-5407d9a3fea1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 13:34:31.463048 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:34:31.463024 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c49c4c7b-3182-4699-898a-5407d9a3fea1-istio-envoy" pod "c49c4c7b-3182-4699-898a-5407d9a3fea1" (UID: "c49c4c7b-3182-4699-898a-5407d9a3fea1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:34:31.568354 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:34:31.568076 5024 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c49c4c7b-3182-4699-898a-5407d9a3fea1-istio-envoy\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:34:31.568354 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:34:31.568110 5024 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c49c4c7b-3182-4699-898a-5407d9a3fea1-istio-token\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:34:31.568354 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:34:31.568125 5024 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c49c4c7b-3182-4699-898a-5407d9a3fea1-istio-podinfo\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:34:31.568354 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:34:31.568140 5024 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c49c4c7b-3182-4699-898a-5407d9a3fea1-istiod-ca-cert\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:34:31.568354 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:34:31.568153 5024 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c49c4c7b-3182-4699-898a-5407d9a3fea1-credential-socket\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:34:31.568354 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:34:31.568218 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7fp4d\" (UniqueName: \"kubernetes.io/projected/c49c4c7b-3182-4699-898a-5407d9a3fea1-kube-api-access-7fp4d\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:34:32.336832 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:34:32.336797 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-wkz2r" Nov 03 13:34:32.337347 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:34:32.336805 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-wkz2r" event={"ID":"c49c4c7b-3182-4699-898a-5407d9a3fea1","Type":"ContainerDied","Data":"586a327000d96a7254f0e5b00388bba99719696e25e0eb79f20ee0a410ac6575"} Nov 03 13:34:32.337347 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:34:32.336968 5024 scope.go:122] "RemoveContainer" containerID="a83ddae672e505fbb2fd4641a2fc8481b8c2a939713bcb369cde5647ec1eaae5" Nov 03 13:34:32.363263 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:34:32.363229 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-wkz2r"] Nov 03 13:34:32.366924 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:34:32.366896 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-wkz2r"] Nov 03 13:34:34.146229 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:34:34.146185 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c49c4c7b-3182-4699-898a-5407d9a3fea1" path="/var/lib/kubelet/pods/c49c4c7b-3182-4699-898a-5407d9a3fea1/volumes" Nov 03 13:34:39.513406 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:34:39.513309 5024 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d7mwz" containerName="extractor" Nov 03 13:34:48.504581 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:34:48.504536 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-pznlh"] Nov 03 13:34:48.505658 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:34:48.505630 5024 cpu_manager.go:414] "containerMap: removing container" podUID="c49c4c7b-3182-4699-898a-5407d9a3fea1" containerName="istio-proxy" Nov 03 13:34:48.505658 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:34:48.505656 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c49c4c7b-3182-4699-898a-5407d9a3fea1" containerName="istio-proxy" Nov 03 13:34:48.505897 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:34:48.505883 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="c49c4c7b-3182-4699-898a-5407d9a3fea1" containerName="istio-proxy" Nov 03 13:34:48.521906 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:34:48.521860 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-pznlh" Nov 03 13:34:48.530513 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:34:48.530474 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-pznlh"] Nov 03 13:34:48.533257 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:34:48.532305 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-2b8a08-openshift-default-dockercfg-dffsv\"" Nov 03 13:34:48.600371 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:34:48.600330 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0757da43-44aa-48e1-b169-00715a78ea02-istio-token\") pod \"e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-pznlh\" (UID: \"0757da43-44aa-48e1-b169-00715a78ea02\") " pod="openshift-ingress/e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-pznlh" Nov 03 13:34:48.600600 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:34:48.600384 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zw559\" (UniqueName: \"kubernetes.io/projected/0757da43-44aa-48e1-b169-00715a78ea02-kube-api-access-zw559\") pod \"e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-pznlh\" (UID: \"0757da43-44aa-48e1-b169-00715a78ea02\") " pod="openshift-ingress/e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-pznlh" Nov 03 13:34:48.600600 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:34:48.600414 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0757da43-44aa-48e1-b169-00715a78ea02-workload-socket\") pod \"e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-pznlh\" (UID: \"0757da43-44aa-48e1-b169-00715a78ea02\") " pod="openshift-ingress/e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-pznlh" Nov 03 13:34:48.600600 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:34:48.600438 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0757da43-44aa-48e1-b169-00715a78ea02-istio-data\") pod \"e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-pznlh\" (UID: \"0757da43-44aa-48e1-b169-00715a78ea02\") " pod="openshift-ingress/e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-pznlh" Nov 03 13:34:48.600600 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:34:48.600473 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0757da43-44aa-48e1-b169-00715a78ea02-istio-podinfo\") pod \"e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-pznlh\" (UID: \"0757da43-44aa-48e1-b169-00715a78ea02\") " pod="openshift-ingress/e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-pznlh" Nov 03 13:34:48.600843 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:34:48.600663 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0757da43-44aa-48e1-b169-00715a78ea02-istio-envoy\") pod \"e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-pznlh\" (UID: \"0757da43-44aa-48e1-b169-00715a78ea02\") " pod="openshift-ingress/e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-pznlh" Nov 03 13:34:48.600843 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:34:48.600749 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0757da43-44aa-48e1-b169-00715a78ea02-credential-socket\") pod \"e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-pznlh\" (UID: \"0757da43-44aa-48e1-b169-00715a78ea02\") " pod="openshift-ingress/e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-pznlh" Nov 03 13:34:48.600843 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:34:48.600794 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0757da43-44aa-48e1-b169-00715a78ea02-workload-certs\") pod \"e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-pznlh\" (UID: \"0757da43-44aa-48e1-b169-00715a78ea02\") " pod="openshift-ingress/e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-pznlh" Nov 03 13:34:48.601016 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:34:48.600883 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0757da43-44aa-48e1-b169-00715a78ea02-istiod-ca-cert\") pod \"e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-pznlh\" (UID: \"0757da43-44aa-48e1-b169-00715a78ea02\") " pod="openshift-ingress/e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-pznlh" Nov 03 13:34:48.715314 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:34:48.712774 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0757da43-44aa-48e1-b169-00715a78ea02-istio-token\") pod \"e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-pznlh\" (UID: \"0757da43-44aa-48e1-b169-00715a78ea02\") " pod="openshift-ingress/e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-pznlh" Nov 03 13:34:48.715314 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:34:48.712846 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zw559\" (UniqueName: \"kubernetes.io/projected/0757da43-44aa-48e1-b169-00715a78ea02-kube-api-access-zw559\") pod \"e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-pznlh\" (UID: \"0757da43-44aa-48e1-b169-00715a78ea02\") " pod="openshift-ingress/e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-pznlh" Nov 03 13:34:48.715314 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:34:48.712875 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0757da43-44aa-48e1-b169-00715a78ea02-workload-socket\") pod \"e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-pznlh\" (UID: \"0757da43-44aa-48e1-b169-00715a78ea02\") " pod="openshift-ingress/e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-pznlh" Nov 03 13:34:48.715314 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:34:48.712921 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0757da43-44aa-48e1-b169-00715a78ea02-istio-data\") pod \"e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-pznlh\" (UID: \"0757da43-44aa-48e1-b169-00715a78ea02\") " pod="openshift-ingress/e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-pznlh" Nov 03 13:34:48.715314 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:34:48.712967 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0757da43-44aa-48e1-b169-00715a78ea02-istio-podinfo\") pod \"e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-pznlh\" (UID: \"0757da43-44aa-48e1-b169-00715a78ea02\") " pod="openshift-ingress/e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-pznlh" Nov 03 13:34:48.715314 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:34:48.713302 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0757da43-44aa-48e1-b169-00715a78ea02-istio-envoy\") pod \"e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-pznlh\" (UID: \"0757da43-44aa-48e1-b169-00715a78ea02\") " pod="openshift-ingress/e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-pznlh" Nov 03 13:34:48.715314 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:34:48.713395 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0757da43-44aa-48e1-b169-00715a78ea02-credential-socket\") pod \"e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-pznlh\" (UID: \"0757da43-44aa-48e1-b169-00715a78ea02\") " pod="openshift-ingress/e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-pznlh" Nov 03 13:34:48.715314 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:34:48.713439 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0757da43-44aa-48e1-b169-00715a78ea02-workload-certs\") pod \"e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-pznlh\" (UID: \"0757da43-44aa-48e1-b169-00715a78ea02\") " pod="openshift-ingress/e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-pznlh" Nov 03 13:34:48.715314 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:34:48.713552 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0757da43-44aa-48e1-b169-00715a78ea02-istiod-ca-cert\") pod \"e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-pznlh\" (UID: \"0757da43-44aa-48e1-b169-00715a78ea02\") " pod="openshift-ingress/e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-pznlh" Nov 03 13:34:48.715314 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:34:48.714779 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0757da43-44aa-48e1-b169-00715a78ea02-istiod-ca-cert\") pod \"e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-pznlh\" (UID: \"0757da43-44aa-48e1-b169-00715a78ea02\") " pod="openshift-ingress/e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-pznlh" Nov 03 13:34:48.716178 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:34:48.715718 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0757da43-44aa-48e1-b169-00715a78ea02-workload-socket\") pod \"e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-pznlh\" (UID: \"0757da43-44aa-48e1-b169-00715a78ea02\") " pod="openshift-ingress/e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-pznlh" Nov 03 13:34:48.716178 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:34:48.715969 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0757da43-44aa-48e1-b169-00715a78ea02-istio-data\") pod \"e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-pznlh\" (UID: \"0757da43-44aa-48e1-b169-00715a78ea02\") " pod="openshift-ingress/e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-pznlh" Nov 03 13:34:48.749024 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:34:48.748659 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0757da43-44aa-48e1-b169-00715a78ea02-workload-certs\") pod \"e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-pznlh\" (UID: \"0757da43-44aa-48e1-b169-00715a78ea02\") " pod="openshift-ingress/e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-pznlh" Nov 03 13:34:48.767495 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:34:48.760304 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0757da43-44aa-48e1-b169-00715a78ea02-credential-socket\") pod \"e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-pznlh\" (UID: \"0757da43-44aa-48e1-b169-00715a78ea02\") " pod="openshift-ingress/e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-pznlh" Nov 03 13:34:48.785804 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:34:48.785725 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0757da43-44aa-48e1-b169-00715a78ea02-istio-token\") pod \"e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-pznlh\" (UID: \"0757da43-44aa-48e1-b169-00715a78ea02\") " pod="openshift-ingress/e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-pznlh" Nov 03 13:34:48.786883 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:34:48.786764 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zw559\" (UniqueName: \"kubernetes.io/projected/0757da43-44aa-48e1-b169-00715a78ea02-kube-api-access-zw559\") pod \"e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-pznlh\" (UID: \"0757da43-44aa-48e1-b169-00715a78ea02\") " pod="openshift-ingress/e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-pznlh" Nov 03 13:34:48.796318 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:34:48.788674 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0757da43-44aa-48e1-b169-00715a78ea02-istio-envoy\") pod \"e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-pznlh\" (UID: \"0757da43-44aa-48e1-b169-00715a78ea02\") " pod="openshift-ingress/e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-pznlh" Nov 03 13:34:48.813975 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:34:48.807967 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0757da43-44aa-48e1-b169-00715a78ea02-istio-podinfo\") pod \"e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-pznlh\" (UID: \"0757da43-44aa-48e1-b169-00715a78ea02\") " pod="openshift-ingress/e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-pznlh" Nov 03 13:34:48.852771 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:34:48.852737 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-pznlh" Nov 03 13:34:49.348676 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:34:49.348643 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0757da43_44aa_48e1_b169_00715a78ea02.slice/crio-eaf6a2b02c4aecdcd60875cd2f4bcd050383f86c3d3cf151196843ef934cc94a WatchSource:0}: Error finding container eaf6a2b02c4aecdcd60875cd2f4bcd050383f86c3d3cf151196843ef934cc94a: Status 404 returned error can't find the container with id eaf6a2b02c4aecdcd60875cd2f4bcd050383f86c3d3cf151196843ef934cc94a Nov 03 13:34:49.349835 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:34:49.349809 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-pznlh"] Nov 03 13:34:49.352152 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:34:49.352105 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:34:49.352248 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:34:49.352192 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:34:49.352248 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:34:49.352239 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:34:49.982844 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:34:49.982800 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-pznlh" event={"ID":"0757da43-44aa-48e1-b169-00715a78ea02","Type":"ContainerStarted","Data":"33974b23654fbff102111f8a70f02b8e2a0ae24b9e456b780a29ae4c7684b063"} Nov 03 13:34:49.983345 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:34:49.982864 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-pznlh" event={"ID":"0757da43-44aa-48e1-b169-00715a78ea02","Type":"ContainerStarted","Data":"eaf6a2b02c4aecdcd60875cd2f4bcd050383f86c3d3cf151196843ef934cc94a"} Nov 03 13:34:50.009330 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:34:50.009242 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-pznlh" podStartSLOduration=2.009202557 podStartE2EDuration="2.009202557s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:34:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:34:50.008074155 +0000 UTC m=+1570.016265439" watchObservedRunningTime="2026-11-03 13:34:50.009202557 +0000 UTC m=+1570.017393833" Nov 03 13:34:50.855308 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:34:50.854650 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-pznlh" Nov 03 13:34:50.863370 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:34:50.863335 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-pznlh" Nov 03 13:34:50.994733 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:34:50.993878 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-pznlh" Nov 03 13:34:50.995722 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:34:50.995693 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-pznlh" Nov 03 13:34:59.273599 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:34:59.273555 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-6z9tv"] Nov 03 13:34:59.274483 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:34:59.274438 5024 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-6z9tv" podUID="7d77cd17-8ce7-4fe7-9b61-7e4c7cdc720b" containerName="istio-proxy" containerID="cri-o://80d72223e1f18c419b2b9bde76343f979e9fad8deaab42e8d5bbca63c6a5f150" gracePeriod=30 Nov 03 13:35:02.786401 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:02.786351 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-69d95d887d-xsbb2"] Nov 03 13:35:02.788257 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:02.788216 5024 cpu_manager.go:414] "containerMap: removing container" podUID="c49c4c7b-3182-4699-898a-5407d9a3fea1" containerName="istio-proxy" Nov 03 13:35:02.788454 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:02.788437 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c49c4c7b-3182-4699-898a-5407d9a3fea1" containerName="istio-proxy" Nov 03 13:35:02.793581 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:02.793551 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="c49c4c7b-3182-4699-898a-5407d9a3fea1" containerName="istio-proxy" Nov 03 13:35:02.812805 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:02.812771 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-69d95d887d-xsbb2" Nov 03 13:35:02.857551 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:02.857516 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-69d95d887d-xsbb2"] Nov 03 13:35:02.878012 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:02.877971 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d6860fac-adfc-4c91-aa20-522cf4b68a2c-routing-overlay-volume\") pod \"payload-processing-external-model-69d95d887d-xsbb2\" (UID: \"d6860fac-adfc-4c91-aa20-522cf4b68a2c\") " pod="models-as-a-service/payload-processing-external-model-69d95d887d-xsbb2" Nov 03 13:35:02.880303 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:02.880247 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6860fac-adfc-4c91-aa20-522cf4b68a2c-plugins-config-volume\") pod \"payload-processing-external-model-69d95d887d-xsbb2\" (UID: \"d6860fac-adfc-4c91-aa20-522cf4b68a2c\") " pod="models-as-a-service/payload-processing-external-model-69d95d887d-xsbb2" Nov 03 13:35:02.880637 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:02.880606 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cfkng\" (UniqueName: \"kubernetes.io/projected/d6860fac-adfc-4c91-aa20-522cf4b68a2c-kube-api-access-cfkng\") pod \"payload-processing-external-model-69d95d887d-xsbb2\" (UID: \"d6860fac-adfc-4c91-aa20-522cf4b68a2c\") " pod="models-as-a-service/payload-processing-external-model-69d95d887d-xsbb2" Nov 03 13:35:02.880920 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:02.880861 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d6860fac-adfc-4c91-aa20-522cf4b68a2c-provider-credentials\") pod \"payload-processing-external-model-69d95d887d-xsbb2\" (UID: \"d6860fac-adfc-4c91-aa20-522cf4b68a2c\") " pod="models-as-a-service/payload-processing-external-model-69d95d887d-xsbb2" Nov 03 13:35:02.992775 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:02.992732 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d6860fac-adfc-4c91-aa20-522cf4b68a2c-routing-overlay-volume\") pod \"payload-processing-external-model-69d95d887d-xsbb2\" (UID: \"d6860fac-adfc-4c91-aa20-522cf4b68a2c\") " pod="models-as-a-service/payload-processing-external-model-69d95d887d-xsbb2" Nov 03 13:35:03.006087 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:02.995825 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d6860fac-adfc-4c91-aa20-522cf4b68a2c-routing-overlay-volume\") pod \"payload-processing-external-model-69d95d887d-xsbb2\" (UID: \"d6860fac-adfc-4c91-aa20-522cf4b68a2c\") " pod="models-as-a-service/payload-processing-external-model-69d95d887d-xsbb2" Nov 03 13:35:03.006610 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:03.006584 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6860fac-adfc-4c91-aa20-522cf4b68a2c-plugins-config-volume\") pod \"payload-processing-external-model-69d95d887d-xsbb2\" (UID: \"d6860fac-adfc-4c91-aa20-522cf4b68a2c\") " pod="models-as-a-service/payload-processing-external-model-69d95d887d-xsbb2" Nov 03 13:35:03.006795 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:03.006779 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfkng\" (UniqueName: \"kubernetes.io/projected/d6860fac-adfc-4c91-aa20-522cf4b68a2c-kube-api-access-cfkng\") pod \"payload-processing-external-model-69d95d887d-xsbb2\" (UID: \"d6860fac-adfc-4c91-aa20-522cf4b68a2c\") " pod="models-as-a-service/payload-processing-external-model-69d95d887d-xsbb2" Nov 03 13:35:03.007110 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:03.007087 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d6860fac-adfc-4c91-aa20-522cf4b68a2c-provider-credentials\") pod \"payload-processing-external-model-69d95d887d-xsbb2\" (UID: \"d6860fac-adfc-4c91-aa20-522cf4b68a2c\") " pod="models-as-a-service/payload-processing-external-model-69d95d887d-xsbb2" Nov 03 13:35:03.009628 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:03.009587 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6860fac-adfc-4c91-aa20-522cf4b68a2c-plugins-config-volume\") pod \"payload-processing-external-model-69d95d887d-xsbb2\" (UID: \"d6860fac-adfc-4c91-aa20-522cf4b68a2c\") " pod="models-as-a-service/payload-processing-external-model-69d95d887d-xsbb2" Nov 03 13:35:03.028251 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:03.028114 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cfkng\" (UniqueName: \"kubernetes.io/projected/d6860fac-adfc-4c91-aa20-522cf4b68a2c-kube-api-access-cfkng\") pod \"payload-processing-external-model-69d95d887d-xsbb2\" (UID: \"d6860fac-adfc-4c91-aa20-522cf4b68a2c\") " pod="models-as-a-service/payload-processing-external-model-69d95d887d-xsbb2" Nov 03 13:35:03.030902 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:03.030871 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d6860fac-adfc-4c91-aa20-522cf4b68a2c-provider-credentials\") pod \"payload-processing-external-model-69d95d887d-xsbb2\" (UID: \"d6860fac-adfc-4c91-aa20-522cf4b68a2c\") " pod="models-as-a-service/payload-processing-external-model-69d95d887d-xsbb2" Nov 03 13:35:03.158696 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:03.158563 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-69d95d887d-xsbb2" Nov 03 13:35:03.868472 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:35:03.868428 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6860fac_adfc_4c91_aa20_522cf4b68a2c.slice/crio-67c145d97fd7f1dea22cbcac403c524783e6c10fd0fa6526efba95836cb73bc7 WatchSource:0}: Error finding container 67c145d97fd7f1dea22cbcac403c524783e6c10fd0fa6526efba95836cb73bc7: Status 404 returned error can't find the container with id 67c145d97fd7f1dea22cbcac403c524783e6c10fd0fa6526efba95836cb73bc7 Nov 03 13:35:03.868898 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:03.868500 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-69d95d887d-xsbb2"] Nov 03 13:35:04.213260 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:35:04.213176 5024 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-c398a670ca11: secret "agc-cred-c398a670ca11" not found Nov 03 13:35:04.213260 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:35:04.213207 5024 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-e7798d3231e5: secret "agc-cred-e7798d3231e5" not found Nov 03 13:35:04.213260 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:35:04.213220 5024 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-5546676475-98nhd: [secret "agc-cred-c398a670ca11" not found, secret "agc-cred-e7798d3231e5" not found] Nov 03 13:35:04.213648 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:35:04.213278 5024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/49f61653-f10c-4692-bc8f-a379cfd3b073-provider-credentials podName:49f61653-f10c-4692-bc8f-a379cfd3b073 nodeName:}" failed. No retries permitted until 2026-11-03 13:35:04.713257197 +0000 UTC m=+1584.721448467 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/49f61653-f10c-4692-bc8f-a379cfd3b073-provider-credentials") pod "payload-processing-external-model-5546676475-98nhd" (UID: "49f61653-f10c-4692-bc8f-a379cfd3b073") : [secret "agc-cred-c398a670ca11" not found, secret "agc-cred-e7798d3231e5" not found] Nov 03 13:35:04.454381 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:04.448220 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-69d95d887d-xsbb2" event={"ID":"d6860fac-adfc-4c91-aa20-522cf4b68a2c","Type":"ContainerStarted","Data":"1f230d39621a9a90a5139c46988eb6bd045018b0736c200295785263783bc80c"} Nov 03 13:35:04.454381 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:04.448314 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-69d95d887d-xsbb2" event={"ID":"d6860fac-adfc-4c91-aa20-522cf4b68a2c","Type":"ContainerStarted","Data":"67c145d97fd7f1dea22cbcac403c524783e6c10fd0fa6526efba95836cb73bc7"} Nov 03 13:35:04.454381 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:04.450449 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-69d95d887d-xsbb2" Nov 03 13:35:04.500360 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:04.496192 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-69d95d887d-xsbb2" podStartSLOduration=2.496169452 podStartE2EDuration="2.496169452s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:35:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:35:04.493729042 +0000 UTC m=+1584.501920320" watchObservedRunningTime="2026-11-03 13:35:04.496169452 +0000 UTC m=+1584.504360730" Nov 03 13:35:05.035314 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:35:05.011491 5024 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-c398a670ca11: secret "agc-cred-c398a670ca11" not found Nov 03 13:35:05.036131 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:35:05.036105 5024 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-e7798d3231e5: secret "agc-cred-e7798d3231e5" not found Nov 03 13:35:05.036270 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:35:05.036258 5024 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-5546676475-98nhd: [secret "agc-cred-c398a670ca11" not found, secret "agc-cred-e7798d3231e5" not found] Nov 03 13:35:05.036492 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:35:05.036474 5024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/49f61653-f10c-4692-bc8f-a379cfd3b073-provider-credentials podName:49f61653-f10c-4692-bc8f-a379cfd3b073 nodeName:}" failed. No retries permitted until 2026-11-03 13:35:06.036438042 +0000 UTC m=+1586.044629299 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/49f61653-f10c-4692-bc8f-a379cfd3b073-provider-credentials") pod "payload-processing-external-model-5546676475-98nhd" (UID: "49f61653-f10c-4692-bc8f-a379cfd3b073") : [secret "agc-cred-c398a670ca11" not found, secret "agc-cred-e7798d3231e5" not found] Nov 03 13:35:05.258310 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:05.257982 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-6z9tv" Nov 03 13:35:05.282309 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:05.277570 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d77cd17-8ce7-4fe7-9b61-7e4c7cdc720b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d77cd17-8ce7-4fe7-9b61-7e4c7cdc720b-istio-envoy\") pod \"7d77cd17-8ce7-4fe7-9b61-7e4c7cdc720b\" (UID: \"7d77cd17-8ce7-4fe7-9b61-7e4c7cdc720b\") " Nov 03 13:35:05.282309 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:05.277631 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d77cd17-8ce7-4fe7-9b61-7e4c7cdc720b-kube-api-access-s9vt9\" (UniqueName: \"kubernetes.io/projected/7d77cd17-8ce7-4fe7-9b61-7e4c7cdc720b-kube-api-access-s9vt9\") pod \"7d77cd17-8ce7-4fe7-9b61-7e4c7cdc720b\" (UID: \"7d77cd17-8ce7-4fe7-9b61-7e4c7cdc720b\") " Nov 03 13:35:05.282309 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:05.277705 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7d77cd17-8ce7-4fe7-9b61-7e4c7cdc720b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d77cd17-8ce7-4fe7-9b61-7e4c7cdc720b-istio-podinfo\") pod \"7d77cd17-8ce7-4fe7-9b61-7e4c7cdc720b\" (UID: \"7d77cd17-8ce7-4fe7-9b61-7e4c7cdc720b\") " Nov 03 13:35:05.282309 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:05.277784 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d77cd17-8ce7-4fe7-9b61-7e4c7cdc720b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d77cd17-8ce7-4fe7-9b61-7e4c7cdc720b-credential-socket\") pod \"7d77cd17-8ce7-4fe7-9b61-7e4c7cdc720b\" (UID: \"7d77cd17-8ce7-4fe7-9b61-7e4c7cdc720b\") " Nov 03 13:35:05.282309 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:05.277897 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d77cd17-8ce7-4fe7-9b61-7e4c7cdc720b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d77cd17-8ce7-4fe7-9b61-7e4c7cdc720b-workload-socket\") pod \"7d77cd17-8ce7-4fe7-9b61-7e4c7cdc720b\" (UID: \"7d77cd17-8ce7-4fe7-9b61-7e4c7cdc720b\") " Nov 03 13:35:05.282309 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:05.277939 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d77cd17-8ce7-4fe7-9b61-7e4c7cdc720b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d77cd17-8ce7-4fe7-9b61-7e4c7cdc720b-workload-certs\") pod \"7d77cd17-8ce7-4fe7-9b61-7e4c7cdc720b\" (UID: \"7d77cd17-8ce7-4fe7-9b61-7e4c7cdc720b\") " Nov 03 13:35:05.282309 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:05.278070 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d77cd17-8ce7-4fe7-9b61-7e4c7cdc720b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d77cd17-8ce7-4fe7-9b61-7e4c7cdc720b-istio-data\") pod \"7d77cd17-8ce7-4fe7-9b61-7e4c7cdc720b\" (UID: \"7d77cd17-8ce7-4fe7-9b61-7e4c7cdc720b\") " Nov 03 13:35:05.282309 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:05.278202 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7d77cd17-8ce7-4fe7-9b61-7e4c7cdc720b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d77cd17-8ce7-4fe7-9b61-7e4c7cdc720b-istiod-ca-cert\") pod \"7d77cd17-8ce7-4fe7-9b61-7e4c7cdc720b\" (UID: \"7d77cd17-8ce7-4fe7-9b61-7e4c7cdc720b\") " Nov 03 13:35:05.282309 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:05.278310 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d77cd17-8ce7-4fe7-9b61-7e4c7cdc720b-istio-token\" (UniqueName: \"kubernetes.io/projected/7d77cd17-8ce7-4fe7-9b61-7e4c7cdc720b-istio-token\") pod \"7d77cd17-8ce7-4fe7-9b61-7e4c7cdc720b\" (UID: \"7d77cd17-8ce7-4fe7-9b61-7e4c7cdc720b\") " Nov 03 13:35:05.339356 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:05.286961 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d77cd17-8ce7-4fe7-9b61-7e4c7cdc720b-workload-certs" pod "7d77cd17-8ce7-4fe7-9b61-7e4c7cdc720b" (UID: "7d77cd17-8ce7-4fe7-9b61-7e4c7cdc720b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:35:05.339356 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:05.287215 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d77cd17-8ce7-4fe7-9b61-7e4c7cdc720b-workload-socket" pod "7d77cd17-8ce7-4fe7-9b61-7e4c7cdc720b" (UID: "7d77cd17-8ce7-4fe7-9b61-7e4c7cdc720b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:35:05.339356 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:05.295485 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d77cd17-8ce7-4fe7-9b61-7e4c7cdc720b-credential-socket" pod "7d77cd17-8ce7-4fe7-9b61-7e4c7cdc720b" (UID: "7d77cd17-8ce7-4fe7-9b61-7e4c7cdc720b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:35:05.339356 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:05.296178 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d77cd17-8ce7-4fe7-9b61-7e4c7cdc720b-istiod-ca-cert" pod "7d77cd17-8ce7-4fe7-9b61-7e4c7cdc720b" (UID: "7d77cd17-8ce7-4fe7-9b61-7e4c7cdc720b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:35:05.339356 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:05.296376 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d77cd17-8ce7-4fe7-9b61-7e4c7cdc720b-istio-data" pod "7d77cd17-8ce7-4fe7-9b61-7e4c7cdc720b" (UID: "7d77cd17-8ce7-4fe7-9b61-7e4c7cdc720b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:35:05.339356 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:05.318746 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7d77cd17-8ce7-4fe7-9b61-7e4c7cdc720b-istio-podinfo" pod "7d77cd17-8ce7-4fe7-9b61-7e4c7cdc720b" (UID: "7d77cd17-8ce7-4fe7-9b61-7e4c7cdc720b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 13:35:05.340825 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:05.340772 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d77cd17-8ce7-4fe7-9b61-7e4c7cdc720b-istio-token" pod "7d77cd17-8ce7-4fe7-9b61-7e4c7cdc720b" (UID: "7d77cd17-8ce7-4fe7-9b61-7e4c7cdc720b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:35:05.388520 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:05.382483 5024 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d77cd17-8ce7-4fe7-9b61-7e4c7cdc720b-istio-data\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:35:05.388520 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:05.382515 5024 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d77cd17-8ce7-4fe7-9b61-7e4c7cdc720b-istiod-ca-cert\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:35:05.388520 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:05.382534 5024 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d77cd17-8ce7-4fe7-9b61-7e4c7cdc720b-istio-token\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:35:05.388520 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:05.382550 5024 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d77cd17-8ce7-4fe7-9b61-7e4c7cdc720b-istio-podinfo\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:35:05.388520 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:05.382563 5024 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d77cd17-8ce7-4fe7-9b61-7e4c7cdc720b-credential-socket\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:35:05.388520 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:05.382577 5024 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d77cd17-8ce7-4fe7-9b61-7e4c7cdc720b-workload-socket\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:35:05.388520 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:05.382590 5024 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d77cd17-8ce7-4fe7-9b61-7e4c7cdc720b-workload-certs\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:35:05.400342 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:05.389482 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d77cd17-8ce7-4fe7-9b61-7e4c7cdc720b-kube-api-access-s9vt9" pod "7d77cd17-8ce7-4fe7-9b61-7e4c7cdc720b" (UID: "7d77cd17-8ce7-4fe7-9b61-7e4c7cdc720b"). InnerVolumeSpecName "kube-api-access-s9vt9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:35:05.453335 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:05.452106 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d77cd17-8ce7-4fe7-9b61-7e4c7cdc720b-istio-envoy" pod "7d77cd17-8ce7-4fe7-9b61-7e4c7cdc720b" (UID: "7d77cd17-8ce7-4fe7-9b61-7e4c7cdc720b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:35:05.560151 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:05.560108 5024 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d77cd17-8ce7-4fe7-9b61-7e4c7cdc720b-istio-envoy\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:35:05.706376 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:05.673400 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s9vt9\" (UniqueName: \"kubernetes.io/projected/7d77cd17-8ce7-4fe7-9b61-7e4c7cdc720b-kube-api-access-s9vt9\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:35:05.706376 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:05.670352 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-6z9tv" event={"ID":"7d77cd17-8ce7-4fe7-9b61-7e4c7cdc720b","Type":"ContainerDied","Data":"80d72223e1f18c419b2b9bde76343f979e9fad8deaab42e8d5bbca63c6a5f150"} Nov 03 13:35:05.706376 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:05.696423 5024 scope.go:122] "RemoveContainer" containerID="80d72223e1f18c419b2b9bde76343f979e9fad8deaab42e8d5bbca63c6a5f150" Nov 03 13:35:05.706376 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:05.670425 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-6z9tv" Nov 03 13:35:05.706376 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:05.670318 5024 generic.go:356] "Generic (PLEG): container finished" podID="7d77cd17-8ce7-4fe7-9b61-7e4c7cdc720b" containerID="80d72223e1f18c419b2b9bde76343f979e9fad8deaab42e8d5bbca63c6a5f150" exitCode=0 Nov 03 13:35:05.781364 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:05.781324 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-6z9tv" event={"ID":"7d77cd17-8ce7-4fe7-9b61-7e4c7cdc720b","Type":"ContainerDied","Data":"d4ee62a1b39081f06a523f86ba39c8fe9bce3b81b579f7ff80e4feb80c9d07e5"} Nov 03 13:35:05.950019 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:05.949994 5024 scope.go:122] "RemoveContainer" containerID="d7b5b985f79627cac6c18273d4df80d60d9c163fd8078d539b655cd26becda5a" Nov 03 13:35:05.960666 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:05.960644 5024 scope.go:122] "RemoveContainer" containerID="80d72223e1f18c419b2b9bde76343f979e9fad8deaab42e8d5bbca63c6a5f150" Nov 03 13:35:05.960962 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:35:05.960940 5024 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"80d72223e1f18c419b2b9bde76343f979e9fad8deaab42e8d5bbca63c6a5f150\": container with ID starting with 80d72223e1f18c419b2b9bde76343f979e9fad8deaab42e8d5bbca63c6a5f150 not found: ID does not exist" containerID="80d72223e1f18c419b2b9bde76343f979e9fad8deaab42e8d5bbca63c6a5f150" Nov 03 13:35:05.960999 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:05.960973 5024 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80d72223e1f18c419b2b9bde76343f979e9fad8deaab42e8d5bbca63c6a5f150"} err="failed to get container status \"80d72223e1f18c419b2b9bde76343f979e9fad8deaab42e8d5bbca63c6a5f150\": rpc error: code = NotFound desc = could not find container \"80d72223e1f18c419b2b9bde76343f979e9fad8deaab42e8d5bbca63c6a5f150\": container with ID starting with 80d72223e1f18c419b2b9bde76343f979e9fad8deaab42e8d5bbca63c6a5f150 not found: ID does not exist" Nov 03 13:35:05.960999 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:05.960993 5024 scope.go:122] "RemoveContainer" containerID="d7b5b985f79627cac6c18273d4df80d60d9c163fd8078d539b655cd26becda5a" Nov 03 13:35:05.961239 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:35:05.961223 5024 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d7b5b985f79627cac6c18273d4df80d60d9c163fd8078d539b655cd26becda5a\": container with ID starting with d7b5b985f79627cac6c18273d4df80d60d9c163fd8078d539b655cd26becda5a not found: ID does not exist" containerID="d7b5b985f79627cac6c18273d4df80d60d9c163fd8078d539b655cd26becda5a" Nov 03 13:35:05.961292 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:05.961243 5024 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7b5b985f79627cac6c18273d4df80d60d9c163fd8078d539b655cd26becda5a"} err="failed to get container status \"d7b5b985f79627cac6c18273d4df80d60d9c163fd8078d539b655cd26becda5a\": rpc error: code = NotFound desc = could not find container \"d7b5b985f79627cac6c18273d4df80d60d9c163fd8078d539b655cd26becda5a\": container with ID starting with d7b5b985f79627cac6c18273d4df80d60d9c163fd8078d539b655cd26becda5a not found: ID does not exist" Nov 03 13:35:06.009829 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:06.009793 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-6z9tv"] Nov 03 13:35:06.013584 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:06.013547 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-6z9tv"] Nov 03 13:35:06.111115 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:35:06.111084 5024 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-c398a670ca11: secret "agc-cred-c398a670ca11" not found Nov 03 13:35:06.111638 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:35:06.111621 5024 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-e7798d3231e5: secret "agc-cred-e7798d3231e5" not found Nov 03 13:35:06.111708 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:35:06.111700 5024 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-5546676475-98nhd: [secret "agc-cred-c398a670ca11" not found, secret "agc-cred-e7798d3231e5" not found] Nov 03 13:35:06.111850 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:35:06.111837 5024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/49f61653-f10c-4692-bc8f-a379cfd3b073-provider-credentials podName:49f61653-f10c-4692-bc8f-a379cfd3b073 nodeName:}" failed. No retries permitted until 2026-11-03 13:35:08.111789173 +0000 UTC m=+1588.119980444 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/49f61653-f10c-4692-bc8f-a379cfd3b073-provider-credentials") pod "payload-processing-external-model-5546676475-98nhd" (UID: "49f61653-f10c-4692-bc8f-a379cfd3b073") : [secret "agc-cred-c398a670ca11" not found, secret "agc-cred-e7798d3231e5" not found] Nov 03 13:35:06.117542 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:06.117515 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d77cd17-8ce7-4fe7-9b61-7e4c7cdc720b" path="/var/lib/kubelet/pods/7d77cd17-8ce7-4fe7-9b61-7e4c7cdc720b/volumes" Nov 03 13:35:08.269632 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:35:08.218922 5024 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-c398a670ca11: secret "agc-cred-c398a670ca11" not found Nov 03 13:35:08.269632 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:35:08.218964 5024 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-e7798d3231e5: secret "agc-cred-e7798d3231e5" not found Nov 03 13:35:08.269632 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:35:08.218976 5024 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-5546676475-98nhd: [secret "agc-cred-c398a670ca11" not found, secret "agc-cred-e7798d3231e5" not found] Nov 03 13:35:08.269632 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:35:08.219047 5024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/49f61653-f10c-4692-bc8f-a379cfd3b073-provider-credentials podName:49f61653-f10c-4692-bc8f-a379cfd3b073 nodeName:}" failed. No retries permitted until 2026-11-03 13:35:12.219028333 +0000 UTC m=+1592.227219587 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/49f61653-f10c-4692-bc8f-a379cfd3b073-provider-credentials") pod "payload-processing-external-model-5546676475-98nhd" (UID: "49f61653-f10c-4692-bc8f-a379cfd3b073") : [secret "agc-cred-c398a670ca11" not found, secret "agc-cred-e7798d3231e5" not found] Nov 03 13:35:09.522666 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:35:09.522613 5024 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d7mwz" containerName="extractor" Nov 03 13:35:12.031621 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:12.031550 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-86e0bec8-g62lg"] Nov 03 13:35:12.038265 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:12.038191 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-86e0bec8-g62lg"] Nov 03 13:35:12.117338 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:12.116697 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="928b4c26-66a3-4996-9b33-b9542a24d479" path="/var/lib/kubelet/pods/928b4c26-66a3-4996-9b33-b9542a24d479/volumes" Nov 03 13:35:12.273519 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:35:12.273470 5024 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-c398a670ca11: secret "agc-cred-c398a670ca11" not found Nov 03 13:35:12.273519 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:35:12.273514 5024 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-e7798d3231e5: secret "agc-cred-e7798d3231e5" not found Nov 03 13:35:12.273519 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:35:12.273527 5024 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-5546676475-98nhd: [secret "agc-cred-c398a670ca11" not found, secret "agc-cred-e7798d3231e5" not found] Nov 03 13:35:12.273794 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:35:12.273607 5024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/49f61653-f10c-4692-bc8f-a379cfd3b073-provider-credentials podName:49f61653-f10c-4692-bc8f-a379cfd3b073 nodeName:}" failed. No retries permitted until 2026-11-03 13:35:20.273585572 +0000 UTC m=+1600.281776844 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/49f61653-f10c-4692-bc8f-a379cfd3b073-provider-credentials") pod "payload-processing-external-model-5546676475-98nhd" (UID: "49f61653-f10c-4692-bc8f-a379cfd3b073") : [secret "agc-cred-c398a670ca11" not found, secret "agc-cred-e7798d3231e5" not found] Nov 03 13:35:12.877154 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:12.877119 5024 generic.go:356] "Generic (PLEG): container finished" podID="7d6b62e0-65bf-4766-93f4-b065c82304fb" containerID="3eed62c34bb403ff209a40eeeb098257d766a8681440034ee7d547e8b364ce98" exitCode=0 Nov 03 13:35:12.877461 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:12.877198 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-2cwvx" event={"ID":"7d6b62e0-65bf-4766-93f4-b065c82304fb","Type":"ContainerDied","Data":"3eed62c34bb403ff209a40eeeb098257d766a8681440034ee7d547e8b364ce98"} Nov 03 13:35:12.877834 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:12.877815 5024 scope.go:122] "RemoveContainer" containerID="3eed62c34bb403ff209a40eeeb098257d766a8681440034ee7d547e8b364ce98" Nov 03 13:35:12.879853 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:12.879810 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:35:12.879945 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:12.879911 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:35:12.879991 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:12.879962 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:35:13.429687 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:13.429652 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-1ec4b892-6b49f5f675-hxqrb"] Nov 03 13:35:13.430434 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:13.430417 5024 cpu_manager.go:414] "containerMap: removing container" podUID="7d77cd17-8ce7-4fe7-9b61-7e4c7cdc720b" containerName="istio-proxy" Nov 03 13:35:13.430496 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:13.430436 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d77cd17-8ce7-4fe7-9b61-7e4c7cdc720b" containerName="istio-proxy" Nov 03 13:35:13.430585 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:13.430575 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d77cd17-8ce7-4fe7-9b61-7e4c7cdc720b" containerName="istio-proxy" Nov 03 13:35:13.434493 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:13.434468 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-1ec4b892-6b49f5f675-hxqrb" Nov 03 13:35:13.448930 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:13.448895 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-1ec4b892\"" Nov 03 13:35:13.451684 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:13.450986 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-1ec4b892-dockercfg-qwwlg\"" Nov 03 13:35:13.469882 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:13.464656 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-1ec4b892-6b49f5f675-hxqrb"] Nov 03 13:35:13.519313 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:13.519253 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qlrb4\" (UniqueName: \"kubernetes.io/projected/47cf254c-610c-4b37-8bcd-dd4c11dcff35-kube-api-access-qlrb4\") pod \"payload-pre-processing-e2e-mt-1ec4b892-6b49f5f675-hxqrb\" (UID: \"47cf254c-610c-4b37-8bcd-dd4c11dcff35\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-1ec4b892-6b49f5f675-hxqrb" Nov 03 13:35:13.519551 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:13.519525 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47cf254c-610c-4b37-8bcd-dd4c11dcff35-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-1ec4b892-6b49f5f675-hxqrb\" (UID: \"47cf254c-610c-4b37-8bcd-dd4c11dcff35\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-1ec4b892-6b49f5f675-hxqrb" Nov 03 13:35:13.629311 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:13.626040 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qlrb4\" (UniqueName: \"kubernetes.io/projected/47cf254c-610c-4b37-8bcd-dd4c11dcff35-kube-api-access-qlrb4\") pod \"payload-pre-processing-e2e-mt-1ec4b892-6b49f5f675-hxqrb\" (UID: \"47cf254c-610c-4b37-8bcd-dd4c11dcff35\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-1ec4b892-6b49f5f675-hxqrb" Nov 03 13:35:13.629311 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:13.626333 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47cf254c-610c-4b37-8bcd-dd4c11dcff35-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-1ec4b892-6b49f5f675-hxqrb\" (UID: \"47cf254c-610c-4b37-8bcd-dd4c11dcff35\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-1ec4b892-6b49f5f675-hxqrb" Nov 03 13:35:13.629311 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:13.627526 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47cf254c-610c-4b37-8bcd-dd4c11dcff35-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-1ec4b892-6b49f5f675-hxqrb\" (UID: \"47cf254c-610c-4b37-8bcd-dd4c11dcff35\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-1ec4b892-6b49f5f675-hxqrb" Nov 03 13:35:13.721400 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:13.720870 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qlrb4\" (UniqueName: \"kubernetes.io/projected/47cf254c-610c-4b37-8bcd-dd4c11dcff35-kube-api-access-qlrb4\") pod \"payload-pre-processing-e2e-mt-1ec4b892-6b49f5f675-hxqrb\" (UID: \"47cf254c-610c-4b37-8bcd-dd4c11dcff35\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-1ec4b892-6b49f5f675-hxqrb" Nov 03 13:35:13.765800 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:13.764567 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-1ec4b892-6b49f5f675-hxqrb" Nov 03 13:35:14.004365 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:13.997434 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-2cwvx" event={"ID":"7d6b62e0-65bf-4766-93f4-b065c82304fb","Type":"ContainerStarted","Data":"061c2f55a0635f745abff9de665d06e2ca33a3f186426b5378c3654d6b59fe1a"} Nov 03 13:35:14.245550 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:14.245515 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-1ec4b892-6b49f5f675-hxqrb"] Nov 03 13:35:14.247411 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:35:14.247360 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47cf254c_610c_4b37_8bcd_dd4c11dcff35.slice/crio-be645f18bac117ad9a3e7eccdf0d496141b9d1928028b159053ac4e8a3bc94c5 WatchSource:0}: Error finding container be645f18bac117ad9a3e7eccdf0d496141b9d1928028b159053ac4e8a3bc94c5: Status 404 returned error can't find the container with id be645f18bac117ad9a3e7eccdf0d496141b9d1928028b159053ac4e8a3bc94c5 Nov 03 13:35:14.255171 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:14.255106 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-2cwvx" Nov 03 13:35:14.256180 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:14.256156 5024 patch_prober.go:29] interesting pod/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-2cwvx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" start-of-body= Nov 03 13:35:14.256294 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:14.256201 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-2cwvx" podUID="7d6b62e0-65bf-4766-93f4-b065c82304fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" Nov 03 13:35:14.551956 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:14.551856 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5546676475-98nhd"] Nov 03 13:35:14.555102 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:14.555073 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-69d95d887d-xsbb2"] Nov 03 13:35:14.564914 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:14.564882 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-69d95d887d-xsbb2" Nov 03 13:35:15.087409 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:15.087185 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-1ec4b892-6b49f5f675-hxqrb" event={"ID":"47cf254c-610c-4b37-8bcd-dd4c11dcff35","Type":"ContainerStarted","Data":"22e4a525e9086e9659155a80406699dcc228893b354d9a9dc27db29f74069f0d"} Nov 03 13:35:15.087409 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:15.087255 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-1ec4b892-6b49f5f675-hxqrb" event={"ID":"47cf254c-610c-4b37-8bcd-dd4c11dcff35","Type":"ContainerStarted","Data":"be645f18bac117ad9a3e7eccdf0d496141b9d1928028b159053ac4e8a3bc94c5"} Nov 03 13:35:15.087409 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:15.087346 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-1ec4b892-6b49f5f675-hxqrb" Nov 03 13:35:15.267304 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:15.263835 5024 patch_prober.go:29] interesting pod/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-2cwvx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" start-of-body= Nov 03 13:35:15.267304 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:15.263912 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-2cwvx" podUID="7d6b62e0-65bf-4766-93f4-b065c82304fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" Nov 03 13:35:16.261817 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:16.258599 5024 patch_prober.go:29] interesting pod/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-2cwvx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" start-of-body= Nov 03 13:35:16.261817 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:16.258682 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-2cwvx" podUID="7d6b62e0-65bf-4766-93f4-b065c82304fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" Nov 03 13:35:17.255620 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:17.255581 5024 patch_prober.go:29] interesting pod/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-2cwvx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" start-of-body= Nov 03 13:35:17.255819 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:17.255659 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-2cwvx" podUID="7d6b62e0-65bf-4766-93f4-b065c82304fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" Nov 03 13:35:17.427313 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:17.423814 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-1ec4b892-6b49f5f675-hxqrb" podStartSLOduration=4.423790719 podStartE2EDuration="4.423790719s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:35:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:35:15.132302445 +0000 UTC m=+1595.140493720" watchObservedRunningTime="2026-11-03 13:35:17.423790719 +0000 UTC m=+1597.431981998" Nov 03 13:35:17.430599 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:17.430557 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-c47670-openshift-default-7f67b9bfc-rv4l7"] Nov 03 13:35:17.432001 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:17.431957 5024 cpu_manager.go:414] "containerMap: removing container" podUID="7d77cd17-8ce7-4fe7-9b61-7e4c7cdc720b" containerName="istio-proxy" Nov 03 13:35:17.432189 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:17.432174 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d77cd17-8ce7-4fe7-9b61-7e4c7cdc720b" containerName="istio-proxy" Nov 03 13:35:17.432625 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:17.432600 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d77cd17-8ce7-4fe7-9b61-7e4c7cdc720b" containerName="istio-proxy" Nov 03 13:35:17.436900 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:17.436869 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-c47670-openshift-default-7f67b9bfc-rv4l7"] Nov 03 13:35:17.437166 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:17.437103 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-c47670-openshift-default-7f67b9bfc-rv4l7" Nov 03 13:35:17.449481 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:17.448673 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/160d4b07-2558-4e4d-afb7-a7d509cc68ca-credential-socket\") pod \"e2e-extmod-a-c47670-openshift-default-7f67b9bfc-rv4l7\" (UID: \"160d4b07-2558-4e4d-afb7-a7d509cc68ca\") " pod="openshift-ingress/e2e-extmod-a-c47670-openshift-default-7f67b9bfc-rv4l7" Nov 03 13:35:17.449481 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:17.448739 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/160d4b07-2558-4e4d-afb7-a7d509cc68ca-workload-socket\") pod \"e2e-extmod-a-c47670-openshift-default-7f67b9bfc-rv4l7\" (UID: \"160d4b07-2558-4e4d-afb7-a7d509cc68ca\") " pod="openshift-ingress/e2e-extmod-a-c47670-openshift-default-7f67b9bfc-rv4l7" Nov 03 13:35:17.449481 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:17.448771 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/160d4b07-2558-4e4d-afb7-a7d509cc68ca-istiod-ca-cert\") pod \"e2e-extmod-a-c47670-openshift-default-7f67b9bfc-rv4l7\" (UID: \"160d4b07-2558-4e4d-afb7-a7d509cc68ca\") " pod="openshift-ingress/e2e-extmod-a-c47670-openshift-default-7f67b9bfc-rv4l7" Nov 03 13:35:17.449481 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:17.448832 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/160d4b07-2558-4e4d-afb7-a7d509cc68ca-workload-certs\") pod \"e2e-extmod-a-c47670-openshift-default-7f67b9bfc-rv4l7\" (UID: \"160d4b07-2558-4e4d-afb7-a7d509cc68ca\") " pod="openshift-ingress/e2e-extmod-a-c47670-openshift-default-7f67b9bfc-rv4l7" Nov 03 13:35:17.449481 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:17.448871 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/160d4b07-2558-4e4d-afb7-a7d509cc68ca-istio-podinfo\") pod \"e2e-extmod-a-c47670-openshift-default-7f67b9bfc-rv4l7\" (UID: \"160d4b07-2558-4e4d-afb7-a7d509cc68ca\") " pod="openshift-ingress/e2e-extmod-a-c47670-openshift-default-7f67b9bfc-rv4l7" Nov 03 13:35:17.449481 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:17.448920 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/160d4b07-2558-4e4d-afb7-a7d509cc68ca-istio-data\") pod \"e2e-extmod-a-c47670-openshift-default-7f67b9bfc-rv4l7\" (UID: \"160d4b07-2558-4e4d-afb7-a7d509cc68ca\") " pod="openshift-ingress/e2e-extmod-a-c47670-openshift-default-7f67b9bfc-rv4l7" Nov 03 13:35:17.449481 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:17.449019 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/160d4b07-2558-4e4d-afb7-a7d509cc68ca-istio-envoy\") pod \"e2e-extmod-a-c47670-openshift-default-7f67b9bfc-rv4l7\" (UID: \"160d4b07-2558-4e4d-afb7-a7d509cc68ca\") " pod="openshift-ingress/e2e-extmod-a-c47670-openshift-default-7f67b9bfc-rv4l7" Nov 03 13:35:17.449481 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:17.449109 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z74vs\" (UniqueName: \"kubernetes.io/projected/160d4b07-2558-4e4d-afb7-a7d509cc68ca-kube-api-access-z74vs\") pod \"e2e-extmod-a-c47670-openshift-default-7f67b9bfc-rv4l7\" (UID: \"160d4b07-2558-4e4d-afb7-a7d509cc68ca\") " pod="openshift-ingress/e2e-extmod-a-c47670-openshift-default-7f67b9bfc-rv4l7" Nov 03 13:35:17.449481 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:17.449236 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/160d4b07-2558-4e4d-afb7-a7d509cc68ca-istio-token\") pod \"e2e-extmod-a-c47670-openshift-default-7f67b9bfc-rv4l7\" (UID: \"160d4b07-2558-4e4d-afb7-a7d509cc68ca\") " pod="openshift-ingress/e2e-extmod-a-c47670-openshift-default-7f67b9bfc-rv4l7" Nov 03 13:35:17.450055 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:17.450019 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-c47670-openshift-default-dockercfg-rlhvv\"" Nov 03 13:35:17.551356 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:17.551231 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/160d4b07-2558-4e4d-afb7-a7d509cc68ca-workload-certs\") pod \"e2e-extmod-a-c47670-openshift-default-7f67b9bfc-rv4l7\" (UID: \"160d4b07-2558-4e4d-afb7-a7d509cc68ca\") " pod="openshift-ingress/e2e-extmod-a-c47670-openshift-default-7f67b9bfc-rv4l7" Nov 03 13:35:17.551356 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:17.551304 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/160d4b07-2558-4e4d-afb7-a7d509cc68ca-istio-podinfo\") pod \"e2e-extmod-a-c47670-openshift-default-7f67b9bfc-rv4l7\" (UID: \"160d4b07-2558-4e4d-afb7-a7d509cc68ca\") " pod="openshift-ingress/e2e-extmod-a-c47670-openshift-default-7f67b9bfc-rv4l7" Nov 03 13:35:17.551356 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:17.551347 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/160d4b07-2558-4e4d-afb7-a7d509cc68ca-istio-data\") pod \"e2e-extmod-a-c47670-openshift-default-7f67b9bfc-rv4l7\" (UID: \"160d4b07-2558-4e4d-afb7-a7d509cc68ca\") " pod="openshift-ingress/e2e-extmod-a-c47670-openshift-default-7f67b9bfc-rv4l7" Nov 03 13:35:17.551656 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:17.551441 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/160d4b07-2558-4e4d-afb7-a7d509cc68ca-istio-envoy\") pod \"e2e-extmod-a-c47670-openshift-default-7f67b9bfc-rv4l7\" (UID: \"160d4b07-2558-4e4d-afb7-a7d509cc68ca\") " pod="openshift-ingress/e2e-extmod-a-c47670-openshift-default-7f67b9bfc-rv4l7" Nov 03 13:35:17.551656 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:17.551529 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z74vs\" (UniqueName: \"kubernetes.io/projected/160d4b07-2558-4e4d-afb7-a7d509cc68ca-kube-api-access-z74vs\") pod \"e2e-extmod-a-c47670-openshift-default-7f67b9bfc-rv4l7\" (UID: \"160d4b07-2558-4e4d-afb7-a7d509cc68ca\") " pod="openshift-ingress/e2e-extmod-a-c47670-openshift-default-7f67b9bfc-rv4l7" Nov 03 13:35:17.551656 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:17.551651 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/160d4b07-2558-4e4d-afb7-a7d509cc68ca-istio-token\") pod \"e2e-extmod-a-c47670-openshift-default-7f67b9bfc-rv4l7\" (UID: \"160d4b07-2558-4e4d-afb7-a7d509cc68ca\") " pod="openshift-ingress/e2e-extmod-a-c47670-openshift-default-7f67b9bfc-rv4l7" Nov 03 13:35:17.551822 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:17.551761 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/160d4b07-2558-4e4d-afb7-a7d509cc68ca-workload-certs\") pod \"e2e-extmod-a-c47670-openshift-default-7f67b9bfc-rv4l7\" (UID: \"160d4b07-2558-4e4d-afb7-a7d509cc68ca\") " pod="openshift-ingress/e2e-extmod-a-c47670-openshift-default-7f67b9bfc-rv4l7" Nov 03 13:35:17.551880 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:17.551816 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/160d4b07-2558-4e4d-afb7-a7d509cc68ca-credential-socket\") pod \"e2e-extmod-a-c47670-openshift-default-7f67b9bfc-rv4l7\" (UID: \"160d4b07-2558-4e4d-afb7-a7d509cc68ca\") " pod="openshift-ingress/e2e-extmod-a-c47670-openshift-default-7f67b9bfc-rv4l7" Nov 03 13:35:17.551880 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:17.551864 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/160d4b07-2558-4e4d-afb7-a7d509cc68ca-workload-socket\") pod \"e2e-extmod-a-c47670-openshift-default-7f67b9bfc-rv4l7\" (UID: \"160d4b07-2558-4e4d-afb7-a7d509cc68ca\") " pod="openshift-ingress/e2e-extmod-a-c47670-openshift-default-7f67b9bfc-rv4l7" Nov 03 13:35:17.551961 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:17.551892 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/160d4b07-2558-4e4d-afb7-a7d509cc68ca-istiod-ca-cert\") pod \"e2e-extmod-a-c47670-openshift-default-7f67b9bfc-rv4l7\" (UID: \"160d4b07-2558-4e4d-afb7-a7d509cc68ca\") " pod="openshift-ingress/e2e-extmod-a-c47670-openshift-default-7f67b9bfc-rv4l7" Nov 03 13:35:17.551961 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:17.551818 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/160d4b07-2558-4e4d-afb7-a7d509cc68ca-istio-data\") pod \"e2e-extmod-a-c47670-openshift-default-7f67b9bfc-rv4l7\" (UID: \"160d4b07-2558-4e4d-afb7-a7d509cc68ca\") " pod="openshift-ingress/e2e-extmod-a-c47670-openshift-default-7f67b9bfc-rv4l7" Nov 03 13:35:17.552208 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:17.552183 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/160d4b07-2558-4e4d-afb7-a7d509cc68ca-credential-socket\") pod \"e2e-extmod-a-c47670-openshift-default-7f67b9bfc-rv4l7\" (UID: \"160d4b07-2558-4e4d-afb7-a7d509cc68ca\") " pod="openshift-ingress/e2e-extmod-a-c47670-openshift-default-7f67b9bfc-rv4l7" Nov 03 13:35:17.552416 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:17.552388 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/160d4b07-2558-4e4d-afb7-a7d509cc68ca-workload-socket\") pod \"e2e-extmod-a-c47670-openshift-default-7f67b9bfc-rv4l7\" (UID: \"160d4b07-2558-4e4d-afb7-a7d509cc68ca\") " pod="openshift-ingress/e2e-extmod-a-c47670-openshift-default-7f67b9bfc-rv4l7" Nov 03 13:35:17.552684 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:17.552664 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/160d4b07-2558-4e4d-afb7-a7d509cc68ca-istiod-ca-cert\") pod \"e2e-extmod-a-c47670-openshift-default-7f67b9bfc-rv4l7\" (UID: \"160d4b07-2558-4e4d-afb7-a7d509cc68ca\") " pod="openshift-ingress/e2e-extmod-a-c47670-openshift-default-7f67b9bfc-rv4l7" Nov 03 13:35:17.555084 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:17.554349 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/160d4b07-2558-4e4d-afb7-a7d509cc68ca-istio-podinfo\") pod \"e2e-extmod-a-c47670-openshift-default-7f67b9bfc-rv4l7\" (UID: \"160d4b07-2558-4e4d-afb7-a7d509cc68ca\") " pod="openshift-ingress/e2e-extmod-a-c47670-openshift-default-7f67b9bfc-rv4l7" Nov 03 13:35:17.557472 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:17.557441 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/160d4b07-2558-4e4d-afb7-a7d509cc68ca-istio-envoy\") pod \"e2e-extmod-a-c47670-openshift-default-7f67b9bfc-rv4l7\" (UID: \"160d4b07-2558-4e4d-afb7-a7d509cc68ca\") " pod="openshift-ingress/e2e-extmod-a-c47670-openshift-default-7f67b9bfc-rv4l7" Nov 03 13:35:17.563078 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:17.563050 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/160d4b07-2558-4e4d-afb7-a7d509cc68ca-istio-token\") pod \"e2e-extmod-a-c47670-openshift-default-7f67b9bfc-rv4l7\" (UID: \"160d4b07-2558-4e4d-afb7-a7d509cc68ca\") " pod="openshift-ingress/e2e-extmod-a-c47670-openshift-default-7f67b9bfc-rv4l7" Nov 03 13:35:17.563331 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:17.563269 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z74vs\" (UniqueName: \"kubernetes.io/projected/160d4b07-2558-4e4d-afb7-a7d509cc68ca-kube-api-access-z74vs\") pod \"e2e-extmod-a-c47670-openshift-default-7f67b9bfc-rv4l7\" (UID: \"160d4b07-2558-4e4d-afb7-a7d509cc68ca\") " pod="openshift-ingress/e2e-extmod-a-c47670-openshift-default-7f67b9bfc-rv4l7" Nov 03 13:35:17.798656 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:17.798611 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-c47670-openshift-default-7f67b9bfc-rv4l7" Nov 03 13:35:18.256858 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:18.256418 5024 patch_prober.go:29] interesting pod/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-2cwvx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" start-of-body= Nov 03 13:35:18.256858 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:18.256487 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-2cwvx" podUID="7d6b62e0-65bf-4766-93f4-b065c82304fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" Nov 03 13:35:18.409954 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:35:18.409858 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod160d4b07_2558_4e4d_afb7_a7d509cc68ca.slice/crio-0858c5f08ab458bb3116acbe14801152ccd7d0227cf6ffe9afdc7be5423aca8f WatchSource:0}: Error finding container 0858c5f08ab458bb3116acbe14801152ccd7d0227cf6ffe9afdc7be5423aca8f: Status 404 returned error can't find the container with id 0858c5f08ab458bb3116acbe14801152ccd7d0227cf6ffe9afdc7be5423aca8f Nov 03 13:35:18.413331 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:18.413249 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-c47670-openshift-default-7f67b9bfc-rv4l7"] Nov 03 13:35:18.417829 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:18.417795 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:35:18.417926 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:18.417877 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:35:18.417926 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:18.417920 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:35:19.255977 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:19.255940 5024 patch_prober.go:29] interesting pod/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-2cwvx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" start-of-body= Nov 03 13:35:19.256462 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:19.256434 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-2cwvx" podUID="7d6b62e0-65bf-4766-93f4-b065c82304fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" Nov 03 13:35:19.264488 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:19.264448 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-c47670-openshift-default-7f67b9bfc-rv4l7" event={"ID":"160d4b07-2558-4e4d-afb7-a7d509cc68ca","Type":"ContainerStarted","Data":"9b8b4fed8f49f179cf7e812cb0767af05932b07fc896bd7af1f6ccf37503d2f8"} Nov 03 13:35:19.264719 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:19.264516 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-c47670-openshift-default-7f67b9bfc-rv4l7" event={"ID":"160d4b07-2558-4e4d-afb7-a7d509cc68ca","Type":"ContainerStarted","Data":"0858c5f08ab458bb3116acbe14801152ccd7d0227cf6ffe9afdc7be5423aca8f"} Nov 03 13:35:19.412304 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:19.396919 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-c47670-openshift-default-7f67b9bfc-rv4l7" podStartSLOduration=2.39689779 podStartE2EDuration="2.39689779s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:35:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:35:19.327706722 +0000 UTC m=+1599.335898000" watchObservedRunningTime="2026-11-03 13:35:19.39689779 +0000 UTC m=+1599.405089069" Nov 03 13:35:19.558455 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:19.558354 5024 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-69d95d887d-xsbb2" podUID="d6860fac-adfc-4c91-aa20-522cf4b68a2c" containerName="payload-processing" containerID="cri-o://1f230d39621a9a90a5139c46988eb6bd045018b0736c200295785263783bc80c" gracePeriod=25 Nov 03 13:35:19.559045 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:19.558806 5024 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-5546676475-98nhd" podUID="49f61653-f10c-4692-bc8f-a379cfd3b073" containerName="payload-processing" containerID="cri-o://6b3ee2edf63c11e57730fbdd335caddd3e364aded43884df66dd1c1cbb610b8d" gracePeriod=25 Nov 03 13:35:19.827359 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:19.824034 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-c47670-openshift-default-7f67b9bfc-rv4l7" Nov 03 13:35:19.867469 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:19.866956 5024 patch_prober.go:29] interesting pod/e2e-extmod-a-c47670-openshift-default-7f67b9bfc-rv4l7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" start-of-body= Nov 03 13:35:19.867469 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:19.867012 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-c47670-openshift-default-7f67b9bfc-rv4l7" podUID="160d4b07-2558-4e4d-afb7-a7d509cc68ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" Nov 03 13:35:20.148690 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:20.148581 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5546676475-98nhd" Nov 03 13:35:20.186574 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:20.186536 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-69d95d887d-xsbb2" Nov 03 13:35:20.232217 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:20.232172 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d6860fac-adfc-4c91-aa20-522cf4b68a2c-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d6860fac-adfc-4c91-aa20-522cf4b68a2c-routing-overlay-volume\") pod \"d6860fac-adfc-4c91-aa20-522cf4b68a2c\" (UID: \"d6860fac-adfc-4c91-aa20-522cf4b68a2c\") " Nov 03 13:35:20.232595 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:20.232570 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6860fac-adfc-4c91-aa20-522cf4b68a2c-provider-credentials\" (UniqueName: \"kubernetes.io/projected/d6860fac-adfc-4c91-aa20-522cf4b68a2c-provider-credentials\") pod \"d6860fac-adfc-4c91-aa20-522cf4b68a2c\" (UID: \"d6860fac-adfc-4c91-aa20-522cf4b68a2c\") " Nov 03 13:35:20.232772 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:20.232739 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6860fac-adfc-4c91-aa20-522cf4b68a2c-routing-overlay-volume" pod "d6860fac-adfc-4c91-aa20-522cf4b68a2c" (UID: "d6860fac-adfc-4c91-aa20-522cf4b68a2c"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:35:20.232894 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:20.232872 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49f61653-f10c-4692-bc8f-a379cfd3b073-kube-api-access-dh74s\" (UniqueName: \"kubernetes.io/projected/49f61653-f10c-4692-bc8f-a379cfd3b073-kube-api-access-dh74s\") pod \"49f61653-f10c-4692-bc8f-a379cfd3b073\" (UID: \"49f61653-f10c-4692-bc8f-a379cfd3b073\") " Nov 03 13:35:20.233048 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:20.233030 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d6860fac-adfc-4c91-aa20-522cf4b68a2c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6860fac-adfc-4c91-aa20-522cf4b68a2c-plugins-config-volume\") pod \"d6860fac-adfc-4c91-aa20-522cf4b68a2c\" (UID: \"d6860fac-adfc-4c91-aa20-522cf4b68a2c\") " Nov 03 13:35:20.233195 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:20.233179 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6860fac-adfc-4c91-aa20-522cf4b68a2c-kube-api-access-cfkng\" (UniqueName: \"kubernetes.io/projected/d6860fac-adfc-4c91-aa20-522cf4b68a2c-kube-api-access-cfkng\") pod \"d6860fac-adfc-4c91-aa20-522cf4b68a2c\" (UID: \"d6860fac-adfc-4c91-aa20-522cf4b68a2c\") " Nov 03 13:35:20.233444 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:20.233425 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/49f61653-f10c-4692-bc8f-a379cfd3b073-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/49f61653-f10c-4692-bc8f-a379cfd3b073-routing-overlay-volume\") pod \"49f61653-f10c-4692-bc8f-a379cfd3b073\" (UID: \"49f61653-f10c-4692-bc8f-a379cfd3b073\") " Nov 03 13:35:20.233710 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:20.233692 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49f61653-f10c-4692-bc8f-a379cfd3b073-provider-credentials\" (UniqueName: \"kubernetes.io/projected/49f61653-f10c-4692-bc8f-a379cfd3b073-provider-credentials\") pod \"49f61653-f10c-4692-bc8f-a379cfd3b073\" (UID: \"49f61653-f10c-4692-bc8f-a379cfd3b073\") " Nov 03 13:35:20.233949 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:20.233917 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6860fac-adfc-4c91-aa20-522cf4b68a2c-plugins-config-volume" pod "d6860fac-adfc-4c91-aa20-522cf4b68a2c" (UID: "d6860fac-adfc-4c91-aa20-522cf4b68a2c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:35:20.234238 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:20.233932 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/49f61653-f10c-4692-bc8f-a379cfd3b073-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49f61653-f10c-4692-bc8f-a379cfd3b073-plugins-config-volume\") pod \"49f61653-f10c-4692-bc8f-a379cfd3b073\" (UID: \"49f61653-f10c-4692-bc8f-a379cfd3b073\") " Nov 03 13:35:20.235309 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:20.235255 5024 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d6860fac-adfc-4c91-aa20-522cf4b68a2c-routing-overlay-volume\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:35:20.235457 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:20.235440 5024 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6860fac-adfc-4c91-aa20-522cf4b68a2c-plugins-config-volume\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:35:20.236393 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:20.234324 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49f61653-f10c-4692-bc8f-a379cfd3b073-plugins-config-volume" pod "49f61653-f10c-4692-bc8f-a379cfd3b073" (UID: "49f61653-f10c-4692-bc8f-a379cfd3b073"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:35:20.236704 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:20.236683 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49f61653-f10c-4692-bc8f-a379cfd3b073-routing-overlay-volume" pod "49f61653-f10c-4692-bc8f-a379cfd3b073" (UID: "49f61653-f10c-4692-bc8f-a379cfd3b073"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:35:20.238232 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:20.238209 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6860fac-adfc-4c91-aa20-522cf4b68a2c-kube-api-access-cfkng" pod "d6860fac-adfc-4c91-aa20-522cf4b68a2c" (UID: "d6860fac-adfc-4c91-aa20-522cf4b68a2c"). InnerVolumeSpecName "kube-api-access-cfkng". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:35:20.238813 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:20.238788 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49f61653-f10c-4692-bc8f-a379cfd3b073-provider-credentials" pod "49f61653-f10c-4692-bc8f-a379cfd3b073" (UID: "49f61653-f10c-4692-bc8f-a379cfd3b073"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:35:20.240047 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:20.240025 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49f61653-f10c-4692-bc8f-a379cfd3b073-kube-api-access-dh74s" pod "49f61653-f10c-4692-bc8f-a379cfd3b073" (UID: "49f61653-f10c-4692-bc8f-a379cfd3b073"). InnerVolumeSpecName "kube-api-access-dh74s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:35:20.241620 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:20.241597 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6860fac-adfc-4c91-aa20-522cf4b68a2c-provider-credentials" pod "d6860fac-adfc-4c91-aa20-522cf4b68a2c" (UID: "d6860fac-adfc-4c91-aa20-522cf4b68a2c"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:35:20.255397 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:20.255269 5024 patch_prober.go:29] interesting pod/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-2cwvx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" start-of-body= Nov 03 13:35:20.255669 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:20.255617 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-2cwvx" podUID="7d6b62e0-65bf-4766-93f4-b065c82304fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" Nov 03 13:35:20.277158 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:20.277123 5024 generic.go:356] "Generic (PLEG): container finished" podID="49f61653-f10c-4692-bc8f-a379cfd3b073" containerID="6b3ee2edf63c11e57730fbdd335caddd3e364aded43884df66dd1c1cbb610b8d" exitCode=0 Nov 03 13:35:20.277634 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:20.277173 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5546676475-98nhd" Nov 03 13:35:20.277634 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:20.277201 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5546676475-98nhd" event={"ID":"49f61653-f10c-4692-bc8f-a379cfd3b073","Type":"ContainerDied","Data":"6b3ee2edf63c11e57730fbdd335caddd3e364aded43884df66dd1c1cbb610b8d"} Nov 03 13:35:20.277634 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:20.277263 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5546676475-98nhd" event={"ID":"49f61653-f10c-4692-bc8f-a379cfd3b073","Type":"ContainerDied","Data":"d0aebfd5855b17467ebb3d0c7519ee23caba2b659b60d50afe608ee7176ed6bc"} Nov 03 13:35:20.277634 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:20.277380 5024 scope.go:122] "RemoveContainer" containerID="6b3ee2edf63c11e57730fbdd335caddd3e364aded43884df66dd1c1cbb610b8d" Nov 03 13:35:20.280603 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:20.279479 5024 generic.go:356] "Generic (PLEG): container finished" podID="d6860fac-adfc-4c91-aa20-522cf4b68a2c" containerID="1f230d39621a9a90a5139c46988eb6bd045018b0736c200295785263783bc80c" exitCode=0 Nov 03 13:35:20.280603 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:20.280521 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-69d95d887d-xsbb2" Nov 03 13:35:20.284476 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:20.284436 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-69d95d887d-xsbb2" event={"ID":"d6860fac-adfc-4c91-aa20-522cf4b68a2c","Type":"ContainerDied","Data":"1f230d39621a9a90a5139c46988eb6bd045018b0736c200295785263783bc80c"} Nov 03 13:35:20.284607 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:20.284498 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-69d95d887d-xsbb2" event={"ID":"d6860fac-adfc-4c91-aa20-522cf4b68a2c","Type":"ContainerDied","Data":"67c145d97fd7f1dea22cbcac403c524783e6c10fd0fa6526efba95836cb73bc7"} Nov 03 13:35:20.293216 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:20.293187 5024 scope.go:122] "RemoveContainer" containerID="6b3ee2edf63c11e57730fbdd335caddd3e364aded43884df66dd1c1cbb610b8d" Nov 03 13:35:20.294324 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:35:20.294167 5024 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b3ee2edf63c11e57730fbdd335caddd3e364aded43884df66dd1c1cbb610b8d\": container with ID starting with 6b3ee2edf63c11e57730fbdd335caddd3e364aded43884df66dd1c1cbb610b8d not found: ID does not exist" containerID="6b3ee2edf63c11e57730fbdd335caddd3e364aded43884df66dd1c1cbb610b8d" Nov 03 13:35:20.294324 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:20.294209 5024 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b3ee2edf63c11e57730fbdd335caddd3e364aded43884df66dd1c1cbb610b8d"} err="failed to get container status \"6b3ee2edf63c11e57730fbdd335caddd3e364aded43884df66dd1c1cbb610b8d\": rpc error: code = NotFound desc = could not find container \"6b3ee2edf63c11e57730fbdd335caddd3e364aded43884df66dd1c1cbb610b8d\": container with ID starting with 6b3ee2edf63c11e57730fbdd335caddd3e364aded43884df66dd1c1cbb610b8d not found: ID does not exist" Nov 03 13:35:20.294324 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:20.294235 5024 scope.go:122] "RemoveContainer" containerID="1f230d39621a9a90a5139c46988eb6bd045018b0736c200295785263783bc80c" Nov 03 13:35:20.313331 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:20.313296 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5546676475-98nhd"] Nov 03 13:35:20.315369 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:20.315340 5024 scope.go:122] "RemoveContainer" containerID="1f230d39621a9a90a5139c46988eb6bd045018b0736c200295785263783bc80c" Nov 03 13:35:20.315816 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:35:20.315791 5024 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f230d39621a9a90a5139c46988eb6bd045018b0736c200295785263783bc80c\": container with ID starting with 1f230d39621a9a90a5139c46988eb6bd045018b0736c200295785263783bc80c not found: ID does not exist" containerID="1f230d39621a9a90a5139c46988eb6bd045018b0736c200295785263783bc80c" Nov 03 13:35:20.315913 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:20.315825 5024 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f230d39621a9a90a5139c46988eb6bd045018b0736c200295785263783bc80c"} err="failed to get container status \"1f230d39621a9a90a5139c46988eb6bd045018b0736c200295785263783bc80c\": rpc error: code = NotFound desc = could not find container \"1f230d39621a9a90a5139c46988eb6bd045018b0736c200295785263783bc80c\": container with ID starting with 1f230d39621a9a90a5139c46988eb6bd045018b0736c200295785263783bc80c not found: ID does not exist" Nov 03 13:35:20.319723 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:20.319696 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-5546676475-98nhd"] Nov 03 13:35:20.326879 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:20.326850 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-69d95d887d-xsbb2"] Nov 03 13:35:20.334187 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:20.334157 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-69d95d887d-xsbb2"] Nov 03 13:35:20.337455 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:20.337382 5024 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49f61653-f10c-4692-bc8f-a379cfd3b073-plugins-config-volume\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:35:20.337455 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:20.337415 5024 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d6860fac-adfc-4c91-aa20-522cf4b68a2c-provider-credentials\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:35:20.337455 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:20.337437 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dh74s\" (UniqueName: \"kubernetes.io/projected/49f61653-f10c-4692-bc8f-a379cfd3b073-kube-api-access-dh74s\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:35:20.337455 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:20.337453 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cfkng\" (UniqueName: \"kubernetes.io/projected/d6860fac-adfc-4c91-aa20-522cf4b68a2c-kube-api-access-cfkng\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:35:20.337692 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:20.337468 5024 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/49f61653-f10c-4692-bc8f-a379cfd3b073-routing-overlay-volume\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:35:20.337692 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:20.337481 5024 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/49f61653-f10c-4692-bc8f-a379cfd3b073-provider-credentials\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:35:20.804617 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:20.804582 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-c47670-openshift-default-7f67b9bfc-rv4l7" Nov 03 13:35:21.149650 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:21.149614 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-2cwvx" Nov 03 13:35:21.297475 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:21.296794 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-2cwvx" Nov 03 13:35:21.346504 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:21.346472 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-c47670-openshift-default-7f67b9bfc-rv4l7" Nov 03 13:35:21.347372 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:21.347347 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-c47670-openshift-default-7f67b9bfc-rv4l7" Nov 03 13:35:21.347671 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:21.347653 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-2cwvx" Nov 03 13:35:22.111302 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:22.111261 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="49f61653-f10c-4692-bc8f-a379cfd3b073" path="/var/lib/kubelet/pods/49f61653-f10c-4692-bc8f-a379cfd3b073/volumes" Nov 03 13:35:22.111849 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:22.111829 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d6860fac-adfc-4c91-aa20-522cf4b68a2c" path="/var/lib/kubelet/pods/d6860fac-adfc-4c91-aa20-522cf4b68a2c/volumes" Nov 03 13:35:23.546360 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:23.539742 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-c47670-54c8bb9b89-fzdnn"] Nov 03 13:35:23.555531 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:23.548150 5024 cpu_manager.go:414] "containerMap: removing container" podUID="49f61653-f10c-4692-bc8f-a379cfd3b073" containerName="payload-processing" Nov 03 13:35:23.555531 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:23.548186 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49f61653-f10c-4692-bc8f-a379cfd3b073" containerName="payload-processing" Nov 03 13:35:23.555531 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:23.548221 5024 cpu_manager.go:414] "containerMap: removing container" podUID="d6860fac-adfc-4c91-aa20-522cf4b68a2c" containerName="payload-processing" Nov 03 13:35:23.555531 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:23.548231 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6860fac-adfc-4c91-aa20-522cf4b68a2c" containerName="payload-processing" Nov 03 13:35:23.555531 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:23.548435 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="49f61653-f10c-4692-bc8f-a379cfd3b073" containerName="payload-processing" Nov 03 13:35:23.555531 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:23.548451 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6860fac-adfc-4c91-aa20-522cf4b68a2c" containerName="payload-processing" Nov 03 13:35:23.555531 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:23.554895 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-c47670-54c8bb9b89-fzdnn" Nov 03 13:35:23.587226 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:23.586964 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-c47670\"" Nov 03 13:35:23.589877 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:23.589852 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-c47670-dockercfg-9tc8h\"" Nov 03 13:35:23.595270 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:23.595237 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-c47670-54c8bb9b89-fzdnn"] Nov 03 13:35:23.628385 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:23.628335 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8sfg\" (UniqueName: \"kubernetes.io/projected/7a30a13a-965d-4291-9acd-f109a6ead710-kube-api-access-m8sfg\") pod \"payload-processing-e2e-extmod-a-c47670-54c8bb9b89-fzdnn\" (UID: \"7a30a13a-965d-4291-9acd-f109a6ead710\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-c47670-54c8bb9b89-fzdnn" Nov 03 13:35:23.628594 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:23.628559 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a30a13a-965d-4291-9acd-f109a6ead710-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-c47670-54c8bb9b89-fzdnn\" (UID: \"7a30a13a-965d-4291-9acd-f109a6ead710\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-c47670-54c8bb9b89-fzdnn" Nov 03 13:35:23.730121 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:23.730081 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8sfg\" (UniqueName: \"kubernetes.io/projected/7a30a13a-965d-4291-9acd-f109a6ead710-kube-api-access-m8sfg\") pod \"payload-processing-e2e-extmod-a-c47670-54c8bb9b89-fzdnn\" (UID: \"7a30a13a-965d-4291-9acd-f109a6ead710\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-c47670-54c8bb9b89-fzdnn" Nov 03 13:35:23.730299 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:23.730215 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a30a13a-965d-4291-9acd-f109a6ead710-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-c47670-54c8bb9b89-fzdnn\" (UID: \"7a30a13a-965d-4291-9acd-f109a6ead710\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-c47670-54c8bb9b89-fzdnn" Nov 03 13:35:23.731039 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:23.731013 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a30a13a-965d-4291-9acd-f109a6ead710-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-c47670-54c8bb9b89-fzdnn\" (UID: \"7a30a13a-965d-4291-9acd-f109a6ead710\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-c47670-54c8bb9b89-fzdnn" Nov 03 13:35:23.748264 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:23.748236 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8sfg\" (UniqueName: \"kubernetes.io/projected/7a30a13a-965d-4291-9acd-f109a6ead710-kube-api-access-m8sfg\") pod \"payload-processing-e2e-extmod-a-c47670-54c8bb9b89-fzdnn\" (UID: \"7a30a13a-965d-4291-9acd-f109a6ead710\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-c47670-54c8bb9b89-fzdnn" Nov 03 13:35:23.917722 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:23.917674 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-c47670-54c8bb9b89-fzdnn" Nov 03 13:35:24.325608 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:35:24.325578 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a30a13a_965d_4291_9acd_f109a6ead710.slice/crio-12b72fc0ff14678f253add4dce17d31b74fc172c0d1bdcc40ead75ecb6cd79f9 WatchSource:0}: Error finding container 12b72fc0ff14678f253add4dce17d31b74fc172c0d1bdcc40ead75ecb6cd79f9: Status 404 returned error can't find the container with id 12b72fc0ff14678f253add4dce17d31b74fc172c0d1bdcc40ead75ecb6cd79f9 Nov 03 13:35:24.327451 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:24.327421 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-c47670-54c8bb9b89-fzdnn"] Nov 03 13:35:24.607817 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:24.607779 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-c47670-54c8bb9b89-fzdnn" event={"ID":"7a30a13a-965d-4291-9acd-f109a6ead710","Type":"ContainerStarted","Data":"ccdfa1856a6bba533687ddc2d36f10d15651ea5354a78e8a75ad8fb767081af5"} Nov 03 13:35:24.608311 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:24.607848 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-c47670-54c8bb9b89-fzdnn" event={"ID":"7a30a13a-965d-4291-9acd-f109a6ead710","Type":"ContainerStarted","Data":"12b72fc0ff14678f253add4dce17d31b74fc172c0d1bdcc40ead75ecb6cd79f9"} Nov 03 13:35:24.608311 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:24.607876 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-c47670-54c8bb9b89-fzdnn" Nov 03 13:35:24.633647 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:24.633593 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-c47670-54c8bb9b89-fzdnn" podStartSLOduration=1.633576245 podStartE2EDuration="1.633576245s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:35:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:35:24.629374834 +0000 UTC m=+1604.637566112" watchObservedRunningTime="2026-11-03 13:35:24.633576245 +0000 UTC m=+1604.641767522" Nov 03 13:35:27.189409 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:27.182535 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-1ec4b892-6b49f5f675-hxqrb" Nov 03 13:35:33.593450 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:33.593383 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w2-a724a6/sub-test-model-a724a6-kserve-57d7c88d9f-5nk9m"] Nov 03 13:35:33.832512 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:33.832476 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-a724a6/sub-test-model-a724a6-kserve-57d7c88d9f-5nk9m"] Nov 03 13:35:33.832864 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:33.832841 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-a724a6/sub-test-model-a724a6-kserve-57d7c88d9f-5nk9m" Nov 03 13:35:33.856599 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:33.856567 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-a724a6\"/\"openshift-service-ca.crt\"" Nov 03 13:35:33.857090 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:33.857065 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-a724a6\"/\"sub-test-model-a724a6-kserve-self-signed-certs\"" Nov 03 13:35:33.857370 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:33.857355 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-a724a6\"/\"default-dockercfg-x2kkq\"" Nov 03 13:35:33.857569 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:33.857553 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-a724a6\"/\"kube-root-ca.crt\"" Nov 03 13:35:33.860113 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:33.860080 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-08111b/premium-w1-08111b-kserve-57555d598-dx9g5" podUID="e45c569a-243f-49ab-8d74-dd08cbb6127c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.134:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 13:35:33.951645 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:33.951612 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3e6246a7-8c0e-4712-8f67-d834948a5673-dshm\") pod \"sub-test-model-a724a6-kserve-57d7c88d9f-5nk9m\" (UID: \"3e6246a7-8c0e-4712-8f67-d834948a5673\") " pod="ai-tenant-e2e-shared-a-w2-a724a6/sub-test-model-a724a6-kserve-57d7c88d9f-5nk9m" Nov 03 13:35:33.952132 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:33.951997 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3e6246a7-8c0e-4712-8f67-d834948a5673-tls-certs\") pod \"sub-test-model-a724a6-kserve-57d7c88d9f-5nk9m\" (UID: \"3e6246a7-8c0e-4712-8f67-d834948a5673\") " pod="ai-tenant-e2e-shared-a-w2-a724a6/sub-test-model-a724a6-kserve-57d7c88d9f-5nk9m" Nov 03 13:35:33.952479 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:33.952458 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3e6246a7-8c0e-4712-8f67-d834948a5673-tmp-dir\") pod \"sub-test-model-a724a6-kserve-57d7c88d9f-5nk9m\" (UID: \"3e6246a7-8c0e-4712-8f67-d834948a5673\") " pod="ai-tenant-e2e-shared-a-w2-a724a6/sub-test-model-a724a6-kserve-57d7c88d9f-5nk9m" Nov 03 13:35:33.952904 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:33.952873 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zs84z\" (UniqueName: \"kubernetes.io/projected/3e6246a7-8c0e-4712-8f67-d834948a5673-kube-api-access-zs84z\") pod \"sub-test-model-a724a6-kserve-57d7c88d9f-5nk9m\" (UID: \"3e6246a7-8c0e-4712-8f67-d834948a5673\") " pod="ai-tenant-e2e-shared-a-w2-a724a6/sub-test-model-a724a6-kserve-57d7c88d9f-5nk9m" Nov 03 13:35:33.953205 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:33.953175 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3e6246a7-8c0e-4712-8f67-d834948a5673-home\") pod \"sub-test-model-a724a6-kserve-57d7c88d9f-5nk9m\" (UID: \"3e6246a7-8c0e-4712-8f67-d834948a5673\") " pod="ai-tenant-e2e-shared-a-w2-a724a6/sub-test-model-a724a6-kserve-57d7c88d9f-5nk9m" Nov 03 13:35:33.953403 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:33.953381 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3e6246a7-8c0e-4712-8f67-d834948a5673-model-cache\") pod \"sub-test-model-a724a6-kserve-57d7c88d9f-5nk9m\" (UID: \"3e6246a7-8c0e-4712-8f67-d834948a5673\") " pod="ai-tenant-e2e-shared-a-w2-a724a6/sub-test-model-a724a6-kserve-57d7c88d9f-5nk9m" Nov 03 13:35:33.975835 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:33.975790 5024 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-vrbqc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 13:35:33.976165 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:33.976129 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-vrbqc" podUID="d12a0b88-93e3-4542-90b0-56a37dd91387" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 13:35:34.083312 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:34.080330 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3e6246a7-8c0e-4712-8f67-d834948a5673-dshm\") pod \"sub-test-model-a724a6-kserve-57d7c88d9f-5nk9m\" (UID: \"3e6246a7-8c0e-4712-8f67-d834948a5673\") " pod="ai-tenant-e2e-shared-a-w2-a724a6/sub-test-model-a724a6-kserve-57d7c88d9f-5nk9m" Nov 03 13:35:34.083312 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:34.080409 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3e6246a7-8c0e-4712-8f67-d834948a5673-tls-certs\") pod \"sub-test-model-a724a6-kserve-57d7c88d9f-5nk9m\" (UID: \"3e6246a7-8c0e-4712-8f67-d834948a5673\") " pod="ai-tenant-e2e-shared-a-w2-a724a6/sub-test-model-a724a6-kserve-57d7c88d9f-5nk9m" Nov 03 13:35:34.083312 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:34.080455 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3e6246a7-8c0e-4712-8f67-d834948a5673-tmp-dir\") pod \"sub-test-model-a724a6-kserve-57d7c88d9f-5nk9m\" (UID: \"3e6246a7-8c0e-4712-8f67-d834948a5673\") " pod="ai-tenant-e2e-shared-a-w2-a724a6/sub-test-model-a724a6-kserve-57d7c88d9f-5nk9m" Nov 03 13:35:34.083312 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:34.080659 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zs84z\" (UniqueName: \"kubernetes.io/projected/3e6246a7-8c0e-4712-8f67-d834948a5673-kube-api-access-zs84z\") pod \"sub-test-model-a724a6-kserve-57d7c88d9f-5nk9m\" (UID: \"3e6246a7-8c0e-4712-8f67-d834948a5673\") " pod="ai-tenant-e2e-shared-a-w2-a724a6/sub-test-model-a724a6-kserve-57d7c88d9f-5nk9m" Nov 03 13:35:34.083312 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:34.080773 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3e6246a7-8c0e-4712-8f67-d834948a5673-home\") pod \"sub-test-model-a724a6-kserve-57d7c88d9f-5nk9m\" (UID: \"3e6246a7-8c0e-4712-8f67-d834948a5673\") " pod="ai-tenant-e2e-shared-a-w2-a724a6/sub-test-model-a724a6-kserve-57d7c88d9f-5nk9m" Nov 03 13:35:34.083312 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:34.080821 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3e6246a7-8c0e-4712-8f67-d834948a5673-model-cache\") pod \"sub-test-model-a724a6-kserve-57d7c88d9f-5nk9m\" (UID: \"3e6246a7-8c0e-4712-8f67-d834948a5673\") " pod="ai-tenant-e2e-shared-a-w2-a724a6/sub-test-model-a724a6-kserve-57d7c88d9f-5nk9m" Nov 03 13:35:34.083312 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:34.081690 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3e6246a7-8c0e-4712-8f67-d834948a5673-tmp-dir\") pod \"sub-test-model-a724a6-kserve-57d7c88d9f-5nk9m\" (UID: \"3e6246a7-8c0e-4712-8f67-d834948a5673\") " pod="ai-tenant-e2e-shared-a-w2-a724a6/sub-test-model-a724a6-kserve-57d7c88d9f-5nk9m" Nov 03 13:35:34.083312 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:34.082927 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3e6246a7-8c0e-4712-8f67-d834948a5673-home\") pod \"sub-test-model-a724a6-kserve-57d7c88d9f-5nk9m\" (UID: \"3e6246a7-8c0e-4712-8f67-d834948a5673\") " pod="ai-tenant-e2e-shared-a-w2-a724a6/sub-test-model-a724a6-kserve-57d7c88d9f-5nk9m" Nov 03 13:35:34.084684 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:34.084655 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3e6246a7-8c0e-4712-8f67-d834948a5673-model-cache\") pod \"sub-test-model-a724a6-kserve-57d7c88d9f-5nk9m\" (UID: \"3e6246a7-8c0e-4712-8f67-d834948a5673\") " pod="ai-tenant-e2e-shared-a-w2-a724a6/sub-test-model-a724a6-kserve-57d7c88d9f-5nk9m" Nov 03 13:35:34.134369 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:34.132793 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3e6246a7-8c0e-4712-8f67-d834948a5673-tls-certs\") pod \"sub-test-model-a724a6-kserve-57d7c88d9f-5nk9m\" (UID: \"3e6246a7-8c0e-4712-8f67-d834948a5673\") " pod="ai-tenant-e2e-shared-a-w2-a724a6/sub-test-model-a724a6-kserve-57d7c88d9f-5nk9m" Nov 03 13:35:34.140763 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:34.140684 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zs84z\" (UniqueName: \"kubernetes.io/projected/3e6246a7-8c0e-4712-8f67-d834948a5673-kube-api-access-zs84z\") pod \"sub-test-model-a724a6-kserve-57d7c88d9f-5nk9m\" (UID: \"3e6246a7-8c0e-4712-8f67-d834948a5673\") " pod="ai-tenant-e2e-shared-a-w2-a724a6/sub-test-model-a724a6-kserve-57d7c88d9f-5nk9m" Nov 03 13:35:34.143443 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:34.143405 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3e6246a7-8c0e-4712-8f67-d834948a5673-dshm\") pod \"sub-test-model-a724a6-kserve-57d7c88d9f-5nk9m\" (UID: \"3e6246a7-8c0e-4712-8f67-d834948a5673\") " pod="ai-tenant-e2e-shared-a-w2-a724a6/sub-test-model-a724a6-kserve-57d7c88d9f-5nk9m" Nov 03 13:35:34.288328 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:34.275202 5024 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-mt-1ec4b892/e2e-lifecycle-model-1ec4b892-kserve-5b96cbdb9c-rl92k" podUID="b1a1aac6-a092-4640-b64d-6b62b12163d2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.146:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 13:35:34.301454 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:34.301402 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-a724a6/sub-test-model-a724a6-kserve-57d7c88d9f-5nk9m" Nov 03 13:35:34.670322 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:34.669880 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-08111b/simulator-w1-08111b-kserve-7f65c5867-btf5b" podUID="e8aba665-e4cc-4e67-8cdd-d09ef7542e15" containerName="main" probeResult="failure" output="Get \"https://10.132.0.133:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 13:35:34.772369 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:34.771536 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d3ca48/premium-w3-d3ca48-kserve-75ffc959b-zvjbc" podUID="add1e8b9-e7fd-4e4d-8d10-f97d0e2612ba" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded" Nov 03 13:35:34.908312 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:34.906440 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-08111b/premium-w1-08111b-kserve-57555d598-dx9g5" podUID="e45c569a-243f-49ab-8d74-dd08cbb6127c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.134:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 13:35:34.908312 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:34.906869 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-08111b/premium-w1-08111b-kserve-57555d598-dx9g5" Nov 03 13:35:35.771721 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:35.753467 5024 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-08111b/premium-w1-08111b-kserve-57555d598-dx9g5" podUID="e45c569a-243f-49ab-8d74-dd08cbb6127c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.134:8000/health\": context deadline exceeded" Nov 03 13:35:35.912365 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:35.897676 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-08111b/simulator-w1-08111b-kserve-7f65c5867-btf5b" podUID="e8aba665-e4cc-4e67-8cdd-d09ef7542e15" containerName="main" probeResult="failure" output="Get \"https://10.132.0.133:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 13:35:35.967329 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:35.965550 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-08111b/simulator-w1-08111b-kserve-7f65c5867-btf5b" Nov 03 13:35:36.087364 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:36.079991 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-08111b/premium-w1-08111b-kserve-57555d598-dx9g5" podUID="e45c569a-243f-49ab-8d74-dd08cbb6127c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.134:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 13:35:36.756435 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:36.756402 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-08111b/simulator-w1-08111b-kserve-7f65c5867-btf5b" Nov 03 13:35:36.756665 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:36.756648 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-a724a6/sub-test-model-a724a6-kserve-57d7c88d9f-5nk9m"] Nov 03 13:35:36.758087 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:36.758064 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-c47670-54c8bb9b89-fzdnn" Nov 03 13:35:36.791354 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:36.783510 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-1ec4b892/e2e-lifecycle-model-1ec4b892-kserve-5b96cbdb9c-rl92k" podUID="b1a1aac6-a092-4640-b64d-6b62b12163d2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.146:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 13:35:36.952702 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:36.951925 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-08111b/premium-w1-08111b-kserve-57555d598-dx9g5" Nov 03 13:35:37.648551 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:37.648500 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-a724a6/sub-test-model-a724a6-kserve-57d7c88d9f-5nk9m" event={"ID":"3e6246a7-8c0e-4712-8f67-d834948a5673","Type":"ContainerStarted","Data":"7cc8141653a4d544c0d8386c404e53b50a74307690e6e671fbf43dec102ecba3"} Nov 03 13:35:37.648551 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:37.648558 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-a724a6/sub-test-model-a724a6-kserve-57d7c88d9f-5nk9m" event={"ID":"3e6246a7-8c0e-4712-8f67-d834948a5673","Type":"ContainerStarted","Data":"ccc6fc1870df4c8aff637321973ff615fe3d32238a16ae27748fc857316666a3"} Nov 03 13:35:37.724323 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:37.723949 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w2-a724a6/sub-test-model-a724a6-kserve-57d7c88d9f-5nk9m" podStartSLOduration=4.723925465 podStartE2EDuration="4.723925465s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:35:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:35:37.70657067 +0000 UTC m=+1617.714761948" watchObservedRunningTime="2026-11-03 13:35:37.723925465 +0000 UTC m=+1617.732116743" Nov 03 13:35:38.303427 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:38.303383 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w2-a724a6/sub-test-model-a724a6-kserve-57d7c88d9f-5nk9m" Nov 03 13:35:39.661485 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:35:39.661437 5024 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d7mwz" containerName="extractor" Nov 03 13:35:44.133962 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:44.133923 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-7e439d04-openshift-default-6ff95548d9-nh4cq"] Nov 03 13:35:44.170178 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:44.170143 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-7e439d04-openshift-default-6ff95548d9-nh4cq"] Nov 03 13:35:44.170406 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:44.170256 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-7e439d04-openshift-default-6ff95548d9-nh4cq" Nov 03 13:35:44.196978 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:44.188913 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-7e439d04-openshift-default-dockercfg-sskjs\"" Nov 03 13:35:44.304324 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:44.271849 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d803a251-b682-4e3e-b636-92fbe8d99c64-workload-socket\") pod \"e2e-mt-7e439d04-openshift-default-6ff95548d9-nh4cq\" (UID: \"d803a251-b682-4e3e-b636-92fbe8d99c64\") " pod="openshift-ingress/e2e-mt-7e439d04-openshift-default-6ff95548d9-nh4cq" Nov 03 13:35:44.304324 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:44.271933 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d803a251-b682-4e3e-b636-92fbe8d99c64-credential-socket\") pod \"e2e-mt-7e439d04-openshift-default-6ff95548d9-nh4cq\" (UID: \"d803a251-b682-4e3e-b636-92fbe8d99c64\") " pod="openshift-ingress/e2e-mt-7e439d04-openshift-default-6ff95548d9-nh4cq" Nov 03 13:35:44.304324 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:44.271988 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tmmc7\" (UniqueName: \"kubernetes.io/projected/d803a251-b682-4e3e-b636-92fbe8d99c64-kube-api-access-tmmc7\") pod \"e2e-mt-7e439d04-openshift-default-6ff95548d9-nh4cq\" (UID: \"d803a251-b682-4e3e-b636-92fbe8d99c64\") " pod="openshift-ingress/e2e-mt-7e439d04-openshift-default-6ff95548d9-nh4cq" Nov 03 13:35:44.304324 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:44.272047 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d803a251-b682-4e3e-b636-92fbe8d99c64-istio-token\") pod \"e2e-mt-7e439d04-openshift-default-6ff95548d9-nh4cq\" (UID: \"d803a251-b682-4e3e-b636-92fbe8d99c64\") " pod="openshift-ingress/e2e-mt-7e439d04-openshift-default-6ff95548d9-nh4cq" Nov 03 13:35:44.304324 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:44.272134 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d803a251-b682-4e3e-b636-92fbe8d99c64-workload-certs\") pod \"e2e-mt-7e439d04-openshift-default-6ff95548d9-nh4cq\" (UID: \"d803a251-b682-4e3e-b636-92fbe8d99c64\") " pod="openshift-ingress/e2e-mt-7e439d04-openshift-default-6ff95548d9-nh4cq" Nov 03 13:35:44.304324 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:44.272211 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d803a251-b682-4e3e-b636-92fbe8d99c64-istio-data\") pod \"e2e-mt-7e439d04-openshift-default-6ff95548d9-nh4cq\" (UID: \"d803a251-b682-4e3e-b636-92fbe8d99c64\") " pod="openshift-ingress/e2e-mt-7e439d04-openshift-default-6ff95548d9-nh4cq" Nov 03 13:35:44.304324 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:44.272352 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d803a251-b682-4e3e-b636-92fbe8d99c64-istiod-ca-cert\") pod \"e2e-mt-7e439d04-openshift-default-6ff95548d9-nh4cq\" (UID: \"d803a251-b682-4e3e-b636-92fbe8d99c64\") " pod="openshift-ingress/e2e-mt-7e439d04-openshift-default-6ff95548d9-nh4cq" Nov 03 13:35:44.304324 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:44.272441 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d803a251-b682-4e3e-b636-92fbe8d99c64-istio-podinfo\") pod \"e2e-mt-7e439d04-openshift-default-6ff95548d9-nh4cq\" (UID: \"d803a251-b682-4e3e-b636-92fbe8d99c64\") " pod="openshift-ingress/e2e-mt-7e439d04-openshift-default-6ff95548d9-nh4cq" Nov 03 13:35:44.304324 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:44.272468 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d803a251-b682-4e3e-b636-92fbe8d99c64-istio-envoy\") pod \"e2e-mt-7e439d04-openshift-default-6ff95548d9-nh4cq\" (UID: \"d803a251-b682-4e3e-b636-92fbe8d99c64\") " pod="openshift-ingress/e2e-mt-7e439d04-openshift-default-6ff95548d9-nh4cq" Nov 03 13:35:44.321971 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:44.307842 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w2-a724a6/sub-test-model-a724a6-kserve-57d7c88d9f-5nk9m" Nov 03 13:35:44.402983 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:44.402747 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d803a251-b682-4e3e-b636-92fbe8d99c64-istio-envoy\") pod \"e2e-mt-7e439d04-openshift-default-6ff95548d9-nh4cq\" (UID: \"d803a251-b682-4e3e-b636-92fbe8d99c64\") " pod="openshift-ingress/e2e-mt-7e439d04-openshift-default-6ff95548d9-nh4cq" Nov 03 13:35:44.407542 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:44.407509 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d803a251-b682-4e3e-b636-92fbe8d99c64-workload-socket\") pod \"e2e-mt-7e439d04-openshift-default-6ff95548d9-nh4cq\" (UID: \"d803a251-b682-4e3e-b636-92fbe8d99c64\") " pod="openshift-ingress/e2e-mt-7e439d04-openshift-default-6ff95548d9-nh4cq" Nov 03 13:35:44.407826 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:44.407810 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d803a251-b682-4e3e-b636-92fbe8d99c64-credential-socket\") pod \"e2e-mt-7e439d04-openshift-default-6ff95548d9-nh4cq\" (UID: \"d803a251-b682-4e3e-b636-92fbe8d99c64\") " pod="openshift-ingress/e2e-mt-7e439d04-openshift-default-6ff95548d9-nh4cq" Nov 03 13:35:44.408014 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:44.407997 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmmc7\" (UniqueName: \"kubernetes.io/projected/d803a251-b682-4e3e-b636-92fbe8d99c64-kube-api-access-tmmc7\") pod \"e2e-mt-7e439d04-openshift-default-6ff95548d9-nh4cq\" (UID: \"d803a251-b682-4e3e-b636-92fbe8d99c64\") " pod="openshift-ingress/e2e-mt-7e439d04-openshift-default-6ff95548d9-nh4cq" Nov 03 13:35:44.408218 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:44.408203 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d803a251-b682-4e3e-b636-92fbe8d99c64-istio-token\") pod \"e2e-mt-7e439d04-openshift-default-6ff95548d9-nh4cq\" (UID: \"d803a251-b682-4e3e-b636-92fbe8d99c64\") " pod="openshift-ingress/e2e-mt-7e439d04-openshift-default-6ff95548d9-nh4cq" Nov 03 13:35:44.408484 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:44.408467 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d803a251-b682-4e3e-b636-92fbe8d99c64-workload-certs\") pod \"e2e-mt-7e439d04-openshift-default-6ff95548d9-nh4cq\" (UID: \"d803a251-b682-4e3e-b636-92fbe8d99c64\") " pod="openshift-ingress/e2e-mt-7e439d04-openshift-default-6ff95548d9-nh4cq" Nov 03 13:35:44.408788 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:44.408771 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d803a251-b682-4e3e-b636-92fbe8d99c64-istio-data\") pod \"e2e-mt-7e439d04-openshift-default-6ff95548d9-nh4cq\" (UID: \"d803a251-b682-4e3e-b636-92fbe8d99c64\") " pod="openshift-ingress/e2e-mt-7e439d04-openshift-default-6ff95548d9-nh4cq" Nov 03 13:35:44.409069 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:44.409054 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d803a251-b682-4e3e-b636-92fbe8d99c64-istiod-ca-cert\") pod \"e2e-mt-7e439d04-openshift-default-6ff95548d9-nh4cq\" (UID: \"d803a251-b682-4e3e-b636-92fbe8d99c64\") " pod="openshift-ingress/e2e-mt-7e439d04-openshift-default-6ff95548d9-nh4cq" Nov 03 13:35:44.409409 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:44.409242 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d803a251-b682-4e3e-b636-92fbe8d99c64-istio-podinfo\") pod \"e2e-mt-7e439d04-openshift-default-6ff95548d9-nh4cq\" (UID: \"d803a251-b682-4e3e-b636-92fbe8d99c64\") " pod="openshift-ingress/e2e-mt-7e439d04-openshift-default-6ff95548d9-nh4cq" Nov 03 13:35:44.413393 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:44.406361 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d803a251-b682-4e3e-b636-92fbe8d99c64-istio-envoy\") pod \"e2e-mt-7e439d04-openshift-default-6ff95548d9-nh4cq\" (UID: \"d803a251-b682-4e3e-b636-92fbe8d99c64\") " pod="openshift-ingress/e2e-mt-7e439d04-openshift-default-6ff95548d9-nh4cq" Nov 03 13:35:44.413393 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:44.410812 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d803a251-b682-4e3e-b636-92fbe8d99c64-workload-socket\") pod \"e2e-mt-7e439d04-openshift-default-6ff95548d9-nh4cq\" (UID: \"d803a251-b682-4e3e-b636-92fbe8d99c64\") " pod="openshift-ingress/e2e-mt-7e439d04-openshift-default-6ff95548d9-nh4cq" Nov 03 13:35:44.413393 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:44.411051 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d803a251-b682-4e3e-b636-92fbe8d99c64-credential-socket\") pod \"e2e-mt-7e439d04-openshift-default-6ff95548d9-nh4cq\" (UID: \"d803a251-b682-4e3e-b636-92fbe8d99c64\") " pod="openshift-ingress/e2e-mt-7e439d04-openshift-default-6ff95548d9-nh4cq" Nov 03 13:35:44.413393 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:44.411620 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d803a251-b682-4e3e-b636-92fbe8d99c64-istio-data\") pod \"e2e-mt-7e439d04-openshift-default-6ff95548d9-nh4cq\" (UID: \"d803a251-b682-4e3e-b636-92fbe8d99c64\") " pod="openshift-ingress/e2e-mt-7e439d04-openshift-default-6ff95548d9-nh4cq" Nov 03 13:35:44.413393 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:44.412055 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d803a251-b682-4e3e-b636-92fbe8d99c64-workload-certs\") pod \"e2e-mt-7e439d04-openshift-default-6ff95548d9-nh4cq\" (UID: \"d803a251-b682-4e3e-b636-92fbe8d99c64\") " pod="openshift-ingress/e2e-mt-7e439d04-openshift-default-6ff95548d9-nh4cq" Nov 03 13:35:44.413840 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:44.413820 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d803a251-b682-4e3e-b636-92fbe8d99c64-istiod-ca-cert\") pod \"e2e-mt-7e439d04-openshift-default-6ff95548d9-nh4cq\" (UID: \"d803a251-b682-4e3e-b636-92fbe8d99c64\") " pod="openshift-ingress/e2e-mt-7e439d04-openshift-default-6ff95548d9-nh4cq" Nov 03 13:35:44.473389 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:44.464738 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmmc7\" (UniqueName: \"kubernetes.io/projected/d803a251-b682-4e3e-b636-92fbe8d99c64-kube-api-access-tmmc7\") pod \"e2e-mt-7e439d04-openshift-default-6ff95548d9-nh4cq\" (UID: \"d803a251-b682-4e3e-b636-92fbe8d99c64\") " pod="openshift-ingress/e2e-mt-7e439d04-openshift-default-6ff95548d9-nh4cq" Nov 03 13:35:44.482799 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:44.482357 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d803a251-b682-4e3e-b636-92fbe8d99c64-istio-token\") pod \"e2e-mt-7e439d04-openshift-default-6ff95548d9-nh4cq\" (UID: \"d803a251-b682-4e3e-b636-92fbe8d99c64\") " pod="openshift-ingress/e2e-mt-7e439d04-openshift-default-6ff95548d9-nh4cq" Nov 03 13:35:44.484262 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:44.483299 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d803a251-b682-4e3e-b636-92fbe8d99c64-istio-podinfo\") pod \"e2e-mt-7e439d04-openshift-default-6ff95548d9-nh4cq\" (UID: \"d803a251-b682-4e3e-b636-92fbe8d99c64\") " pod="openshift-ingress/e2e-mt-7e439d04-openshift-default-6ff95548d9-nh4cq" Nov 03 13:35:44.574557 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:44.573665 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w2-a724a6/sub-test-model-a724a6-kserve-57d7c88d9f-5nk9m" Nov 03 13:35:44.664081 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:44.663414 5024 scope.go:122] "RemoveContainer" containerID="d0901a47ddce81ec654fecc910a2c87fe5c16dd9d7b1d0f27056796f7e761cc7" Nov 03 13:35:44.776911 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:44.776873 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-7e439d04-openshift-default-6ff95548d9-nh4cq" Nov 03 13:35:44.969221 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:44.969138 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w2-a724a6/sub-test-model-a724a6-kserve-57d7c88d9f-5nk9m" Nov 03 13:35:45.243116 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:45.243087 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-7e439d04-openshift-default-6ff95548d9-nh4cq"] Nov 03 13:35:45.243668 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:35:45.243638 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd803a251_b682_4e3e_b636_92fbe8d99c64.slice/crio-da27de2a8641b82b03433b9c19996cab401abe49ed6fb8922b77c191921c9309 WatchSource:0}: Error finding container da27de2a8641b82b03433b9c19996cab401abe49ed6fb8922b77c191921c9309: Status 404 returned error can't find the container with id da27de2a8641b82b03433b9c19996cab401abe49ed6fb8922b77c191921c9309 Nov 03 13:35:45.246929 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:45.246889 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:35:45.247039 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:45.246964 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:35:45.247039 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:45.246994 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:35:45.934927 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:45.934882 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7e439d04-openshift-default-6ff95548d9-nh4cq" event={"ID":"d803a251-b682-4e3e-b636-92fbe8d99c64","Type":"ContainerStarted","Data":"4dec6fb29c63ccf03e05c3d8d8c49f7e33af7c51bae00ab63511a676b90329c8"} Nov 03 13:35:45.937017 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:45.934959 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7e439d04-openshift-default-6ff95548d9-nh4cq" event={"ID":"d803a251-b682-4e3e-b636-92fbe8d99c64","Type":"ContainerStarted","Data":"da27de2a8641b82b03433b9c19996cab401abe49ed6fb8922b77c191921c9309"} Nov 03 13:35:45.965767 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:45.963040 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-7e439d04-openshift-default-6ff95548d9-nh4cq" podStartSLOduration=1.9630216379999998 podStartE2EDuration="1.963021638s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:35:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:35:45.962873167 +0000 UTC m=+1625.971064470" watchObservedRunningTime="2026-11-03 13:35:45.963021638 +0000 UTC m=+1625.971212916" Nov 03 13:35:46.777756 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:46.777722 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-7e439d04-openshift-default-6ff95548d9-nh4cq" Nov 03 13:35:46.784978 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:46.784945 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-7e439d04-openshift-default-6ff95548d9-nh4cq" Nov 03 13:35:46.945172 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:46.945136 5024 generic.go:356] "Generic (PLEG): container finished" podID="0757da43-44aa-48e1-b169-00715a78ea02" containerID="33974b23654fbff102111f8a70f02b8e2a0ae24b9e456b780a29ae4c7684b063" exitCode=0 Nov 03 13:35:46.945365 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:46.945218 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-pznlh" event={"ID":"0757da43-44aa-48e1-b169-00715a78ea02","Type":"ContainerDied","Data":"33974b23654fbff102111f8a70f02b8e2a0ae24b9e456b780a29ae4c7684b063"} Nov 03 13:35:46.945836 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:46.945818 5024 scope.go:122] "RemoveContainer" containerID="33974b23654fbff102111f8a70f02b8e2a0ae24b9e456b780a29ae4c7684b063" Nov 03 13:35:46.945900 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:46.945877 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-7e439d04-openshift-default-6ff95548d9-nh4cq" Nov 03 13:35:46.947089 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:46.947065 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-7e439d04-openshift-default-6ff95548d9-nh4cq" Nov 03 13:35:46.947840 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:46.947811 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:35:46.947908 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:46.947882 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:35:46.947978 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:46.947915 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:35:47.954929 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:47.954890 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-pznlh" event={"ID":"0757da43-44aa-48e1-b169-00715a78ea02","Type":"ContainerStarted","Data":"1cd0f17dbeb3270cfe79f3934f9fcbc3b4612cdc3a703cfabbc47c47e6cc1815"} Nov 03 13:35:48.322957 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:48.322862 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-1ec4b892/e2e-lifecycle-model-1ec4b892-kserve-5b96cbdb9c-rl92k"] Nov 03 13:35:48.857849 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:48.857814 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-pznlh" Nov 03 13:35:48.858364 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:48.858336 5024 patch_prober.go:29] interesting pod/e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-pznlh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" start-of-body= Nov 03 13:35:48.858585 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:48.858556 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-pznlh" podUID="0757da43-44aa-48e1-b169-00715a78ea02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" Nov 03 13:35:49.854569 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:49.854516 5024 patch_prober.go:29] interesting pod/e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-pznlh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" start-of-body= Nov 03 13:35:49.855075 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:49.854604 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-pznlh" podUID="0757da43-44aa-48e1-b169-00715a78ea02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" Nov 03 13:35:50.854020 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:50.853976 5024 patch_prober.go:29] interesting pod/e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-pznlh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" start-of-body= Nov 03 13:35:50.854245 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:50.854053 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-pznlh" podUID="0757da43-44aa-48e1-b169-00715a78ea02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" Nov 03 13:35:51.000470 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:51.000430 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-pznlh" Nov 03 13:35:51.855455 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:51.855405 5024 patch_prober.go:29] interesting pod/e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-pznlh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" start-of-body= Nov 03 13:35:51.855715 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:51.855690 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-pznlh" podUID="0757da43-44aa-48e1-b169-00715a78ea02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" Nov 03 13:35:52.853756 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:52.853716 5024 patch_prober.go:29] interesting pod/e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-pznlh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" start-of-body= Nov 03 13:35:52.854404 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:52.854369 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-pznlh" podUID="0757da43-44aa-48e1-b169-00715a78ea02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" Nov 03 13:35:53.854902 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:53.854408 5024 patch_prober.go:29] interesting pod/e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-pznlh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" start-of-body= Nov 03 13:35:53.854902 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:53.854483 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-pznlh" podUID="0757da43-44aa-48e1-b169-00715a78ea02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" Nov 03 13:35:54.853906 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:54.853867 5024 patch_prober.go:29] interesting pod/e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-pznlh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" start-of-body= Nov 03 13:35:54.854114 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:54.853938 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-pznlh" podUID="0757da43-44aa-48e1-b169-00715a78ea02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" Nov 03 13:35:55.858750 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:55.858713 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-pznlh" Nov 03 13:35:55.859981 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:55.859956 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-pznlh" Nov 03 13:35:58.853058 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:58.853017 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-9f71f2-5b8844dc8c-6xt6h"] Nov 03 13:35:58.857910 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:58.857881 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-9f71f2-5b8844dc8c-6xt6h" Nov 03 13:35:58.868975 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:58.868946 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-9f71f2-5b8844dc8c-6xt6h"] Nov 03 13:35:58.934309 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:58.933794 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-9f71f2-dockercfg-5wn4s\"" Nov 03 13:35:58.934309 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:58.933903 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-9f71f2\"" Nov 03 13:35:58.946725 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:58.935817 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f47e8fd5-9a67-45ca-a817-05b9e20654cc-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-9f71f2-5b8844dc8c-6xt6h\" (UID: \"f47e8fd5-9a67-45ca-a817-05b9e20654cc\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-9f71f2-5b8844dc8c-6xt6h" Nov 03 13:35:58.946725 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:58.935862 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9hlqh\" (UniqueName: \"kubernetes.io/projected/f47e8fd5-9a67-45ca-a817-05b9e20654cc-kube-api-access-9hlqh\") pod \"payload-processing-e2e-extmod-b-9f71f2-5b8844dc8c-6xt6h\" (UID: \"f47e8fd5-9a67-45ca-a817-05b9e20654cc\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-9f71f2-5b8844dc8c-6xt6h" Nov 03 13:35:59.244420 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:59.226013 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f47e8fd5-9a67-45ca-a817-05b9e20654cc-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-9f71f2-5b8844dc8c-6xt6h\" (UID: \"f47e8fd5-9a67-45ca-a817-05b9e20654cc\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-9f71f2-5b8844dc8c-6xt6h" Nov 03 13:35:59.244420 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:59.226069 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hlqh\" (UniqueName: \"kubernetes.io/projected/f47e8fd5-9a67-45ca-a817-05b9e20654cc-kube-api-access-9hlqh\") pod \"payload-processing-e2e-extmod-b-9f71f2-5b8844dc8c-6xt6h\" (UID: \"f47e8fd5-9a67-45ca-a817-05b9e20654cc\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-9f71f2-5b8844dc8c-6xt6h" Nov 03 13:35:59.246830 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:59.246793 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f47e8fd5-9a67-45ca-a817-05b9e20654cc-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-9f71f2-5b8844dc8c-6xt6h\" (UID: \"f47e8fd5-9a67-45ca-a817-05b9e20654cc\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-9f71f2-5b8844dc8c-6xt6h" Nov 03 13:35:59.283413 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:59.271138 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9hlqh\" (UniqueName: \"kubernetes.io/projected/f47e8fd5-9a67-45ca-a817-05b9e20654cc-kube-api-access-9hlqh\") pod \"payload-processing-e2e-extmod-b-9f71f2-5b8844dc8c-6xt6h\" (UID: \"f47e8fd5-9a67-45ca-a817-05b9e20654cc\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-9f71f2-5b8844dc8c-6xt6h" Nov 03 13:35:59.511716 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:59.511622 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-9f71f2-5b8844dc8c-6xt6h" Nov 03 13:35:59.809979 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:35:59.809944 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf47e8fd5_9a67_45ca_a817_05b9e20654cc.slice/crio-c252fa015508deeb50fe885bcf8387d8010ac6995c34025e6315aa29c03c8291 WatchSource:0}: Error finding container c252fa015508deeb50fe885bcf8387d8010ac6995c34025e6315aa29c03c8291: Status 404 returned error can't find the container with id c252fa015508deeb50fe885bcf8387d8010ac6995c34025e6315aa29c03c8291 Nov 03 13:35:59.811836 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:35:59.811808 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-9f71f2-5b8844dc8c-6xt6h"] Nov 03 13:36:00.528577 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:36:00.526248 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-9f71f2-5b8844dc8c-6xt6h" event={"ID":"f47e8fd5-9a67-45ca-a817-05b9e20654cc","Type":"ContainerStarted","Data":"9e5f2654511b7a99377d16600dae81be35cfb95d749a69bcf9a601ca0d177ac3"} Nov 03 13:36:00.528577 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:36:00.526340 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-9f71f2-5b8844dc8c-6xt6h" event={"ID":"f47e8fd5-9a67-45ca-a817-05b9e20654cc","Type":"ContainerStarted","Data":"c252fa015508deeb50fe885bcf8387d8010ac6995c34025e6315aa29c03c8291"} Nov 03 13:36:00.528577 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:36:00.527436 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-9f71f2-5b8844dc8c-6xt6h" Nov 03 13:36:00.564056 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:36:00.563990 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-9f71f2-5b8844dc8c-6xt6h" podStartSLOduration=2.563968487 podStartE2EDuration="2.563968487s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:35:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:36:00.56219531 +0000 UTC m=+1640.570386589" watchObservedRunningTime="2026-11-03 13:36:00.563968487 +0000 UTC m=+1640.572159765" Nov 03 13:36:03.346385 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:36:03.346338 5024 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-mt-1ec4b892/e2e-lifecycle-model-1ec4b892-kserve-5b96cbdb9c-rl92k" podUID="b1a1aac6-a092-4640-b64d-6b62b12163d2" containerName="main" containerID="cri-o://ed935b179655f78e8113be171468f27d20e1cde0a0327b370d200890690bf32d" gracePeriod=45 Nov 03 13:36:03.510055 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:36:03.510015 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-1ec4b892/e2e-lifecycle-model-1ec4b892-kserve-5b96cbdb9c-rl92k" podUID="b1a1aac6-a092-4640-b64d-6b62b12163d2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.146:8000/ready\": dial tcp 10.132.0.146:8000: connect: connection refused" Nov 03 13:36:03.642813 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:36:03.642776 5024 generic.go:356] "Generic (PLEG): container finished" podID="b1a1aac6-a092-4640-b64d-6b62b12163d2" containerID="ed935b179655f78e8113be171468f27d20e1cde0a0327b370d200890690bf32d" exitCode=0 Nov 03 13:36:03.643415 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:36:03.643094 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-1ec4b892/e2e-lifecycle-model-1ec4b892-kserve-5b96cbdb9c-rl92k" event={"ID":"b1a1aac6-a092-4640-b64d-6b62b12163d2","Type":"ContainerDied","Data":"ed935b179655f78e8113be171468f27d20e1cde0a0327b370d200890690bf32d"} Nov 03 13:36:03.801313 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:36:03.781536 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-1ec4b892/e2e-lifecycle-model-1ec4b892-kserve-5b96cbdb9c-rl92k" Nov 03 13:36:03.948134 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:36:03.948041 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b1a1aac6-a092-4640-b64d-6b62b12163d2-kube-api-access-zkwg8\" (UniqueName: \"kubernetes.io/projected/b1a1aac6-a092-4640-b64d-6b62b12163d2-kube-api-access-zkwg8\") pod \"b1a1aac6-a092-4640-b64d-6b62b12163d2\" (UID: \"b1a1aac6-a092-4640-b64d-6b62b12163d2\") " Nov 03 13:36:03.948134 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:36:03.948121 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b1a1aac6-a092-4640-b64d-6b62b12163d2-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b1a1aac6-a092-4640-b64d-6b62b12163d2-dshm\") pod \"b1a1aac6-a092-4640-b64d-6b62b12163d2\" (UID: \"b1a1aac6-a092-4640-b64d-6b62b12163d2\") " Nov 03 13:36:03.948413 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:36:03.948204 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b1a1aac6-a092-4640-b64d-6b62b12163d2-tls-certs\" (UniqueName: \"kubernetes.io/secret/b1a1aac6-a092-4640-b64d-6b62b12163d2-tls-certs\") pod \"b1a1aac6-a092-4640-b64d-6b62b12163d2\" (UID: \"b1a1aac6-a092-4640-b64d-6b62b12163d2\") " Nov 03 13:36:03.948465 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:36:03.948409 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b1a1aac6-a092-4640-b64d-6b62b12163d2-home\" (UniqueName: \"kubernetes.io/empty-dir/b1a1aac6-a092-4640-b64d-6b62b12163d2-home\") pod \"b1a1aac6-a092-4640-b64d-6b62b12163d2\" (UID: \"b1a1aac6-a092-4640-b64d-6b62b12163d2\") " Nov 03 13:36:03.948561 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:36:03.948545 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b1a1aac6-a092-4640-b64d-6b62b12163d2-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1a1aac6-a092-4640-b64d-6b62b12163d2-tmp-dir\") pod \"b1a1aac6-a092-4640-b64d-6b62b12163d2\" (UID: \"b1a1aac6-a092-4640-b64d-6b62b12163d2\") " Nov 03 13:36:03.948652 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:36:03.948637 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b1a1aac6-a092-4640-b64d-6b62b12163d2-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b1a1aac6-a092-4640-b64d-6b62b12163d2-model-cache\") pod \"b1a1aac6-a092-4640-b64d-6b62b12163d2\" (UID: \"b1a1aac6-a092-4640-b64d-6b62b12163d2\") " Nov 03 13:36:03.949628 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:36:03.949595 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1a1aac6-a092-4640-b64d-6b62b12163d2-model-cache" pod "b1a1aac6-a092-4640-b64d-6b62b12163d2" (UID: "b1a1aac6-a092-4640-b64d-6b62b12163d2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:36:03.950369 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:36:03.950337 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1a1aac6-a092-4640-b64d-6b62b12163d2-tmp-dir" pod "b1a1aac6-a092-4640-b64d-6b62b12163d2" (UID: "b1a1aac6-a092-4640-b64d-6b62b12163d2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:36:03.950537 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:36:03.950511 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1a1aac6-a092-4640-b64d-6b62b12163d2-home" pod "b1a1aac6-a092-4640-b64d-6b62b12163d2" (UID: "b1a1aac6-a092-4640-b64d-6b62b12163d2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:36:03.965270 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:36:03.959897 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1a1aac6-a092-4640-b64d-6b62b12163d2-dshm" pod "b1a1aac6-a092-4640-b64d-6b62b12163d2" (UID: "b1a1aac6-a092-4640-b64d-6b62b12163d2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:36:03.965270 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:36:03.964693 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1a1aac6-a092-4640-b64d-6b62b12163d2-tls-certs" pod "b1a1aac6-a092-4640-b64d-6b62b12163d2" (UID: "b1a1aac6-a092-4640-b64d-6b62b12163d2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 13:36:03.965270 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:36:03.964875 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1a1aac6-a092-4640-b64d-6b62b12163d2-kube-api-access-zkwg8" pod "b1a1aac6-a092-4640-b64d-6b62b12163d2" (UID: "b1a1aac6-a092-4640-b64d-6b62b12163d2"). InnerVolumeSpecName "kube-api-access-zkwg8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:36:04.055378 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:36:04.051624 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-7e439d04-ffd9b45f6-lt8m6"] Nov 03 13:36:04.055378 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:36:04.052870 5024 cpu_manager.go:414] "containerMap: removing container" podUID="b1a1aac6-a092-4640-b64d-6b62b12163d2" containerName="main" Nov 03 13:36:04.055378 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:36:04.052894 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1a1aac6-a092-4640-b64d-6b62b12163d2" containerName="main" Nov 03 13:36:04.055378 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:36:04.053132 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="b1a1aac6-a092-4640-b64d-6b62b12163d2" containerName="main" Nov 03 13:36:04.058643 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:36:04.058601 5024 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b1a1aac6-a092-4640-b64d-6b62b12163d2-model-cache\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:36:04.058876 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:36:04.058856 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zkwg8\" (UniqueName: \"kubernetes.io/projected/b1a1aac6-a092-4640-b64d-6b62b12163d2-kube-api-access-zkwg8\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:36:04.059000 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:36:04.058987 5024 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b1a1aac6-a092-4640-b64d-6b62b12163d2-dshm\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:36:04.059141 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:36:04.059126 5024 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b1a1aac6-a092-4640-b64d-6b62b12163d2-tls-certs\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:36:04.059253 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:36:04.059242 5024 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b1a1aac6-a092-4640-b64d-6b62b12163d2-home\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:36:04.059687 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:36:04.059664 5024 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1a1aac6-a092-4640-b64d-6b62b12163d2-tmp-dir\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:36:04.064660 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:36:04.061028 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7e439d04-ffd9b45f6-lt8m6" Nov 03 13:36:04.067329 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:36:04.067263 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-7e439d04-ffd9b45f6-lt8m6"] Nov 03 13:36:04.069380 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:36:04.068552 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-7e439d04-serving-cert\"" Nov 03 13:36:04.165306 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:36:04.164596 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zf7cg\" (UniqueName: \"kubernetes.io/projected/cfb86542-1303-482c-a6ce-bee9bc81da27-kube-api-access-zf7cg\") pod \"maas-api-e2e-mt-7e439d04-ffd9b45f6-lt8m6\" (UID: \"cfb86542-1303-482c-a6ce-bee9bc81da27\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7e439d04-ffd9b45f6-lt8m6" Nov 03 13:36:04.196308 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:36:04.189566 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cfb86542-1303-482c-a6ce-bee9bc81da27-metrics-certs\") pod \"maas-api-e2e-mt-7e439d04-ffd9b45f6-lt8m6\" (UID: \"cfb86542-1303-482c-a6ce-bee9bc81da27\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7e439d04-ffd9b45f6-lt8m6" Nov 03 13:36:04.196308 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:36:04.189677 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cfb86542-1303-482c-a6ce-bee9bc81da27-openshift-service-ca\") pod \"maas-api-e2e-mt-7e439d04-ffd9b45f6-lt8m6\" (UID: \"cfb86542-1303-482c-a6ce-bee9bc81da27\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7e439d04-ffd9b45f6-lt8m6" Nov 03 13:36:04.196308 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:36:04.190009 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cfb86542-1303-482c-a6ce-bee9bc81da27-maas-api-tls\") pod \"maas-api-e2e-mt-7e439d04-ffd9b45f6-lt8m6\" (UID: \"cfb86542-1303-482c-a6ce-bee9bc81da27\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7e439d04-ffd9b45f6-lt8m6" Nov 03 13:36:04.344454 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:36:04.337753 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cfb86542-1303-482c-a6ce-bee9bc81da27-maas-api-tls\") pod \"maas-api-e2e-mt-7e439d04-ffd9b45f6-lt8m6\" (UID: \"cfb86542-1303-482c-a6ce-bee9bc81da27\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7e439d04-ffd9b45f6-lt8m6" Nov 03 13:36:04.344454 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:36:04.337902 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zf7cg\" (UniqueName: \"kubernetes.io/projected/cfb86542-1303-482c-a6ce-bee9bc81da27-kube-api-access-zf7cg\") pod \"maas-api-e2e-mt-7e439d04-ffd9b45f6-lt8m6\" (UID: \"cfb86542-1303-482c-a6ce-bee9bc81da27\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7e439d04-ffd9b45f6-lt8m6" Nov 03 13:36:04.344454 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:36:04.338034 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cfb86542-1303-482c-a6ce-bee9bc81da27-metrics-certs\") pod \"maas-api-e2e-mt-7e439d04-ffd9b45f6-lt8m6\" (UID: \"cfb86542-1303-482c-a6ce-bee9bc81da27\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7e439d04-ffd9b45f6-lt8m6" Nov 03 13:36:04.344454 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:36:04.338102 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cfb86542-1303-482c-a6ce-bee9bc81da27-openshift-service-ca\") pod \"maas-api-e2e-mt-7e439d04-ffd9b45f6-lt8m6\" (UID: \"cfb86542-1303-482c-a6ce-bee9bc81da27\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7e439d04-ffd9b45f6-lt8m6" Nov 03 13:36:04.376385 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:36:04.361465 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cfb86542-1303-482c-a6ce-bee9bc81da27-openshift-service-ca\") pod \"maas-api-e2e-mt-7e439d04-ffd9b45f6-lt8m6\" (UID: \"cfb86542-1303-482c-a6ce-bee9bc81da27\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7e439d04-ffd9b45f6-lt8m6" Nov 03 13:36:04.401318 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:36:04.394051 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zf7cg\" (UniqueName: \"kubernetes.io/projected/cfb86542-1303-482c-a6ce-bee9bc81da27-kube-api-access-zf7cg\") pod \"maas-api-e2e-mt-7e439d04-ffd9b45f6-lt8m6\" (UID: \"cfb86542-1303-482c-a6ce-bee9bc81da27\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7e439d04-ffd9b45f6-lt8m6" Nov 03 13:36:04.401318 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:36:04.394251 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cfb86542-1303-482c-a6ce-bee9bc81da27-metrics-certs\") pod \"maas-api-e2e-mt-7e439d04-ffd9b45f6-lt8m6\" (UID: \"cfb86542-1303-482c-a6ce-bee9bc81da27\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7e439d04-ffd9b45f6-lt8m6" Nov 03 13:36:04.427003 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:36:04.426917 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cfb86542-1303-482c-a6ce-bee9bc81da27-maas-api-tls\") pod \"maas-api-e2e-mt-7e439d04-ffd9b45f6-lt8m6\" (UID: \"cfb86542-1303-482c-a6ce-bee9bc81da27\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7e439d04-ffd9b45f6-lt8m6" Nov 03 13:36:04.429935 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:36:04.429671 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7e439d04-ffd9b45f6-lt8m6" Nov 03 13:36:04.809319 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:36:04.808763 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-1ec4b892/e2e-lifecycle-model-1ec4b892-kserve-5b96cbdb9c-rl92k" event={"ID":"b1a1aac6-a092-4640-b64d-6b62b12163d2","Type":"ContainerDied","Data":"aaa648165d1f2e58d5ef68c14793faafa0b30a0dc92aa64836d7d7f63943f9e9"} Nov 03 13:36:04.809319 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:36:04.808829 5024 scope.go:122] "RemoveContainer" containerID="ed935b179655f78e8113be171468f27d20e1cde0a0327b370d200890690bf32d" Nov 03 13:36:04.809319 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:36:04.809009 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-1ec4b892/e2e-lifecycle-model-1ec4b892-kserve-5b96cbdb9c-rl92k" Nov 03 13:36:04.961699 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:36:04.961632 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-1ec4b892/e2e-lifecycle-model-1ec4b892-kserve-5b96cbdb9c-rl92k"] Nov 03 13:36:05.113927 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:36:05.113841 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-1ec4b892/e2e-lifecycle-model-1ec4b892-kserve-5b96cbdb9c-rl92k"] Nov 03 13:36:05.114210 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:36:05.114194 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-7e439d04-ffd9b45f6-lt8m6"] Nov 03 13:36:05.831377 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:36:05.831340 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7e439d04-ffd9b45f6-lt8m6" event={"ID":"cfb86542-1303-482c-a6ce-bee9bc81da27","Type":"ContainerStarted","Data":"379075d5f585fb7d526e85720b5a4ccda1c02e11d0085d32b195fa2e10219136"} Nov 03 13:36:06.134518 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:36:06.134478 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b1a1aac6-a092-4640-b64d-6b62b12163d2" path="/var/lib/kubelet/pods/b1a1aac6-a092-4640-b64d-6b62b12163d2/volumes" Nov 03 13:36:06.957310 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:36:06.956322 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7e439d04-ffd9b45f6-lt8m6" event={"ID":"cfb86542-1303-482c-a6ce-bee9bc81da27","Type":"ContainerStarted","Data":"4dcb14124747e301c972f6b37212b6e22ed87aa250ec6801cc9799cf045faea7"} Nov 03 13:36:06.957310 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:36:06.957002 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7e439d04-ffd9b45f6-lt8m6" Nov 03 13:36:07.000954 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:36:07.000879 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7e439d04-ffd9b45f6-lt8m6" podStartSLOduration=3.356467919 podStartE2EDuration="4.000854766s" totalImagesPullingTime="644.386847ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:36:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:36:06.995107716 +0000 UTC m=+1647.003298995" watchObservedRunningTime="2026-11-03 13:36:07.000854766 +0000 UTC m=+1647.009046061" Nov 03 13:36:09.518556 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:36:09.518506 5024 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d7mwz" containerName="extractor" Nov 03 13:36:12.658318 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:36:12.656268 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-9f71f2-5b8844dc8c-6xt6h" Nov 03 13:36:13.042906 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:36:13.030693 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7e439d04-ffd9b45f6-lt8m6" Nov 03 13:36:14.065568 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:36:14.065534 5024 generic.go:356] "Generic (PLEG): container finished" podID="160d4b07-2558-4e4d-afb7-a7d509cc68ca" containerID="9b8b4fed8f49f179cf7e812cb0767af05932b07fc896bd7af1f6ccf37503d2f8" exitCode=0 Nov 03 13:36:14.066029 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:36:14.065603 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-c47670-openshift-default-7f67b9bfc-rv4l7" event={"ID":"160d4b07-2558-4e4d-afb7-a7d509cc68ca","Type":"ContainerDied","Data":"9b8b4fed8f49f179cf7e812cb0767af05932b07fc896bd7af1f6ccf37503d2f8"} Nov 03 13:36:14.066100 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:36:14.066084 5024 scope.go:122] "RemoveContainer" containerID="9b8b4fed8f49f179cf7e812cb0767af05932b07fc896bd7af1f6ccf37503d2f8" Nov 03 13:36:14.068505 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:36:14.068470 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:36:14.068614 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:36:14.068569 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:36:14.068663 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:36:14.068641 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:36:15.117336 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:36:15.114147 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-c47670-openshift-default-7f67b9bfc-rv4l7" event={"ID":"160d4b07-2558-4e4d-afb7-a7d509cc68ca","Type":"ContainerStarted","Data":"4245b3cae40c16aaa5bb911f6a4236f951bba5bd44441134ccc19ae9ba252f86"} Nov 03 13:36:15.838309 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:36:15.822585 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-c47670-openshift-default-7f67b9bfc-rv4l7" Nov 03 13:36:15.894772 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:36:15.851126 5024 patch_prober.go:29] interesting pod/e2e-extmod-a-c47670-openshift-default-7f67b9bfc-rv4l7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" start-of-body= Nov 03 13:36:15.894772 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:36:15.851196 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-c47670-openshift-default-7f67b9bfc-rv4l7" podUID="160d4b07-2558-4e4d-afb7-a7d509cc68ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" Nov 03 13:36:16.802809 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:36:16.802763 5024 patch_prober.go:29] interesting pod/e2e-extmod-a-c47670-openshift-default-7f67b9bfc-rv4l7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" start-of-body= Nov 03 13:36:16.803420 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:36:16.803387 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-c47670-openshift-default-7f67b9bfc-rv4l7" podUID="160d4b07-2558-4e4d-afb7-a7d509cc68ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" Nov 03 13:36:17.799691 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:36:17.799651 5024 patch_prober.go:29] interesting pod/e2e-extmod-a-c47670-openshift-default-7f67b9bfc-rv4l7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" start-of-body= Nov 03 13:36:17.799870 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:36:17.799704 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-c47670-openshift-default-7f67b9bfc-rv4l7" podUID="160d4b07-2558-4e4d-afb7-a7d509cc68ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" Nov 03 13:36:18.800310 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:36:18.799687 5024 patch_prober.go:29] interesting pod/e2e-extmod-a-c47670-openshift-default-7f67b9bfc-rv4l7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" start-of-body= Nov 03 13:36:18.800310 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:36:18.799748 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-c47670-openshift-default-7f67b9bfc-rv4l7" podUID="160d4b07-2558-4e4d-afb7-a7d509cc68ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" Nov 03 13:36:19.804149 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:36:19.802748 5024 patch_prober.go:29] interesting pod/e2e-extmod-a-c47670-openshift-default-7f67b9bfc-rv4l7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" start-of-body= Nov 03 13:36:19.804149 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:36:19.802930 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-c47670-openshift-default-7f67b9bfc-rv4l7" podUID="160d4b07-2558-4e4d-afb7-a7d509cc68ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" Nov 03 13:36:20.800083 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:36:20.800049 5024 patch_prober.go:29] interesting pod/e2e-extmod-a-c47670-openshift-default-7f67b9bfc-rv4l7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" start-of-body= Nov 03 13:36:20.800277 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:36:20.800106 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-c47670-openshift-default-7f67b9bfc-rv4l7" podUID="160d4b07-2558-4e4d-afb7-a7d509cc68ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" Nov 03 13:36:21.348318 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:36:21.347209 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-c47670-openshift-default-7f67b9bfc-rv4l7" Nov 03 13:36:21.805873 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:36:21.805841 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-c47670-openshift-default-7f67b9bfc-rv4l7" Nov 03 13:36:22.187692 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:36:22.187659 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-c47670-openshift-default-7f67b9bfc-rv4l7" Nov 03 13:36:27.340032 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:36:27.339990 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-1ec4b892-6b49f5f675-hxqrb"] Nov 03 13:36:32.341227 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:36:32.341165 5024 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-1ec4b892-6b49f5f675-hxqrb" podUID="47cf254c-610c-4b37-8bcd-dd4c11dcff35" containerName="payload-pre-processing" containerID="cri-o://22e4a525e9086e9659155a80406699dcc228893b354d9a9dc27db29f74069f0d" gracePeriod=25 Nov 03 13:36:32.708577 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:36:32.708202 5024 generic.go:356] "Generic (PLEG): container finished" podID="47cf254c-610c-4b37-8bcd-dd4c11dcff35" containerID="22e4a525e9086e9659155a80406699dcc228893b354d9a9dc27db29f74069f0d" exitCode=0 Nov 03 13:36:32.708577 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:36:32.708514 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-1ec4b892-6b49f5f675-hxqrb" event={"ID":"47cf254c-610c-4b37-8bcd-dd4c11dcff35","Type":"ContainerDied","Data":"22e4a525e9086e9659155a80406699dcc228893b354d9a9dc27db29f74069f0d"} Nov 03 13:36:32.768903 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:36:32.768584 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-1ec4b892-6b49f5f675-hxqrb" Nov 03 13:36:32.869322 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:36:32.865358 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/47cf254c-610c-4b37-8bcd-dd4c11dcff35-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47cf254c-610c-4b37-8bcd-dd4c11dcff35-plugins-config-volume\") pod \"47cf254c-610c-4b37-8bcd-dd4c11dcff35\" (UID: \"47cf254c-610c-4b37-8bcd-dd4c11dcff35\") " Nov 03 13:36:32.869322 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:36:32.865584 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47cf254c-610c-4b37-8bcd-dd4c11dcff35-kube-api-access-qlrb4\" (UniqueName: \"kubernetes.io/projected/47cf254c-610c-4b37-8bcd-dd4c11dcff35-kube-api-access-qlrb4\") pod \"47cf254c-610c-4b37-8bcd-dd4c11dcff35\" (UID: \"47cf254c-610c-4b37-8bcd-dd4c11dcff35\") " Nov 03 13:36:32.869322 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:36:32.866110 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47cf254c-610c-4b37-8bcd-dd4c11dcff35-plugins-config-volume" pod "47cf254c-610c-4b37-8bcd-dd4c11dcff35" (UID: "47cf254c-610c-4b37-8bcd-dd4c11dcff35"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:36:32.869322 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:36:32.866475 5024 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47cf254c-610c-4b37-8bcd-dd4c11dcff35-plugins-config-volume\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:36:32.904179 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:36:32.903872 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47cf254c-610c-4b37-8bcd-dd4c11dcff35-kube-api-access-qlrb4" pod "47cf254c-610c-4b37-8bcd-dd4c11dcff35" (UID: "47cf254c-610c-4b37-8bcd-dd4c11dcff35"). InnerVolumeSpecName "kube-api-access-qlrb4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:36:32.968476 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:36:32.968386 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qlrb4\" (UniqueName: \"kubernetes.io/projected/47cf254c-610c-4b37-8bcd-dd4c11dcff35-kube-api-access-qlrb4\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:36:33.719563 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:36:33.719522 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-1ec4b892-6b49f5f675-hxqrb" event={"ID":"47cf254c-610c-4b37-8bcd-dd4c11dcff35","Type":"ContainerDied","Data":"be645f18bac117ad9a3e7eccdf0d496141b9d1928028b159053ac4e8a3bc94c5"} Nov 03 13:36:33.719563 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:36:33.719560 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-1ec4b892-6b49f5f675-hxqrb" Nov 03 13:36:33.720100 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:36:33.719601 5024 scope.go:122] "RemoveContainer" containerID="22e4a525e9086e9659155a80406699dcc228893b354d9a9dc27db29f74069f0d" Nov 03 13:36:33.749807 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:36:33.749772 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-1ec4b892-6b49f5f675-hxqrb"] Nov 03 13:36:33.753545 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:36:33.753514 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-1ec4b892-6b49f5f675-hxqrb"] Nov 03 13:36:34.112859 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:36:34.112824 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="47cf254c-610c-4b37-8bcd-dd4c11dcff35" path="/var/lib/kubelet/pods/47cf254c-610c-4b37-8bcd-dd4c11dcff35/volumes" Nov 03 13:36:35.524995 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:36:35.524956 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-c47670/payload-processing-external-model-e2e-extmod-a-c47670-659497kbr"] Nov 03 13:36:35.525905 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:36:35.525884 5024 cpu_manager.go:414] "containerMap: removing container" podUID="47cf254c-610c-4b37-8bcd-dd4c11dcff35" containerName="payload-pre-processing" Nov 03 13:36:35.525970 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:36:35.525906 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47cf254c-610c-4b37-8bcd-dd4c11dcff35" containerName="payload-pre-processing" Nov 03 13:36:35.526044 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:36:35.526035 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="47cf254c-610c-4b37-8bcd-dd4c11dcff35" containerName="payload-pre-processing" Nov 03 13:36:35.531268 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:36:35.531247 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-c47670/payload-processing-external-model-e2e-extmod-a-c47670-659497kbr" Nov 03 13:36:35.537884 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:36:35.537833 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-c47670\"/\"e2e-cmp-legacy-9e93d6-provider-cred\"" Nov 03 13:36:35.538048 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:36:35.538015 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-c47670\"/\"payload-processing-external-model-e2e--27ff040c-dockercfg-trh9n\"" Nov 03 13:36:35.538114 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:36:35.538057 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-c47670\"/\"payload-processing-external-model-plugins-e2e-extmod-a-c47670\"" Nov 03 13:36:35.538114 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:36:35.538112 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-c47670\"/\"routing-overlay\"" Nov 03 13:36:35.538225 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:36:35.538164 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-c47670\"/\"openshift-service-ca.crt\"" Nov 03 13:36:35.539703 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:36:35.539679 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-c47670/payload-processing-external-model-e2e-extmod-a-c47670-659497kbr"] Nov 03 13:36:35.539839 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:36:35.539709 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-c47670\"/\"kube-root-ca.crt\"" Nov 03 13:36:35.776398 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:36:35.771687 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-44r8k\" (UniqueName: \"kubernetes.io/projected/52b866fe-9c71-40c3-b14a-4674654d638e-kube-api-access-44r8k\") pod \"payload-processing-external-model-e2e-extmod-a-c47670-659497kbr\" (UID: \"52b866fe-9c71-40c3-b14a-4674654d638e\") " pod="ai-tenant-e2e-extmod-a-c47670/payload-processing-external-model-e2e-extmod-a-c47670-659497kbr" Nov 03 13:36:35.776398 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:36:35.771922 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/52b866fe-9c71-40c3-b14a-4674654d638e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-c47670-659497kbr\" (UID: \"52b866fe-9c71-40c3-b14a-4674654d638e\") " pod="ai-tenant-e2e-extmod-a-c47670/payload-processing-external-model-e2e-extmod-a-c47670-659497kbr" Nov 03 13:36:35.776398 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:36:35.771989 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/52b866fe-9c71-40c3-b14a-4674654d638e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-c47670-659497kbr\" (UID: \"52b866fe-9c71-40c3-b14a-4674654d638e\") " pod="ai-tenant-e2e-extmod-a-c47670/payload-processing-external-model-e2e-extmod-a-c47670-659497kbr" Nov 03 13:36:35.776398 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:36:35.772076 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52b866fe-9c71-40c3-b14a-4674654d638e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-c47670-659497kbr\" (UID: \"52b866fe-9c71-40c3-b14a-4674654d638e\") " pod="ai-tenant-e2e-extmod-a-c47670/payload-processing-external-model-e2e-extmod-a-c47670-659497kbr" Nov 03 13:36:35.882389 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:36:35.881616 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/52b866fe-9c71-40c3-b14a-4674654d638e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-c47670-659497kbr\" (UID: \"52b866fe-9c71-40c3-b14a-4674654d638e\") " pod="ai-tenant-e2e-extmod-a-c47670/payload-processing-external-model-e2e-extmod-a-c47670-659497kbr" Nov 03 13:36:35.882389 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:36:35.881725 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/52b866fe-9c71-40c3-b14a-4674654d638e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-c47670-659497kbr\" (UID: \"52b866fe-9c71-40c3-b14a-4674654d638e\") " pod="ai-tenant-e2e-extmod-a-c47670/payload-processing-external-model-e2e-extmod-a-c47670-659497kbr" Nov 03 13:36:35.882389 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:36:35.881829 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52b866fe-9c71-40c3-b14a-4674654d638e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-c47670-659497kbr\" (UID: \"52b866fe-9c71-40c3-b14a-4674654d638e\") " pod="ai-tenant-e2e-extmod-a-c47670/payload-processing-external-model-e2e-extmod-a-c47670-659497kbr" Nov 03 13:36:35.882389 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:36:35.881897 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-44r8k\" (UniqueName: \"kubernetes.io/projected/52b866fe-9c71-40c3-b14a-4674654d638e-kube-api-access-44r8k\") pod \"payload-processing-external-model-e2e-extmod-a-c47670-659497kbr\" (UID: \"52b866fe-9c71-40c3-b14a-4674654d638e\") " pod="ai-tenant-e2e-extmod-a-c47670/payload-processing-external-model-e2e-extmod-a-c47670-659497kbr" Nov 03 13:36:35.883736 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:36:35.883707 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/52b866fe-9c71-40c3-b14a-4674654d638e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-c47670-659497kbr\" (UID: \"52b866fe-9c71-40c3-b14a-4674654d638e\") " pod="ai-tenant-e2e-extmod-a-c47670/payload-processing-external-model-e2e-extmod-a-c47670-659497kbr" Nov 03 13:36:35.902337 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:36:35.885182 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52b866fe-9c71-40c3-b14a-4674654d638e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-c47670-659497kbr\" (UID: \"52b866fe-9c71-40c3-b14a-4674654d638e\") " pod="ai-tenant-e2e-extmod-a-c47670/payload-processing-external-model-e2e-extmod-a-c47670-659497kbr" Nov 03 13:36:35.917505 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:36:35.917469 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-44r8k\" (UniqueName: \"kubernetes.io/projected/52b866fe-9c71-40c3-b14a-4674654d638e-kube-api-access-44r8k\") pod \"payload-processing-external-model-e2e-extmod-a-c47670-659497kbr\" (UID: \"52b866fe-9c71-40c3-b14a-4674654d638e\") " pod="ai-tenant-e2e-extmod-a-c47670/payload-processing-external-model-e2e-extmod-a-c47670-659497kbr" Nov 03 13:36:35.960309 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:36:35.946416 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/52b866fe-9c71-40c3-b14a-4674654d638e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-c47670-659497kbr\" (UID: \"52b866fe-9c71-40c3-b14a-4674654d638e\") " pod="ai-tenant-e2e-extmod-a-c47670/payload-processing-external-model-e2e-extmod-a-c47670-659497kbr" Nov 03 13:36:36.157531 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:36:36.157218 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-c47670/payload-processing-external-model-e2e-extmod-a-c47670-659497kbr" Nov 03 13:36:36.562854 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:36:36.562817 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52b866fe_9c71_40c3_b14a_4674654d638e.slice/crio-f6ae5e17f85b67f0aaf77e89b03af59040bc4bc58112755ed2761d1a8a096023 WatchSource:0}: Error finding container f6ae5e17f85b67f0aaf77e89b03af59040bc4bc58112755ed2761d1a8a096023: Status 404 returned error can't find the container with id f6ae5e17f85b67f0aaf77e89b03af59040bc4bc58112755ed2761d1a8a096023 Nov 03 13:36:36.565443 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:36:36.565416 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-c47670/payload-processing-external-model-e2e-extmod-a-c47670-659497kbr"] Nov 03 13:36:37.212896 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:36:37.212853 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-c47670/payload-processing-external-model-e2e-extmod-a-c47670-659497kbr" event={"ID":"52b866fe-9c71-40c3-b14a-4674654d638e","Type":"ContainerStarted","Data":"60c6eaad1b5bc456680e7e8b9b64b770d3cdaa8ddfbfa33bf837e796a954b48f"} Nov 03 13:36:37.213093 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:36:37.212923 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-c47670/payload-processing-external-model-e2e-extmod-a-c47670-659497kbr" event={"ID":"52b866fe-9c71-40c3-b14a-4674654d638e","Type":"ContainerStarted","Data":"f6ae5e17f85b67f0aaf77e89b03af59040bc4bc58112755ed2761d1a8a096023"} Nov 03 13:36:37.213093 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:36:37.212966 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-c47670/payload-processing-external-model-e2e-extmod-a-c47670-659497kbr" Nov 03 13:36:37.238509 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:36:37.238433 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-c47670/payload-processing-external-model-e2e-extmod-a-c47670-659497kbr" podStartSLOduration=2.238412 podStartE2EDuration="2.238412s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:36:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:36:37.232513287 +0000 UTC m=+1677.240704568" watchObservedRunningTime="2026-11-03 13:36:37.238412 +0000 UTC m=+1677.246603279" Nov 03 13:36:39.512033 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:36:39.511991 5024 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d7mwz" containerName="extractor" Nov 03 13:36:42.881524 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:36:42.881486 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-1ec4b892-openshift-default-6c8678f7c-prh98"] Nov 03 13:36:42.883571 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:36:42.883518 5024 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-1ec4b892-openshift-default-6c8678f7c-prh98" podUID="752f4211-a41b-4b2e-b568-3b3bc1ce0989" containerName="istio-proxy" containerID="cri-o://856e798c4f8001afe2ebf9e60f45eab903bdd2fd33cfeffdaa7f430dd00e4129" gracePeriod=30 Nov 03 13:36:43.295089 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:36:43.294953 5024 generic.go:356] "Generic (PLEG): container finished" podID="d803a251-b682-4e3e-b636-92fbe8d99c64" containerID="4dec6fb29c63ccf03e05c3d8d8c49f7e33af7c51bae00ab63511a676b90329c8" exitCode=0 Nov 03 13:36:43.295331 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:36:43.295252 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7e439d04-openshift-default-6ff95548d9-nh4cq" event={"ID":"d803a251-b682-4e3e-b636-92fbe8d99c64","Type":"ContainerDied","Data":"4dec6fb29c63ccf03e05c3d8d8c49f7e33af7c51bae00ab63511a676b90329c8"} Nov 03 13:36:43.295923 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:36:43.295877 5024 scope.go:122] "RemoveContainer" containerID="4dec6fb29c63ccf03e05c3d8d8c49f7e33af7c51bae00ab63511a676b90329c8" Nov 03 13:36:43.298810 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:36:43.298763 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:36:43.298948 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:36:43.298869 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:36:43.299017 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:36:43.298945 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:36:44.307172 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:36:44.307137 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7e439d04-openshift-default-6ff95548d9-nh4cq" event={"ID":"d803a251-b682-4e3e-b636-92fbe8d99c64","Type":"ContainerStarted","Data":"f6b31f5d82254e5c941ba580acabd1e344fbb1eb1048bc02fca7c08a8bae2b22"} Nov 03 13:36:44.759462 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:36:44.759427 5024 scope.go:122] "RemoveContainer" containerID="3e5de967862f794dcb98270059cb343b2490d3b5a8dd35d16aeedeb789be3873" Nov 03 13:36:44.789140 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:36:44.789108 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-7e439d04-openshift-default-6ff95548d9-nh4cq" Nov 03 13:36:44.789580 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:36:44.789556 5024 patch_prober.go:29] interesting pod/e2e-mt-7e439d04-openshift-default-6ff95548d9-nh4cq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" start-of-body= Nov 03 13:36:44.789885 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:36:44.789854 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7e439d04-openshift-default-6ff95548d9-nh4cq" podUID="d803a251-b682-4e3e-b636-92fbe8d99c64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" Nov 03 13:36:45.785308 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:36:45.780775 5024 patch_prober.go:29] interesting pod/e2e-mt-7e439d04-openshift-default-6ff95548d9-nh4cq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" start-of-body= Nov 03 13:36:45.785308 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:36:45.780871 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7e439d04-openshift-default-6ff95548d9-nh4cq" podUID="d803a251-b682-4e3e-b636-92fbe8d99c64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" Nov 03 13:36:46.802498 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:36:46.798656 5024 patch_prober.go:29] interesting pod/e2e-mt-7e439d04-openshift-default-6ff95548d9-nh4cq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" start-of-body= Nov 03 13:36:46.802498 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:36:46.798734 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7e439d04-openshift-default-6ff95548d9-nh4cq" podUID="d803a251-b682-4e3e-b636-92fbe8d99c64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" Nov 03 13:36:46.958432 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:36:46.958392 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-7e439d04-openshift-default-6ff95548d9-nh4cq" Nov 03 13:36:47.803917 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:36:47.802767 5024 patch_prober.go:29] interesting pod/e2e-mt-7e439d04-openshift-default-6ff95548d9-nh4cq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" start-of-body= Nov 03 13:36:47.803917 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:36:47.802828 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7e439d04-openshift-default-6ff95548d9-nh4cq" podUID="d803a251-b682-4e3e-b636-92fbe8d99c64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" Nov 03 13:36:48.321012 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:36:48.320975 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-c47670/payload-processing-external-model-e2e-extmod-a-c47670-659497kbr" Nov 03 13:36:48.425027 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:36:48.424985 5024 generic.go:356] "Generic (PLEG): container finished" podID="752f4211-a41b-4b2e-b568-3b3bc1ce0989" containerID="856e798c4f8001afe2ebf9e60f45eab903bdd2fd33cfeffdaa7f430dd00e4129" exitCode=0 Nov 03 13:36:48.425183 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:36:48.425114 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-1ec4b892-openshift-default-6c8678f7c-prh98" event={"ID":"752f4211-a41b-4b2e-b568-3b3bc1ce0989","Type":"ContainerDied","Data":"856e798c4f8001afe2ebf9e60f45eab903bdd2fd33cfeffdaa7f430dd00e4129"} Nov 03 13:36:48.425183 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:36:48.425166 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-1ec4b892-openshift-default-6c8678f7c-prh98" event={"ID":"752f4211-a41b-4b2e-b568-3b3bc1ce0989","Type":"ContainerDied","Data":"e7f9accae9d900ef81ca1b69ed184e3ea1b053524e94297d63d22fa9c68c6133"} Nov 03 13:36:48.425418 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:36:48.425187 5024 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e7f9accae9d900ef81ca1b69ed184e3ea1b053524e94297d63d22fa9c68c6133" Nov 03 13:36:48.425418 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:36:48.425205 5024 scope.go:122] "RemoveContainer" containerID="7724369ad432b2c96a54f3a089844dd113f77e4ccf6f578ac8e8719894f4ee45" Nov 03 13:36:48.443373 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:36:48.443347 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-1ec4b892-openshift-default-6c8678f7c-prh98" Nov 03 13:36:48.469951 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:36:48.469912 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/752f4211-a41b-4b2e-b568-3b3bc1ce0989-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/752f4211-a41b-4b2e-b568-3b3bc1ce0989-credential-socket\") pod \"752f4211-a41b-4b2e-b568-3b3bc1ce0989\" (UID: \"752f4211-a41b-4b2e-b568-3b3bc1ce0989\") " Nov 03 13:36:48.470127 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:36:48.469977 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/752f4211-a41b-4b2e-b568-3b3bc1ce0989-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/752f4211-a41b-4b2e-b568-3b3bc1ce0989-istio-podinfo\") pod \"752f4211-a41b-4b2e-b568-3b3bc1ce0989\" (UID: \"752f4211-a41b-4b2e-b568-3b3bc1ce0989\") " Nov 03 13:36:48.470127 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:36:48.470014 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/752f4211-a41b-4b2e-b568-3b3bc1ce0989-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/752f4211-a41b-4b2e-b568-3b3bc1ce0989-istiod-ca-cert\") pod \"752f4211-a41b-4b2e-b568-3b3bc1ce0989\" (UID: \"752f4211-a41b-4b2e-b568-3b3bc1ce0989\") " Nov 03 13:36:48.470127 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:36:48.470048 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/752f4211-a41b-4b2e-b568-3b3bc1ce0989-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/752f4211-a41b-4b2e-b568-3b3bc1ce0989-workload-socket\") pod \"752f4211-a41b-4b2e-b568-3b3bc1ce0989\" (UID: \"752f4211-a41b-4b2e-b568-3b3bc1ce0989\") " Nov 03 13:36:48.495308 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:36:48.470275 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/752f4211-a41b-4b2e-b568-3b3bc1ce0989-kube-api-access-zcplm\" (UniqueName: \"kubernetes.io/projected/752f4211-a41b-4b2e-b568-3b3bc1ce0989-kube-api-access-zcplm\") pod \"752f4211-a41b-4b2e-b568-3b3bc1ce0989\" (UID: \"752f4211-a41b-4b2e-b568-3b3bc1ce0989\") " Nov 03 13:36:48.495308 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:36:48.470344 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/752f4211-a41b-4b2e-b568-3b3bc1ce0989-credential-socket" pod "752f4211-a41b-4b2e-b568-3b3bc1ce0989" (UID: "752f4211-a41b-4b2e-b568-3b3bc1ce0989"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:36:48.495308 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:36:48.470486 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/752f4211-a41b-4b2e-b568-3b3bc1ce0989-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/752f4211-a41b-4b2e-b568-3b3bc1ce0989-istio-data\") pod \"752f4211-a41b-4b2e-b568-3b3bc1ce0989\" (UID: \"752f4211-a41b-4b2e-b568-3b3bc1ce0989\") " Nov 03 13:36:48.495308 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:36:48.470546 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/752f4211-a41b-4b2e-b568-3b3bc1ce0989-istio-token\" (UniqueName: \"kubernetes.io/projected/752f4211-a41b-4b2e-b568-3b3bc1ce0989-istio-token\") pod \"752f4211-a41b-4b2e-b568-3b3bc1ce0989\" (UID: \"752f4211-a41b-4b2e-b568-3b3bc1ce0989\") " Nov 03 13:36:48.495308 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:36:48.470588 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/752f4211-a41b-4b2e-b568-3b3bc1ce0989-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/752f4211-a41b-4b2e-b568-3b3bc1ce0989-istio-envoy\") pod \"752f4211-a41b-4b2e-b568-3b3bc1ce0989\" (UID: \"752f4211-a41b-4b2e-b568-3b3bc1ce0989\") " Nov 03 13:36:48.495308 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:36:48.470597 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/752f4211-a41b-4b2e-b568-3b3bc1ce0989-istiod-ca-cert" pod "752f4211-a41b-4b2e-b568-3b3bc1ce0989" (UID: "752f4211-a41b-4b2e-b568-3b3bc1ce0989"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:36:48.495308 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:36:48.470713 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/752f4211-a41b-4b2e-b568-3b3bc1ce0989-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/752f4211-a41b-4b2e-b568-3b3bc1ce0989-workload-certs\") pod \"752f4211-a41b-4b2e-b568-3b3bc1ce0989\" (UID: \"752f4211-a41b-4b2e-b568-3b3bc1ce0989\") " Nov 03 13:36:48.495308 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:36:48.471658 5024 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/752f4211-a41b-4b2e-b568-3b3bc1ce0989-credential-socket\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:36:48.495308 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:36:48.471679 5024 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/752f4211-a41b-4b2e-b568-3b3bc1ce0989-istiod-ca-cert\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:36:48.495308 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:36:48.471937 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/752f4211-a41b-4b2e-b568-3b3bc1ce0989-workload-certs" pod "752f4211-a41b-4b2e-b568-3b3bc1ce0989" (UID: "752f4211-a41b-4b2e-b568-3b3bc1ce0989"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:36:48.495308 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:36:48.472111 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/752f4211-a41b-4b2e-b568-3b3bc1ce0989-istio-data" pod "752f4211-a41b-4b2e-b568-3b3bc1ce0989" (UID: "752f4211-a41b-4b2e-b568-3b3bc1ce0989"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:36:48.495308 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:36:48.481474 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/752f4211-a41b-4b2e-b568-3b3bc1ce0989-workload-socket" pod "752f4211-a41b-4b2e-b568-3b3bc1ce0989" (UID: "752f4211-a41b-4b2e-b568-3b3bc1ce0989"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:36:48.513691 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:36:48.513474 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/752f4211-a41b-4b2e-b568-3b3bc1ce0989-istio-token" pod "752f4211-a41b-4b2e-b568-3b3bc1ce0989" (UID: "752f4211-a41b-4b2e-b568-3b3bc1ce0989"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:36:48.513691 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:36:48.513593 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/752f4211-a41b-4b2e-b568-3b3bc1ce0989-istio-envoy" pod "752f4211-a41b-4b2e-b568-3b3bc1ce0989" (UID: "752f4211-a41b-4b2e-b568-3b3bc1ce0989"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:36:48.515532 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:36:48.514779 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/752f4211-a41b-4b2e-b568-3b3bc1ce0989-istio-podinfo" pod "752f4211-a41b-4b2e-b568-3b3bc1ce0989" (UID: "752f4211-a41b-4b2e-b568-3b3bc1ce0989"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 13:36:48.515532 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:36:48.514903 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/752f4211-a41b-4b2e-b568-3b3bc1ce0989-kube-api-access-zcplm" pod "752f4211-a41b-4b2e-b568-3b3bc1ce0989" (UID: "752f4211-a41b-4b2e-b568-3b3bc1ce0989"). InnerVolumeSpecName "kube-api-access-zcplm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:36:48.605757 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:36:48.579775 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zcplm\" (UniqueName: \"kubernetes.io/projected/752f4211-a41b-4b2e-b568-3b3bc1ce0989-kube-api-access-zcplm\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:36:48.605757 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:36:48.604423 5024 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/752f4211-a41b-4b2e-b568-3b3bc1ce0989-istio-data\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:36:48.605757 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:36:48.604459 5024 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/752f4211-a41b-4b2e-b568-3b3bc1ce0989-istio-token\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:36:48.605757 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:36:48.604474 5024 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/752f4211-a41b-4b2e-b568-3b3bc1ce0989-istio-envoy\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:36:48.605757 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:36:48.604490 5024 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/752f4211-a41b-4b2e-b568-3b3bc1ce0989-workload-certs\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:36:48.605757 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:36:48.604504 5024 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/752f4211-a41b-4b2e-b568-3b3bc1ce0989-istio-podinfo\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:36:48.605757 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:36:48.604522 5024 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/752f4211-a41b-4b2e-b568-3b3bc1ce0989-workload-socket\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:36:48.784361 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:36:48.783706 5024 patch_prober.go:29] interesting pod/e2e-mt-7e439d04-openshift-default-6ff95548d9-nh4cq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" start-of-body= Nov 03 13:36:48.784361 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:36:48.783770 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7e439d04-openshift-default-6ff95548d9-nh4cq" podUID="d803a251-b682-4e3e-b636-92fbe8d99c64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" Nov 03 13:36:49.456087 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:36:49.456057 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-1ec4b892-openshift-default-6c8678f7c-prh98" Nov 03 13:36:49.523066 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:36:49.523025 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-1ec4b892-openshift-default-6c8678f7c-prh98"] Nov 03 13:36:49.533471 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:36:49.533438 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-1ec4b892-openshift-default-6c8678f7c-prh98"] Nov 03 13:36:49.792362 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:36:49.787441 5024 patch_prober.go:29] interesting pod/e2e-mt-7e439d04-openshift-default-6ff95548d9-nh4cq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" start-of-body= Nov 03 13:36:49.792362 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:36:49.787515 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7e439d04-openshift-default-6ff95548d9-nh4cq" podUID="d803a251-b682-4e3e-b636-92fbe8d99c64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" Nov 03 13:36:50.260312 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:36:50.259997 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="752f4211-a41b-4b2e-b568-3b3bc1ce0989" path="/var/lib/kubelet/pods/752f4211-a41b-4b2e-b568-3b3bc1ce0989/volumes" Nov 03 13:36:50.824777 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:36:50.824737 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-7e439d04-openshift-default-6ff95548d9-nh4cq" Nov 03 13:36:50.858341 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:36:50.858210 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-7e439d04-openshift-default-6ff95548d9-nh4cq" Nov 03 13:36:52.604753 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:36:52.604704 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-autores-146318/autores-model-146318-kserve-76f88db5c-h8z2m"] Nov 03 13:36:52.605824 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:36:52.605796 5024 cpu_manager.go:414] "containerMap: removing container" podUID="752f4211-a41b-4b2e-b568-3b3bc1ce0989" containerName="istio-proxy" Nov 03 13:36:52.605936 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:36:52.605836 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="752f4211-a41b-4b2e-b568-3b3bc1ce0989" containerName="istio-proxy" Nov 03 13:36:52.605936 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:36:52.605910 5024 cpu_manager.go:414] "containerMap: removing container" podUID="752f4211-a41b-4b2e-b568-3b3bc1ce0989" containerName="istio-proxy" Nov 03 13:36:52.605936 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:36:52.605918 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="752f4211-a41b-4b2e-b568-3b3bc1ce0989" containerName="istio-proxy" Nov 03 13:36:52.606104 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:36:52.606087 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="752f4211-a41b-4b2e-b568-3b3bc1ce0989" containerName="istio-proxy" Nov 03 13:36:52.615080 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:36:52.615028 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-146318/autores-model-146318-kserve-76f88db5c-h8z2m" Nov 03 13:36:52.626012 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:36:52.625976 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/80edac2d-0809-40a9-8db2-8fa0fd40d02d-tls-certs\") pod \"autores-model-146318-kserve-76f88db5c-h8z2m\" (UID: \"80edac2d-0809-40a9-8db2-8fa0fd40d02d\") " pod="ai-tenant-e2e-autores-146318/autores-model-146318-kserve-76f88db5c-h8z2m" Nov 03 13:36:52.626371 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:36:52.626343 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/80edac2d-0809-40a9-8db2-8fa0fd40d02d-model-cache\") pod \"autores-model-146318-kserve-76f88db5c-h8z2m\" (UID: \"80edac2d-0809-40a9-8db2-8fa0fd40d02d\") " pod="ai-tenant-e2e-autores-146318/autores-model-146318-kserve-76f88db5c-h8z2m" Nov 03 13:36:52.626609 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:36:52.626589 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/80edac2d-0809-40a9-8db2-8fa0fd40d02d-home\") pod \"autores-model-146318-kserve-76f88db5c-h8z2m\" (UID: \"80edac2d-0809-40a9-8db2-8fa0fd40d02d\") " pod="ai-tenant-e2e-autores-146318/autores-model-146318-kserve-76f88db5c-h8z2m" Nov 03 13:36:52.626784 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:36:52.626768 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/80edac2d-0809-40a9-8db2-8fa0fd40d02d-tmp-dir\") pod \"autores-model-146318-kserve-76f88db5c-h8z2m\" (UID: \"80edac2d-0809-40a9-8db2-8fa0fd40d02d\") " pod="ai-tenant-e2e-autores-146318/autores-model-146318-kserve-76f88db5c-h8z2m" Nov 03 13:36:52.627035 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:36:52.626979 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qtwwx\" (UniqueName: \"kubernetes.io/projected/80edac2d-0809-40a9-8db2-8fa0fd40d02d-kube-api-access-qtwwx\") pod \"autores-model-146318-kserve-76f88db5c-h8z2m\" (UID: \"80edac2d-0809-40a9-8db2-8fa0fd40d02d\") " pod="ai-tenant-e2e-autores-146318/autores-model-146318-kserve-76f88db5c-h8z2m" Nov 03 13:36:52.627217 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:36:52.627200 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/80edac2d-0809-40a9-8db2-8fa0fd40d02d-dshm\") pod \"autores-model-146318-kserve-76f88db5c-h8z2m\" (UID: \"80edac2d-0809-40a9-8db2-8fa0fd40d02d\") " pod="ai-tenant-e2e-autores-146318/autores-model-146318-kserve-76f88db5c-h8z2m" Nov 03 13:36:52.627568 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:36:52.627548 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-146318/autores-model-146318-kserve-76f88db5c-h8z2m"] Nov 03 13:36:52.627708 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:36:52.627625 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-146318\"/\"kube-root-ca.crt\"" Nov 03 13:36:52.630630 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:36:52.630585 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-146318\"/\"default-dockercfg-fv26v\"" Nov 03 13:36:52.630740 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:36:52.630631 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-146318\"/\"autores-model-146318-kserve-self-signed-certs\"" Nov 03 13:36:52.728033 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:36:52.727991 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-146318\"/\"openshift-service-ca.crt\"" Nov 03 13:36:52.728604 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:36:52.728570 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/80edac2d-0809-40a9-8db2-8fa0fd40d02d-tmp-dir\") pod \"autores-model-146318-kserve-76f88db5c-h8z2m\" (UID: \"80edac2d-0809-40a9-8db2-8fa0fd40d02d\") " pod="ai-tenant-e2e-autores-146318/autores-model-146318-kserve-76f88db5c-h8z2m" Nov 03 13:36:52.728751 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:36:52.728683 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qtwwx\" (UniqueName: \"kubernetes.io/projected/80edac2d-0809-40a9-8db2-8fa0fd40d02d-kube-api-access-qtwwx\") pod \"autores-model-146318-kserve-76f88db5c-h8z2m\" (UID: \"80edac2d-0809-40a9-8db2-8fa0fd40d02d\") " pod="ai-tenant-e2e-autores-146318/autores-model-146318-kserve-76f88db5c-h8z2m" Nov 03 13:36:52.728751 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:36:52.728718 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/80edac2d-0809-40a9-8db2-8fa0fd40d02d-dshm\") pod \"autores-model-146318-kserve-76f88db5c-h8z2m\" (UID: \"80edac2d-0809-40a9-8db2-8fa0fd40d02d\") " pod="ai-tenant-e2e-autores-146318/autores-model-146318-kserve-76f88db5c-h8z2m" Nov 03 13:36:52.728877 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:36:52.728777 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/80edac2d-0809-40a9-8db2-8fa0fd40d02d-tls-certs\") pod \"autores-model-146318-kserve-76f88db5c-h8z2m\" (UID: \"80edac2d-0809-40a9-8db2-8fa0fd40d02d\") " pod="ai-tenant-e2e-autores-146318/autores-model-146318-kserve-76f88db5c-h8z2m" Nov 03 13:36:52.728935 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:36:52.728898 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/80edac2d-0809-40a9-8db2-8fa0fd40d02d-model-cache\") pod \"autores-model-146318-kserve-76f88db5c-h8z2m\" (UID: \"80edac2d-0809-40a9-8db2-8fa0fd40d02d\") " pod="ai-tenant-e2e-autores-146318/autores-model-146318-kserve-76f88db5c-h8z2m" Nov 03 13:36:52.728988 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:36:52.728963 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/80edac2d-0809-40a9-8db2-8fa0fd40d02d-tmp-dir\") pod \"autores-model-146318-kserve-76f88db5c-h8z2m\" (UID: \"80edac2d-0809-40a9-8db2-8fa0fd40d02d\") " pod="ai-tenant-e2e-autores-146318/autores-model-146318-kserve-76f88db5c-h8z2m" Nov 03 13:36:52.729049 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:36:52.729039 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/80edac2d-0809-40a9-8db2-8fa0fd40d02d-home\") pod \"autores-model-146318-kserve-76f88db5c-h8z2m\" (UID: \"80edac2d-0809-40a9-8db2-8fa0fd40d02d\") " pod="ai-tenant-e2e-autores-146318/autores-model-146318-kserve-76f88db5c-h8z2m" Nov 03 13:36:52.729272 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:36:52.729250 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/80edac2d-0809-40a9-8db2-8fa0fd40d02d-model-cache\") pod \"autores-model-146318-kserve-76f88db5c-h8z2m\" (UID: \"80edac2d-0809-40a9-8db2-8fa0fd40d02d\") " pod="ai-tenant-e2e-autores-146318/autores-model-146318-kserve-76f88db5c-h8z2m" Nov 03 13:36:52.729418 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:36:52.729302 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/80edac2d-0809-40a9-8db2-8fa0fd40d02d-home\") pod \"autores-model-146318-kserve-76f88db5c-h8z2m\" (UID: \"80edac2d-0809-40a9-8db2-8fa0fd40d02d\") " pod="ai-tenant-e2e-autores-146318/autores-model-146318-kserve-76f88db5c-h8z2m" Nov 03 13:36:52.731273 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:36:52.731247 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/80edac2d-0809-40a9-8db2-8fa0fd40d02d-dshm\") pod \"autores-model-146318-kserve-76f88db5c-h8z2m\" (UID: \"80edac2d-0809-40a9-8db2-8fa0fd40d02d\") " pod="ai-tenant-e2e-autores-146318/autores-model-146318-kserve-76f88db5c-h8z2m" Nov 03 13:36:52.731428 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:36:52.731378 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/80edac2d-0809-40a9-8db2-8fa0fd40d02d-tls-certs\") pod \"autores-model-146318-kserve-76f88db5c-h8z2m\" (UID: \"80edac2d-0809-40a9-8db2-8fa0fd40d02d\") " pod="ai-tenant-e2e-autores-146318/autores-model-146318-kserve-76f88db5c-h8z2m" Nov 03 13:36:52.742428 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:36:52.742389 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qtwwx\" (UniqueName: \"kubernetes.io/projected/80edac2d-0809-40a9-8db2-8fa0fd40d02d-kube-api-access-qtwwx\") pod \"autores-model-146318-kserve-76f88db5c-h8z2m\" (UID: \"80edac2d-0809-40a9-8db2-8fa0fd40d02d\") " pod="ai-tenant-e2e-autores-146318/autores-model-146318-kserve-76f88db5c-h8z2m" Nov 03 13:36:52.947383 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:36:52.945592 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-146318/autores-model-146318-kserve-76f88db5c-h8z2m" Nov 03 13:36:53.398722 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:36:53.398680 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-146318/autores-model-146318-kserve-76f88db5c-h8z2m"] Nov 03 13:36:53.400086 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:36:53.400054 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod80edac2d_0809_40a9_8db2_8fa0fd40d02d.slice/crio-0512e27116ce7f70b6b59deb72b9922a54e23b8d11246a3e19cebce16ee979e2 WatchSource:0}: Error finding container 0512e27116ce7f70b6b59deb72b9922a54e23b8d11246a3e19cebce16ee979e2: Status 404 returned error can't find the container with id 0512e27116ce7f70b6b59deb72b9922a54e23b8d11246a3e19cebce16ee979e2 Nov 03 13:36:53.650380 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:36:53.649548 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-146318/autores-model-146318-kserve-76f88db5c-h8z2m" event={"ID":"80edac2d-0809-40a9-8db2-8fa0fd40d02d","Type":"ContainerStarted","Data":"b70415ff937bd1002dde22994c12397ce727e99d57de5ae5f96967ee474903c3"} Nov 03 13:36:53.650380 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:36:53.649612 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-146318/autores-model-146318-kserve-76f88db5c-h8z2m" event={"ID":"80edac2d-0809-40a9-8db2-8fa0fd40d02d","Type":"ContainerStarted","Data":"0512e27116ce7f70b6b59deb72b9922a54e23b8d11246a3e19cebce16ee979e2"} Nov 03 13:36:53.684311 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:36:53.683497 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-autores-146318/autores-model-146318-kserve-76f88db5c-h8z2m" podStartSLOduration=1.6834742820000002 podStartE2EDuration="1.683474282s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:36:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:36:53.681669199 +0000 UTC m=+1693.689860473" watchObservedRunningTime="2026-11-03 13:36:53.683474282 +0000 UTC m=+1693.691665560" Nov 03 13:36:53.957364 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:36:53.956897 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-autores-146318/autores-model-146318-kserve-76f88db5c-h8z2m" Nov 03 13:37:02.946103 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:37:02.946064 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-autores-146318/autores-model-146318-kserve-76f88db5c-h8z2m" Nov 03 13:37:02.964454 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:37:02.964410 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-autores-146318/autores-model-146318-kserve-76f88db5c-h8z2m" Nov 03 13:37:03.757900 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:37:03.757864 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-autores-146318/autores-model-146318-kserve-76f88db5c-h8z2m" Nov 03 13:37:06.473230 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:37:06.473192 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-146318/autores-model-146318-kserve-76f88db5c-h8z2m"] Nov 03 13:37:09.513193 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:37:09.513150 5024 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d7mwz" containerName="extractor" Nov 03 13:37:21.384101 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:37:21.384049 5024 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-08111b/simulator-w1-08111b-kserve-7f65c5867-btf5b" podUID="e8aba665-e4cc-4e67-8cdd-d09ef7542e15" containerName="main" probeResult="failure" output="Get \"https://10.132.0.133:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 13:37:21.448815 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:37:21.447734 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-08111b/simulator-w1-08111b-kserve-7f65c5867-btf5b" podUID="e8aba665-e4cc-4e67-8cdd-d09ef7542e15" containerName="main" probeResult="failure" output="Get \"https://10.132.0.133:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 13:37:21.529587 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:37:21.528927 5024 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-autores-146318/autores-model-146318-kserve-76f88db5c-h8z2m" podUID="80edac2d-0809-40a9-8db2-8fa0fd40d02d" containerName="main" containerID="cri-o://b70415ff937bd1002dde22994c12397ce727e99d57de5ae5f96967ee474903c3" gracePeriod=45 Nov 03 13:37:21.731490 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:37:21.731384 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d3ca48/premium-w3-d3ca48-kserve-75ffc959b-zvjbc" podUID="add1e8b9-e7fd-4e4d-8d10-f97d0e2612ba" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 13:37:21.762710 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:37:21.761726 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-autores-146318/autores-model-146318-kserve-76f88db5c-h8z2m" podUID="80edac2d-0809-40a9-8db2-8fa0fd40d02d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.158:8000/ready\": context deadline exceeded" Nov 03 13:37:21.826830 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:37:21.826789 5024 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-h2ns8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 13:37:21.827116 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:37:21.827082 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-h2ns8" podUID="155ba865-dcfd-4203-82b9-5399ca58d988" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 13:37:21.938344 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:37:21.934235 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w2-a724a6/sub-test-model-a724a6-kserve-57d7c88d9f-5nk9m" podUID="3e6246a7-8c0e-4712-8f67-d834948a5673" containerName="main" probeResult="failure" output="Get \"https://10.132.0.153:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 13:37:22.082367 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:37:22.051401 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-autores-146318/autores-model-146318-kserve-76f88db5c-h8z2m" podUID="80edac2d-0809-40a9-8db2-8fa0fd40d02d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.158:8000/ready\": EOF" Nov 03 13:37:22.082367 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:37:22.051625 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-autores-146318/autores-model-146318-kserve-76f88db5c-h8z2m" Nov 03 13:37:22.090122 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:37:22.087652 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-08111b/premium-w1-08111b-kserve-57555d598-dx9g5" podUID="e45c569a-243f-49ab-8d74-dd08cbb6127c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.134:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 13:37:22.748709 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:37:22.748611 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-autores-146318/autores-model-146318-kserve-76f88db5c-h8z2m" podUID="80edac2d-0809-40a9-8db2-8fa0fd40d02d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.158:8000/ready\": dial tcp 10.132.0.158:8000: connect: connection refused" Nov 03 13:37:22.927249 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:37:22.927212 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-146318/autores-model-146318-kserve-76f88db5c-h8z2m" Nov 03 13:37:23.043485 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:37:23.043383 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80edac2d-0809-40a9-8db2-8fa0fd40d02d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/80edac2d-0809-40a9-8db2-8fa0fd40d02d-model-cache\") pod \"80edac2d-0809-40a9-8db2-8fa0fd40d02d\" (UID: \"80edac2d-0809-40a9-8db2-8fa0fd40d02d\") " Nov 03 13:37:23.043930 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:37:23.043897 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80edac2d-0809-40a9-8db2-8fa0fd40d02d-home\" (UniqueName: \"kubernetes.io/empty-dir/80edac2d-0809-40a9-8db2-8fa0fd40d02d-home\") pod \"80edac2d-0809-40a9-8db2-8fa0fd40d02d\" (UID: \"80edac2d-0809-40a9-8db2-8fa0fd40d02d\") " Nov 03 13:37:23.044131 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:37:23.044108 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/80edac2d-0809-40a9-8db2-8fa0fd40d02d-tls-certs\" (UniqueName: \"kubernetes.io/secret/80edac2d-0809-40a9-8db2-8fa0fd40d02d-tls-certs\") pod \"80edac2d-0809-40a9-8db2-8fa0fd40d02d\" (UID: \"80edac2d-0809-40a9-8db2-8fa0fd40d02d\") " Nov 03 13:37:23.044341 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:37:23.044314 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80edac2d-0809-40a9-8db2-8fa0fd40d02d-kube-api-access-qtwwx\" (UniqueName: \"kubernetes.io/projected/80edac2d-0809-40a9-8db2-8fa0fd40d02d-kube-api-access-qtwwx\") pod \"80edac2d-0809-40a9-8db2-8fa0fd40d02d\" (UID: \"80edac2d-0809-40a9-8db2-8fa0fd40d02d\") " Nov 03 13:37:23.045868 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:37:23.045845 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80edac2d-0809-40a9-8db2-8fa0fd40d02d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/80edac2d-0809-40a9-8db2-8fa0fd40d02d-dshm\") pod \"80edac2d-0809-40a9-8db2-8fa0fd40d02d\" (UID: \"80edac2d-0809-40a9-8db2-8fa0fd40d02d\") " Nov 03 13:37:23.046212 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:37:23.046180 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80edac2d-0809-40a9-8db2-8fa0fd40d02d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/80edac2d-0809-40a9-8db2-8fa0fd40d02d-tmp-dir\") pod \"80edac2d-0809-40a9-8db2-8fa0fd40d02d\" (UID: \"80edac2d-0809-40a9-8db2-8fa0fd40d02d\") " Nov 03 13:37:23.048497 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:37:23.048471 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80edac2d-0809-40a9-8db2-8fa0fd40d02d-tmp-dir" pod "80edac2d-0809-40a9-8db2-8fa0fd40d02d" (UID: "80edac2d-0809-40a9-8db2-8fa0fd40d02d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:37:23.048976 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:37:23.048954 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80edac2d-0809-40a9-8db2-8fa0fd40d02d-model-cache" pod "80edac2d-0809-40a9-8db2-8fa0fd40d02d" (UID: "80edac2d-0809-40a9-8db2-8fa0fd40d02d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:37:23.049357 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:37:23.049335 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80edac2d-0809-40a9-8db2-8fa0fd40d02d-home" pod "80edac2d-0809-40a9-8db2-8fa0fd40d02d" (UID: "80edac2d-0809-40a9-8db2-8fa0fd40d02d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:37:23.108383 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:37:23.108335 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80edac2d-0809-40a9-8db2-8fa0fd40d02d-dshm" pod "80edac2d-0809-40a9-8db2-8fa0fd40d02d" (UID: "80edac2d-0809-40a9-8db2-8fa0fd40d02d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:37:23.118608 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:37:23.117808 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/80edac2d-0809-40a9-8db2-8fa0fd40d02d-tls-certs" pod "80edac2d-0809-40a9-8db2-8fa0fd40d02d" (UID: "80edac2d-0809-40a9-8db2-8fa0fd40d02d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 13:37:23.127830 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:37:23.127784 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80edac2d-0809-40a9-8db2-8fa0fd40d02d-kube-api-access-qtwwx" pod "80edac2d-0809-40a9-8db2-8fa0fd40d02d" (UID: "80edac2d-0809-40a9-8db2-8fa0fd40d02d"). InnerVolumeSpecName "kube-api-access-qtwwx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:37:23.150349 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:37:23.150316 5024 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/80edac2d-0809-40a9-8db2-8fa0fd40d02d-home\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:37:23.150488 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:37:23.150437 5024 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/80edac2d-0809-40a9-8db2-8fa0fd40d02d-tls-certs\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:37:23.150488 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:37:23.150474 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qtwwx\" (UniqueName: \"kubernetes.io/projected/80edac2d-0809-40a9-8db2-8fa0fd40d02d-kube-api-access-qtwwx\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:37:23.150599 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:37:23.150543 5024 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/80edac2d-0809-40a9-8db2-8fa0fd40d02d-dshm\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:37:23.150599 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:37:23.150561 5024 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/80edac2d-0809-40a9-8db2-8fa0fd40d02d-tmp-dir\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:37:23.150599 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:37:23.150576 5024 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/80edac2d-0809-40a9-8db2-8fa0fd40d02d-model-cache\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:37:23.220659 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:37:23.220612 5024 generic.go:356] "Generic (PLEG): container finished" podID="80edac2d-0809-40a9-8db2-8fa0fd40d02d" containerID="b70415ff937bd1002dde22994c12397ce727e99d57de5ae5f96967ee474903c3" exitCode=0 Nov 03 13:37:23.220659 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:37:23.220653 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-146318/autores-model-146318-kserve-76f88db5c-h8z2m" Nov 03 13:37:23.220907 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:37:23.220691 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-146318/autores-model-146318-kserve-76f88db5c-h8z2m" event={"ID":"80edac2d-0809-40a9-8db2-8fa0fd40d02d","Type":"ContainerDied","Data":"b70415ff937bd1002dde22994c12397ce727e99d57de5ae5f96967ee474903c3"} Nov 03 13:37:23.220907 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:37:23.220735 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-146318/autores-model-146318-kserve-76f88db5c-h8z2m" event={"ID":"80edac2d-0809-40a9-8db2-8fa0fd40d02d","Type":"ContainerDied","Data":"0512e27116ce7f70b6b59deb72b9922a54e23b8d11246a3e19cebce16ee979e2"} Nov 03 13:37:23.220907 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:37:23.220755 5024 scope.go:122] "RemoveContainer" containerID="b70415ff937bd1002dde22994c12397ce727e99d57de5ae5f96967ee474903c3" Nov 03 13:37:23.234923 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:37:23.234901 5024 scope.go:122] "RemoveContainer" containerID="b70415ff937bd1002dde22994c12397ce727e99d57de5ae5f96967ee474903c3" Nov 03 13:37:23.235322 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:37:23.235258 5024 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b70415ff937bd1002dde22994c12397ce727e99d57de5ae5f96967ee474903c3\": container with ID starting with b70415ff937bd1002dde22994c12397ce727e99d57de5ae5f96967ee474903c3 not found: ID does not exist" containerID="b70415ff937bd1002dde22994c12397ce727e99d57de5ae5f96967ee474903c3" Nov 03 13:37:23.235415 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:37:23.235314 5024 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b70415ff937bd1002dde22994c12397ce727e99d57de5ae5f96967ee474903c3"} err="failed to get container status \"b70415ff937bd1002dde22994c12397ce727e99d57de5ae5f96967ee474903c3\": rpc error: code = NotFound desc = could not find container \"b70415ff937bd1002dde22994c12397ce727e99d57de5ae5f96967ee474903c3\": container with ID starting with b70415ff937bd1002dde22994c12397ce727e99d57de5ae5f96967ee474903c3 not found: ID does not exist" Nov 03 13:37:23.260006 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:37:23.259925 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-146318/autores-model-146318-kserve-76f88db5c-h8z2m"] Nov 03 13:37:23.265548 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:37:23.265510 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-autores-146318/autores-model-146318-kserve-76f88db5c-h8z2m"] Nov 03 13:37:24.112714 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:37:24.112678 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="80edac2d-0809-40a9-8db2-8fa0fd40d02d" path="/var/lib/kubelet/pods/80edac2d-0809-40a9-8db2-8fa0fd40d02d/volumes" Nov 03 13:37:39.512252 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:37:39.512157 5024 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d7mwz" containerName="extractor" Nov 03 13:37:58.746221 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:37:58.746180 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-explicit-6f33a0-openshift-default-649df665d5-rrthx"] Nov 03 13:37:58.747364 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:37:58.747338 5024 cpu_manager.go:414] "containerMap: removing container" podUID="80edac2d-0809-40a9-8db2-8fa0fd40d02d" containerName="main" Nov 03 13:37:58.747364 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:37:58.747362 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80edac2d-0809-40a9-8db2-8fa0fd40d02d" containerName="main" Nov 03 13:37:58.747562 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:37:58.747523 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="80edac2d-0809-40a9-8db2-8fa0fd40d02d" containerName="main" Nov 03 13:37:58.747562 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:37:58.747538 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="752f4211-a41b-4b2e-b568-3b3bc1ce0989" containerName="istio-proxy" Nov 03 13:37:58.751608 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:37:58.751585 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-6f33a0-openshift-default-649df665d5-rrthx" Nov 03 13:37:58.763684 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:37:58.763653 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-explicit-6f33a0-openshift-default-dockercfg-ptpd2\"" Nov 03 13:37:58.771870 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:37:58.771840 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-6f33a0-openshift-default-649df665d5-rrthx"] Nov 03 13:37:58.797857 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:37:58.797813 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0bde2d4-7fae-41d0-960a-20a0e664d597-credential-socket\") pod \"e2e-explicit-6f33a0-openshift-default-649df665d5-rrthx\" (UID: \"a0bde2d4-7fae-41d0-960a-20a0e664d597\") " pod="openshift-ingress/e2e-explicit-6f33a0-openshift-default-649df665d5-rrthx" Nov 03 13:37:58.798051 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:37:58.797876 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0bde2d4-7fae-41d0-960a-20a0e664d597-istiod-ca-cert\") pod \"e2e-explicit-6f33a0-openshift-default-649df665d5-rrthx\" (UID: \"a0bde2d4-7fae-41d0-960a-20a0e664d597\") " pod="openshift-ingress/e2e-explicit-6f33a0-openshift-default-649df665d5-rrthx" Nov 03 13:37:58.798051 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:37:58.797921 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0bde2d4-7fae-41d0-960a-20a0e664d597-istio-envoy\") pod \"e2e-explicit-6f33a0-openshift-default-649df665d5-rrthx\" (UID: \"a0bde2d4-7fae-41d0-960a-20a0e664d597\") " pod="openshift-ingress/e2e-explicit-6f33a0-openshift-default-649df665d5-rrthx" Nov 03 13:37:58.798051 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:37:58.797949 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n5c7c\" (UniqueName: \"kubernetes.io/projected/a0bde2d4-7fae-41d0-960a-20a0e664d597-kube-api-access-n5c7c\") pod \"e2e-explicit-6f33a0-openshift-default-649df665d5-rrthx\" (UID: \"a0bde2d4-7fae-41d0-960a-20a0e664d597\") " pod="openshift-ingress/e2e-explicit-6f33a0-openshift-default-649df665d5-rrthx" Nov 03 13:37:58.798051 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:37:58.797981 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0bde2d4-7fae-41d0-960a-20a0e664d597-istio-podinfo\") pod \"e2e-explicit-6f33a0-openshift-default-649df665d5-rrthx\" (UID: \"a0bde2d4-7fae-41d0-960a-20a0e664d597\") " pod="openshift-ingress/e2e-explicit-6f33a0-openshift-default-649df665d5-rrthx" Nov 03 13:37:58.798051 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:37:58.798033 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0bde2d4-7fae-41d0-960a-20a0e664d597-istio-token\") pod \"e2e-explicit-6f33a0-openshift-default-649df665d5-rrthx\" (UID: \"a0bde2d4-7fae-41d0-960a-20a0e664d597\") " pod="openshift-ingress/e2e-explicit-6f33a0-openshift-default-649df665d5-rrthx" Nov 03 13:37:58.798451 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:37:58.798075 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0bde2d4-7fae-41d0-960a-20a0e664d597-workload-certs\") pod \"e2e-explicit-6f33a0-openshift-default-649df665d5-rrthx\" (UID: \"a0bde2d4-7fae-41d0-960a-20a0e664d597\") " pod="openshift-ingress/e2e-explicit-6f33a0-openshift-default-649df665d5-rrthx" Nov 03 13:37:58.798451 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:37:58.798119 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0bde2d4-7fae-41d0-960a-20a0e664d597-istio-data\") pod \"e2e-explicit-6f33a0-openshift-default-649df665d5-rrthx\" (UID: \"a0bde2d4-7fae-41d0-960a-20a0e664d597\") " pod="openshift-ingress/e2e-explicit-6f33a0-openshift-default-649df665d5-rrthx" Nov 03 13:37:58.798451 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:37:58.798157 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0bde2d4-7fae-41d0-960a-20a0e664d597-workload-socket\") pod \"e2e-explicit-6f33a0-openshift-default-649df665d5-rrthx\" (UID: \"a0bde2d4-7fae-41d0-960a-20a0e664d597\") " pod="openshift-ingress/e2e-explicit-6f33a0-openshift-default-649df665d5-rrthx" Nov 03 13:37:58.922366 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:37:58.903325 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0bde2d4-7fae-41d0-960a-20a0e664d597-istio-data\") pod \"e2e-explicit-6f33a0-openshift-default-649df665d5-rrthx\" (UID: \"a0bde2d4-7fae-41d0-960a-20a0e664d597\") " pod="openshift-ingress/e2e-explicit-6f33a0-openshift-default-649df665d5-rrthx" Nov 03 13:37:58.922366 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:37:58.903398 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0bde2d4-7fae-41d0-960a-20a0e664d597-workload-socket\") pod \"e2e-explicit-6f33a0-openshift-default-649df665d5-rrthx\" (UID: \"a0bde2d4-7fae-41d0-960a-20a0e664d597\") " pod="openshift-ingress/e2e-explicit-6f33a0-openshift-default-649df665d5-rrthx" Nov 03 13:37:58.922366 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:37:58.903667 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0bde2d4-7fae-41d0-960a-20a0e664d597-credential-socket\") pod \"e2e-explicit-6f33a0-openshift-default-649df665d5-rrthx\" (UID: \"a0bde2d4-7fae-41d0-960a-20a0e664d597\") " pod="openshift-ingress/e2e-explicit-6f33a0-openshift-default-649df665d5-rrthx" Nov 03 13:37:58.922366 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:37:58.903725 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0bde2d4-7fae-41d0-960a-20a0e664d597-istiod-ca-cert\") pod \"e2e-explicit-6f33a0-openshift-default-649df665d5-rrthx\" (UID: \"a0bde2d4-7fae-41d0-960a-20a0e664d597\") " pod="openshift-ingress/e2e-explicit-6f33a0-openshift-default-649df665d5-rrthx" Nov 03 13:37:58.922366 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:37:58.903775 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0bde2d4-7fae-41d0-960a-20a0e664d597-istio-envoy\") pod \"e2e-explicit-6f33a0-openshift-default-649df665d5-rrthx\" (UID: \"a0bde2d4-7fae-41d0-960a-20a0e664d597\") " pod="openshift-ingress/e2e-explicit-6f33a0-openshift-default-649df665d5-rrthx" Nov 03 13:37:58.922366 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:37:58.903809 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5c7c\" (UniqueName: \"kubernetes.io/projected/a0bde2d4-7fae-41d0-960a-20a0e664d597-kube-api-access-n5c7c\") pod \"e2e-explicit-6f33a0-openshift-default-649df665d5-rrthx\" (UID: \"a0bde2d4-7fae-41d0-960a-20a0e664d597\") " pod="openshift-ingress/e2e-explicit-6f33a0-openshift-default-649df665d5-rrthx" Nov 03 13:37:58.922366 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:37:58.903846 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0bde2d4-7fae-41d0-960a-20a0e664d597-istio-podinfo\") pod \"e2e-explicit-6f33a0-openshift-default-649df665d5-rrthx\" (UID: \"a0bde2d4-7fae-41d0-960a-20a0e664d597\") " pod="openshift-ingress/e2e-explicit-6f33a0-openshift-default-649df665d5-rrthx" Nov 03 13:37:58.922366 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:37:58.903908 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0bde2d4-7fae-41d0-960a-20a0e664d597-istio-token\") pod \"e2e-explicit-6f33a0-openshift-default-649df665d5-rrthx\" (UID: \"a0bde2d4-7fae-41d0-960a-20a0e664d597\") " pod="openshift-ingress/e2e-explicit-6f33a0-openshift-default-649df665d5-rrthx" Nov 03 13:37:58.922366 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:37:58.903960 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0bde2d4-7fae-41d0-960a-20a0e664d597-workload-certs\") pod \"e2e-explicit-6f33a0-openshift-default-649df665d5-rrthx\" (UID: \"a0bde2d4-7fae-41d0-960a-20a0e664d597\") " pod="openshift-ingress/e2e-explicit-6f33a0-openshift-default-649df665d5-rrthx" Nov 03 13:37:58.922366 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:37:58.904501 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0bde2d4-7fae-41d0-960a-20a0e664d597-workload-certs\") pod \"e2e-explicit-6f33a0-openshift-default-649df665d5-rrthx\" (UID: \"a0bde2d4-7fae-41d0-960a-20a0e664d597\") " pod="openshift-ingress/e2e-explicit-6f33a0-openshift-default-649df665d5-rrthx" Nov 03 13:37:58.922366 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:37:58.904767 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0bde2d4-7fae-41d0-960a-20a0e664d597-istio-data\") pod \"e2e-explicit-6f33a0-openshift-default-649df665d5-rrthx\" (UID: \"a0bde2d4-7fae-41d0-960a-20a0e664d597\") " pod="openshift-ingress/e2e-explicit-6f33a0-openshift-default-649df665d5-rrthx" Nov 03 13:37:58.922366 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:37:58.904999 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0bde2d4-7fae-41d0-960a-20a0e664d597-workload-socket\") pod \"e2e-explicit-6f33a0-openshift-default-649df665d5-rrthx\" (UID: \"a0bde2d4-7fae-41d0-960a-20a0e664d597\") " pod="openshift-ingress/e2e-explicit-6f33a0-openshift-default-649df665d5-rrthx" Nov 03 13:37:58.922366 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:37:58.905224 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0bde2d4-7fae-41d0-960a-20a0e664d597-credential-socket\") pod \"e2e-explicit-6f33a0-openshift-default-649df665d5-rrthx\" (UID: \"a0bde2d4-7fae-41d0-960a-20a0e664d597\") " pod="openshift-ingress/e2e-explicit-6f33a0-openshift-default-649df665d5-rrthx" Nov 03 13:37:58.922366 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:37:58.905917 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0bde2d4-7fae-41d0-960a-20a0e664d597-istiod-ca-cert\") pod \"e2e-explicit-6f33a0-openshift-default-649df665d5-rrthx\" (UID: \"a0bde2d4-7fae-41d0-960a-20a0e664d597\") " pod="openshift-ingress/e2e-explicit-6f33a0-openshift-default-649df665d5-rrthx" Nov 03 13:37:59.170312 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:37:59.162187 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0bde2d4-7fae-41d0-960a-20a0e664d597-istio-envoy\") pod \"e2e-explicit-6f33a0-openshift-default-649df665d5-rrthx\" (UID: \"a0bde2d4-7fae-41d0-960a-20a0e664d597\") " pod="openshift-ingress/e2e-explicit-6f33a0-openshift-default-649df665d5-rrthx" Nov 03 13:37:59.170312 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:37:59.163113 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5c7c\" (UniqueName: \"kubernetes.io/projected/a0bde2d4-7fae-41d0-960a-20a0e664d597-kube-api-access-n5c7c\") pod \"e2e-explicit-6f33a0-openshift-default-649df665d5-rrthx\" (UID: \"a0bde2d4-7fae-41d0-960a-20a0e664d597\") " pod="openshift-ingress/e2e-explicit-6f33a0-openshift-default-649df665d5-rrthx" Nov 03 13:37:59.170312 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:37:59.166673 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0bde2d4-7fae-41d0-960a-20a0e664d597-istio-token\") pod \"e2e-explicit-6f33a0-openshift-default-649df665d5-rrthx\" (UID: \"a0bde2d4-7fae-41d0-960a-20a0e664d597\") " pod="openshift-ingress/e2e-explicit-6f33a0-openshift-default-649df665d5-rrthx" Nov 03 13:37:59.170312 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:37:59.168906 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0bde2d4-7fae-41d0-960a-20a0e664d597-istio-podinfo\") pod \"e2e-explicit-6f33a0-openshift-default-649df665d5-rrthx\" (UID: \"a0bde2d4-7fae-41d0-960a-20a0e664d597\") " pod="openshift-ingress/e2e-explicit-6f33a0-openshift-default-649df665d5-rrthx" Nov 03 13:37:59.367541 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:37:59.367506 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-6f33a0-openshift-default-649df665d5-rrthx" Nov 03 13:37:59.690452 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:37:59.687461 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-6f33a0-openshift-default-649df665d5-rrthx"] Nov 03 13:37:59.690452 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:37:59.688127 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0bde2d4_7fae_41d0_960a_20a0e664d597.slice/crio-b18bff99869fa2e2275ed08b90973ab221190e1acc6ebebc960c34af52aebf9c WatchSource:0}: Error finding container b18bff99869fa2e2275ed08b90973ab221190e1acc6ebebc960c34af52aebf9c: Status 404 returned error can't find the container with id b18bff99869fa2e2275ed08b90973ab221190e1acc6ebebc960c34af52aebf9c Nov 03 13:37:59.690915 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:37:59.690869 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:37:59.690988 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:37:59.690941 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:37:59.690988 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:37:59.690971 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:37:59.728367 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:37:59.728330 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-6f33a0-openshift-default-649df665d5-rrthx" event={"ID":"a0bde2d4-7fae-41d0-960a-20a0e664d597","Type":"ContainerStarted","Data":"b18bff99869fa2e2275ed08b90973ab221190e1acc6ebebc960c34af52aebf9c"} Nov 03 13:38:00.737582 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:00.737541 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-6f33a0-openshift-default-649df665d5-rrthx" event={"ID":"a0bde2d4-7fae-41d0-960a-20a0e664d597","Type":"ContainerStarted","Data":"0f0b1f5e44436747e56dfcf0213c92ce11a6c6f6a5e4882be66718fb8f5c3618"} Nov 03 13:38:00.769338 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:00.769263 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-explicit-6f33a0-openshift-default-649df665d5-rrthx" podStartSLOduration=2.769248132 podStartE2EDuration="2.769248132s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:37:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:38:00.765338357 +0000 UTC m=+1760.773529633" watchObservedRunningTime="2026-11-03 13:38:00.769248132 +0000 UTC m=+1760.777439408" Nov 03 13:38:01.369786 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:01.369754 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-6f33a0-openshift-default-649df665d5-rrthx" Nov 03 13:38:01.391470 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:01.391438 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-6f33a0-openshift-default-649df665d5-rrthx" Nov 03 13:38:02.019306 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:02.017768 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-6f33a0-openshift-default-649df665d5-rrthx" Nov 03 13:38:02.029634 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:02.029597 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-6f33a0-openshift-default-649df665d5-rrthx" Nov 03 13:38:03.721389 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:03.721334 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-d7gxh"] Nov 03 13:38:03.723349 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:03.723312 5024 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-d7gxh" podUID="6ec0db42-ee6d-496c-8792-a17fe716e820" containerName="istio-proxy" containerID="cri-o://6c3dc16a75a1544367fe596cd5d106649ccf4e7a165aa80d943b388a325928d7" gracePeriod=30 Nov 03 13:38:04.494805 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:04.494711 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-csthf"] Nov 03 13:38:04.495277 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:04.495243 5024 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-csthf" podUID="3c69ea47-20cb-40de-94fb-add95bd74c10" containerName="istio-proxy" containerID="cri-o://e5e570b28483fbb69bd7d73ee91848f5f59c37b0211abca330431bad5432b4d5" gracePeriod=30 Nov 03 13:38:07.561614 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:07.561570 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-c47670/payload-processing-external-model-e2e-extmod-a-c47670-659497kbr"] Nov 03 13:38:08.825922 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:08.825882 5024 patch_prober.go:29] interesting pod/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-d7gxh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body= Nov 03 13:38:08.826415 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:08.825953 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-d7gxh" podUID="6ec0db42-ee6d-496c-8792-a17fe716e820" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" Nov 03 13:38:09.176176 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:09.176131 5024 generic.go:356] "Generic (PLEG): container finished" podID="6ec0db42-ee6d-496c-8792-a17fe716e820" containerID="6c3dc16a75a1544367fe596cd5d106649ccf4e7a165aa80d943b388a325928d7" exitCode=0 Nov 03 13:38:09.181253 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:09.181214 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-d7gxh" event={"ID":"6ec0db42-ee6d-496c-8792-a17fe716e820","Type":"ContainerDied","Data":"6c3dc16a75a1544367fe596cd5d106649ccf4e7a165aa80d943b388a325928d7"} Nov 03 13:38:09.181530 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:09.181507 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-d7gxh" event={"ID":"6ec0db42-ee6d-496c-8792-a17fe716e820","Type":"ContainerDied","Data":"aff61b0630e8ffc87d23002febe3c6f023c341ce8d198d6e11dd6ac97cc63138"} Nov 03 13:38:09.181655 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:09.181640 5024 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aff61b0630e8ffc87d23002febe3c6f023c341ce8d198d6e11dd6ac97cc63138" Nov 03 13:38:09.199886 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:09.199857 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-d7gxh" Nov 03 13:38:09.230727 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:09.230667 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ec0db42-ee6d-496c-8792-a17fe716e820-kube-api-access-fdr27\" (UniqueName: \"kubernetes.io/projected/6ec0db42-ee6d-496c-8792-a17fe716e820-kube-api-access-fdr27\") pod \"6ec0db42-ee6d-496c-8792-a17fe716e820\" (UID: \"6ec0db42-ee6d-496c-8792-a17fe716e820\") " Nov 03 13:38:09.230924 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:09.230870 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ec0db42-ee6d-496c-8792-a17fe716e820-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ec0db42-ee6d-496c-8792-a17fe716e820-istio-data\") pod \"6ec0db42-ee6d-496c-8792-a17fe716e820\" (UID: \"6ec0db42-ee6d-496c-8792-a17fe716e820\") " Nov 03 13:38:09.231341 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:09.231070 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6ec0db42-ee6d-496c-8792-a17fe716e820-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ec0db42-ee6d-496c-8792-a17fe716e820-istio-podinfo\") pod \"6ec0db42-ee6d-496c-8792-a17fe716e820\" (UID: \"6ec0db42-ee6d-496c-8792-a17fe716e820\") " Nov 03 13:38:09.231341 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:09.231111 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ec0db42-ee6d-496c-8792-a17fe716e820-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ec0db42-ee6d-496c-8792-a17fe716e820-workload-certs\") pod \"6ec0db42-ee6d-496c-8792-a17fe716e820\" (UID: \"6ec0db42-ee6d-496c-8792-a17fe716e820\") " Nov 03 13:38:09.231341 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:09.231148 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ec0db42-ee6d-496c-8792-a17fe716e820-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ec0db42-ee6d-496c-8792-a17fe716e820-workload-socket\") pod \"6ec0db42-ee6d-496c-8792-a17fe716e820\" (UID: \"6ec0db42-ee6d-496c-8792-a17fe716e820\") " Nov 03 13:38:09.231341 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:09.231218 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ec0db42-ee6d-496c-8792-a17fe716e820-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ec0db42-ee6d-496c-8792-a17fe716e820-istio-envoy\") pod \"6ec0db42-ee6d-496c-8792-a17fe716e820\" (UID: \"6ec0db42-ee6d-496c-8792-a17fe716e820\") " Nov 03 13:38:09.231341 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:09.231262 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ec0db42-ee6d-496c-8792-a17fe716e820-istio-token\" (UniqueName: \"kubernetes.io/projected/6ec0db42-ee6d-496c-8792-a17fe716e820-istio-token\") pod \"6ec0db42-ee6d-496c-8792-a17fe716e820\" (UID: \"6ec0db42-ee6d-496c-8792-a17fe716e820\") " Nov 03 13:38:09.231341 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:09.231327 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ec0db42-ee6d-496c-8792-a17fe716e820-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ec0db42-ee6d-496c-8792-a17fe716e820-credential-socket\") pod \"6ec0db42-ee6d-496c-8792-a17fe716e820\" (UID: \"6ec0db42-ee6d-496c-8792-a17fe716e820\") " Nov 03 13:38:09.231698 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:09.231439 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ec0db42-ee6d-496c-8792-a17fe716e820-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ec0db42-ee6d-496c-8792-a17fe716e820-istiod-ca-cert\") pod \"6ec0db42-ee6d-496c-8792-a17fe716e820\" (UID: \"6ec0db42-ee6d-496c-8792-a17fe716e820\") " Nov 03 13:38:09.232496 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:09.232456 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ec0db42-ee6d-496c-8792-a17fe716e820-istio-data" pod "6ec0db42-ee6d-496c-8792-a17fe716e820" (UID: "6ec0db42-ee6d-496c-8792-a17fe716e820"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:38:09.232652 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:09.232622 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ec0db42-ee6d-496c-8792-a17fe716e820-istiod-ca-cert" pod "6ec0db42-ee6d-496c-8792-a17fe716e820" (UID: "6ec0db42-ee6d-496c-8792-a17fe716e820"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:38:09.232902 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:09.232880 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ec0db42-ee6d-496c-8792-a17fe716e820-workload-socket" pod "6ec0db42-ee6d-496c-8792-a17fe716e820" (UID: "6ec0db42-ee6d-496c-8792-a17fe716e820"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:38:09.233541 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:09.233464 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ec0db42-ee6d-496c-8792-a17fe716e820-workload-certs" pod "6ec0db42-ee6d-496c-8792-a17fe716e820" (UID: "6ec0db42-ee6d-496c-8792-a17fe716e820"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:38:09.233742 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:09.233523 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ec0db42-ee6d-496c-8792-a17fe716e820-credential-socket" pod "6ec0db42-ee6d-496c-8792-a17fe716e820" (UID: "6ec0db42-ee6d-496c-8792-a17fe716e820"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:38:09.235975 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:09.235948 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ec0db42-ee6d-496c-8792-a17fe716e820-istio-token" pod "6ec0db42-ee6d-496c-8792-a17fe716e820" (UID: "6ec0db42-ee6d-496c-8792-a17fe716e820"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:38:09.236685 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:09.236568 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ec0db42-ee6d-496c-8792-a17fe716e820-istio-envoy" pod "6ec0db42-ee6d-496c-8792-a17fe716e820" (UID: "6ec0db42-ee6d-496c-8792-a17fe716e820"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:38:09.236806 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:09.236782 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ec0db42-ee6d-496c-8792-a17fe716e820-kube-api-access-fdr27" pod "6ec0db42-ee6d-496c-8792-a17fe716e820" (UID: "6ec0db42-ee6d-496c-8792-a17fe716e820"). InnerVolumeSpecName "kube-api-access-fdr27". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:38:09.238334 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:09.238238 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6ec0db42-ee6d-496c-8792-a17fe716e820-istio-podinfo" pod "6ec0db42-ee6d-496c-8792-a17fe716e820" (UID: "6ec0db42-ee6d-496c-8792-a17fe716e820"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 13:38:09.355576 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:09.355529 5024 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ec0db42-ee6d-496c-8792-a17fe716e820-istio-podinfo\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:38:09.355576 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:09.355577 5024 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ec0db42-ee6d-496c-8792-a17fe716e820-workload-certs\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:38:09.355834 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:09.355593 5024 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ec0db42-ee6d-496c-8792-a17fe716e820-workload-socket\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:38:09.355834 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:09.355607 5024 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ec0db42-ee6d-496c-8792-a17fe716e820-istio-envoy\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:38:09.355834 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:09.355625 5024 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ec0db42-ee6d-496c-8792-a17fe716e820-istio-token\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:38:09.355834 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:09.355640 5024 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ec0db42-ee6d-496c-8792-a17fe716e820-credential-socket\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:38:09.355834 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:09.355654 5024 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ec0db42-ee6d-496c-8792-a17fe716e820-istiod-ca-cert\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:38:09.355834 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:09.355671 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fdr27\" (UniqueName: \"kubernetes.io/projected/6ec0db42-ee6d-496c-8792-a17fe716e820-kube-api-access-fdr27\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:38:09.355834 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:09.355686 5024 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ec0db42-ee6d-496c-8792-a17fe716e820-istio-data\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:38:09.534777 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:38:09.534738 5024 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d7mwz" containerName="extractor" Nov 03 13:38:09.913630 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:09.913600 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-csthf" Nov 03 13:38:09.965584 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:09.965539 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3c69ea47-20cb-40de-94fb-add95bd74c10-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c69ea47-20cb-40de-94fb-add95bd74c10-istiod-ca-cert\") pod \"3c69ea47-20cb-40de-94fb-add95bd74c10\" (UID: \"3c69ea47-20cb-40de-94fb-add95bd74c10\") " Nov 03 13:38:09.965758 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:09.965682 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c69ea47-20cb-40de-94fb-add95bd74c10-istio-token\" (UniqueName: \"kubernetes.io/projected/3c69ea47-20cb-40de-94fb-add95bd74c10-istio-token\") pod \"3c69ea47-20cb-40de-94fb-add95bd74c10\" (UID: \"3c69ea47-20cb-40de-94fb-add95bd74c10\") " Nov 03 13:38:09.965758 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:09.965750 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c69ea47-20cb-40de-94fb-add95bd74c10-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c69ea47-20cb-40de-94fb-add95bd74c10-istio-data\") pod \"3c69ea47-20cb-40de-94fb-add95bd74c10\" (UID: \"3c69ea47-20cb-40de-94fb-add95bd74c10\") " Nov 03 13:38:09.965907 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:09.965890 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c69ea47-20cb-40de-94fb-add95bd74c10-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c69ea47-20cb-40de-94fb-add95bd74c10-istio-envoy\") pod \"3c69ea47-20cb-40de-94fb-add95bd74c10\" (UID: \"3c69ea47-20cb-40de-94fb-add95bd74c10\") " Nov 03 13:38:09.965976 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:09.965936 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c69ea47-20cb-40de-94fb-add95bd74c10-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c69ea47-20cb-40de-94fb-add95bd74c10-workload-socket\") pod \"3c69ea47-20cb-40de-94fb-add95bd74c10\" (UID: \"3c69ea47-20cb-40de-94fb-add95bd74c10\") " Nov 03 13:38:09.965976 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:09.965965 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c69ea47-20cb-40de-94fb-add95bd74c10-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c69ea47-20cb-40de-94fb-add95bd74c10-workload-certs\") pod \"3c69ea47-20cb-40de-94fb-add95bd74c10\" (UID: \"3c69ea47-20cb-40de-94fb-add95bd74c10\") " Nov 03 13:38:09.966118 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:09.966101 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3c69ea47-20cb-40de-94fb-add95bd74c10-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c69ea47-20cb-40de-94fb-add95bd74c10-istio-podinfo\") pod \"3c69ea47-20cb-40de-94fb-add95bd74c10\" (UID: \"3c69ea47-20cb-40de-94fb-add95bd74c10\") " Nov 03 13:38:09.966182 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:09.966159 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c69ea47-20cb-40de-94fb-add95bd74c10-kube-api-access-qmfz7\" (UniqueName: \"kubernetes.io/projected/3c69ea47-20cb-40de-94fb-add95bd74c10-kube-api-access-qmfz7\") pod \"3c69ea47-20cb-40de-94fb-add95bd74c10\" (UID: \"3c69ea47-20cb-40de-94fb-add95bd74c10\") " Nov 03 13:38:09.966246 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:09.965960 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c69ea47-20cb-40de-94fb-add95bd74c10-istiod-ca-cert" pod "3c69ea47-20cb-40de-94fb-add95bd74c10" (UID: "3c69ea47-20cb-40de-94fb-add95bd74c10"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:38:09.966246 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:09.966232 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c69ea47-20cb-40de-94fb-add95bd74c10-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c69ea47-20cb-40de-94fb-add95bd74c10-credential-socket\") pod \"3c69ea47-20cb-40de-94fb-add95bd74c10\" (UID: \"3c69ea47-20cb-40de-94fb-add95bd74c10\") " Nov 03 13:38:09.966659 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:09.966632 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c69ea47-20cb-40de-94fb-add95bd74c10-workload-certs" pod "3c69ea47-20cb-40de-94fb-add95bd74c10" (UID: "3c69ea47-20cb-40de-94fb-add95bd74c10"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:38:09.966794 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:09.966042 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c69ea47-20cb-40de-94fb-add95bd74c10-istio-data" pod "3c69ea47-20cb-40de-94fb-add95bd74c10" (UID: "3c69ea47-20cb-40de-94fb-add95bd74c10"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:38:09.966794 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:09.966215 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c69ea47-20cb-40de-94fb-add95bd74c10-workload-socket" pod "3c69ea47-20cb-40de-94fb-add95bd74c10" (UID: "3c69ea47-20cb-40de-94fb-add95bd74c10"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:38:09.966983 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:09.966873 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c69ea47-20cb-40de-94fb-add95bd74c10-credential-socket" pod "3c69ea47-20cb-40de-94fb-add95bd74c10" (UID: "3c69ea47-20cb-40de-94fb-add95bd74c10"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:38:09.967570 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:09.967539 5024 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c69ea47-20cb-40de-94fb-add95bd74c10-istiod-ca-cert\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:38:09.967701 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:09.967573 5024 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c69ea47-20cb-40de-94fb-add95bd74c10-istio-data\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:38:09.967701 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:09.967587 5024 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c69ea47-20cb-40de-94fb-add95bd74c10-workload-socket\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:38:09.967701 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:09.967602 5024 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c69ea47-20cb-40de-94fb-add95bd74c10-workload-certs\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:38:09.967701 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:09.967616 5024 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c69ea47-20cb-40de-94fb-add95bd74c10-credential-socket\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:38:09.968453 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:09.968430 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c69ea47-20cb-40de-94fb-add95bd74c10-istio-envoy" pod "3c69ea47-20cb-40de-94fb-add95bd74c10" (UID: "3c69ea47-20cb-40de-94fb-add95bd74c10"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:38:09.969168 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:09.969136 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c69ea47-20cb-40de-94fb-add95bd74c10-istio-token" pod "3c69ea47-20cb-40de-94fb-add95bd74c10" (UID: "3c69ea47-20cb-40de-94fb-add95bd74c10"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:38:09.969655 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:09.969619 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c69ea47-20cb-40de-94fb-add95bd74c10-kube-api-access-qmfz7" pod "3c69ea47-20cb-40de-94fb-add95bd74c10" (UID: "3c69ea47-20cb-40de-94fb-add95bd74c10"). InnerVolumeSpecName "kube-api-access-qmfz7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:38:09.970450 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:09.970420 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3c69ea47-20cb-40de-94fb-add95bd74c10-istio-podinfo" pod "3c69ea47-20cb-40de-94fb-add95bd74c10" (UID: "3c69ea47-20cb-40de-94fb-add95bd74c10"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 13:38:10.069761 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:10.069679 5024 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c69ea47-20cb-40de-94fb-add95bd74c10-istio-envoy\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:38:10.069761 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:10.069714 5024 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c69ea47-20cb-40de-94fb-add95bd74c10-istio-podinfo\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:38:10.069761 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:10.069724 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qmfz7\" (UniqueName: \"kubernetes.io/projected/3c69ea47-20cb-40de-94fb-add95bd74c10-kube-api-access-qmfz7\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:38:10.069761 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:10.069735 5024 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c69ea47-20cb-40de-94fb-add95bd74c10-istio-token\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:38:10.192552 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:10.192511 5024 generic.go:356] "Generic (PLEG): container finished" podID="3c69ea47-20cb-40de-94fb-add95bd74c10" containerID="e5e570b28483fbb69bd7d73ee91848f5f59c37b0211abca330431bad5432b4d5" exitCode=0 Nov 03 13:38:10.192788 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:10.192583 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-csthf" Nov 03 13:38:10.192788 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:10.192593 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-csthf" event={"ID":"3c69ea47-20cb-40de-94fb-add95bd74c10","Type":"ContainerDied","Data":"e5e570b28483fbb69bd7d73ee91848f5f59c37b0211abca330431bad5432b4d5"} Nov 03 13:38:10.192788 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:10.192691 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-csthf" event={"ID":"3c69ea47-20cb-40de-94fb-add95bd74c10","Type":"ContainerDied","Data":"6445b6239a63794829fb307c11feccbb7e2fba95503bd1e8e5f39dd65bfdc11c"} Nov 03 13:38:10.192788 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:10.192721 5024 scope.go:122] "RemoveContainer" containerID="e5e570b28483fbb69bd7d73ee91848f5f59c37b0211abca330431bad5432b4d5" Nov 03 13:38:10.193049 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:10.192913 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-d7gxh" Nov 03 13:38:10.204233 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:10.204202 5024 scope.go:122] "RemoveContainer" containerID="fb1b7bcd2ac51511eb46ebfa4e730d348cddc59d0a252d65cd58e6b9191402d5" Nov 03 13:38:10.217069 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:10.217044 5024 scope.go:122] "RemoveContainer" containerID="e5e570b28483fbb69bd7d73ee91848f5f59c37b0211abca330431bad5432b4d5" Nov 03 13:38:10.217408 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:38:10.217384 5024 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e5e570b28483fbb69bd7d73ee91848f5f59c37b0211abca330431bad5432b4d5\": container with ID starting with e5e570b28483fbb69bd7d73ee91848f5f59c37b0211abca330431bad5432b4d5 not found: ID does not exist" containerID="e5e570b28483fbb69bd7d73ee91848f5f59c37b0211abca330431bad5432b4d5" Nov 03 13:38:10.217502 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:10.217417 5024 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5e570b28483fbb69bd7d73ee91848f5f59c37b0211abca330431bad5432b4d5"} err="failed to get container status \"e5e570b28483fbb69bd7d73ee91848f5f59c37b0211abca330431bad5432b4d5\": rpc error: code = NotFound desc = could not find container \"e5e570b28483fbb69bd7d73ee91848f5f59c37b0211abca330431bad5432b4d5\": container with ID starting with e5e570b28483fbb69bd7d73ee91848f5f59c37b0211abca330431bad5432b4d5 not found: ID does not exist" Nov 03 13:38:10.217502 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:10.217441 5024 scope.go:122] "RemoveContainer" containerID="fb1b7bcd2ac51511eb46ebfa4e730d348cddc59d0a252d65cd58e6b9191402d5" Nov 03 13:38:10.217806 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:38:10.217781 5024 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb1b7bcd2ac51511eb46ebfa4e730d348cddc59d0a252d65cd58e6b9191402d5\": container with ID starting with fb1b7bcd2ac51511eb46ebfa4e730d348cddc59d0a252d65cd58e6b9191402d5 not found: ID does not exist" containerID="fb1b7bcd2ac51511eb46ebfa4e730d348cddc59d0a252d65cd58e6b9191402d5" Nov 03 13:38:10.217911 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:10.217812 5024 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb1b7bcd2ac51511eb46ebfa4e730d348cddc59d0a252d65cd58e6b9191402d5"} err="failed to get container status \"fb1b7bcd2ac51511eb46ebfa4e730d348cddc59d0a252d65cd58e6b9191402d5\": rpc error: code = NotFound desc = could not find container \"fb1b7bcd2ac51511eb46ebfa4e730d348cddc59d0a252d65cd58e6b9191402d5\": container with ID starting with fb1b7bcd2ac51511eb46ebfa4e730d348cddc59d0a252d65cd58e6b9191402d5 not found: ID does not exist" Nov 03 13:38:10.223146 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:10.223114 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-d7gxh"] Nov 03 13:38:10.229276 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:10.229240 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-d7gxh"] Nov 03 13:38:10.242454 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:10.242422 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-csthf"] Nov 03 13:38:10.248271 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:10.248242 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-8f8aa8-openshift-default-7b6b47ddd7-csthf"] Nov 03 13:38:10.936073 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:10.936037 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w0-8f8aa8/rate-test-model-8f8aa8-kserve-7786f67f6f-mbztf"] Nov 03 13:38:10.937384 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:10.937351 5024 cpu_manager.go:414] "containerMap: removing container" podUID="6ec0db42-ee6d-496c-8792-a17fe716e820" containerName="istio-proxy" Nov 03 13:38:10.937384 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:10.937376 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ec0db42-ee6d-496c-8792-a17fe716e820" containerName="istio-proxy" Nov 03 13:38:10.937573 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:10.937415 5024 cpu_manager.go:414] "containerMap: removing container" podUID="3c69ea47-20cb-40de-94fb-add95bd74c10" containerName="istio-proxy" Nov 03 13:38:10.937573 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:10.937423 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c69ea47-20cb-40de-94fb-add95bd74c10" containerName="istio-proxy" Nov 03 13:38:10.937671 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:10.937654 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c69ea47-20cb-40de-94fb-add95bd74c10" containerName="istio-proxy" Nov 03 13:38:10.937718 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:10.937673 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ec0db42-ee6d-496c-8792-a17fe716e820" containerName="istio-proxy" Nov 03 13:38:10.945178 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:10.945151 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-8f8aa8/rate-test-model-8f8aa8-kserve-7786f67f6f-mbztf" Nov 03 13:38:10.959203 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:10.958139 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w0-8f8aa8/rate-test-model-8f8aa8-kserve-7786f67f6f-mbztf"] Nov 03 13:38:10.959828 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:10.959799 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w0-8f8aa8\"/\"kube-root-ca.crt\"" Nov 03 13:38:10.960077 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:10.960059 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w0-8f8aa8\"/\"openshift-service-ca.crt\"" Nov 03 13:38:10.962064 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:10.962041 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w0-8f8aa8\"/\"default-dockercfg-94nwg\"" Nov 03 13:38:10.962274 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:10.962252 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w0-8f8aa8\"/\"rate-test-model-8f8aa8-kserve-self-signed-certs\"" Nov 03 13:38:10.990061 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:10.989975 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5fabbac7-721a-4f35-bd22-e86df0cecc62-tmp-dir\") pod \"rate-test-model-8f8aa8-kserve-7786f67f6f-mbztf\" (UID: \"5fabbac7-721a-4f35-bd22-e86df0cecc62\") " pod="ai-tenant-e2e-shared-a-w0-8f8aa8/rate-test-model-8f8aa8-kserve-7786f67f6f-mbztf" Nov 03 13:38:10.990263 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:10.990111 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5fabbac7-721a-4f35-bd22-e86df0cecc62-tls-certs\") pod \"rate-test-model-8f8aa8-kserve-7786f67f6f-mbztf\" (UID: \"5fabbac7-721a-4f35-bd22-e86df0cecc62\") " pod="ai-tenant-e2e-shared-a-w0-8f8aa8/rate-test-model-8f8aa8-kserve-7786f67f6f-mbztf" Nov 03 13:38:10.990263 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:10.990199 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5fabbac7-721a-4f35-bd22-e86df0cecc62-dshm\") pod \"rate-test-model-8f8aa8-kserve-7786f67f6f-mbztf\" (UID: \"5fabbac7-721a-4f35-bd22-e86df0cecc62\") " pod="ai-tenant-e2e-shared-a-w0-8f8aa8/rate-test-model-8f8aa8-kserve-7786f67f6f-mbztf" Nov 03 13:38:10.990410 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:10.990306 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dmvw6\" (UniqueName: \"kubernetes.io/projected/5fabbac7-721a-4f35-bd22-e86df0cecc62-kube-api-access-dmvw6\") pod \"rate-test-model-8f8aa8-kserve-7786f67f6f-mbztf\" (UID: \"5fabbac7-721a-4f35-bd22-e86df0cecc62\") " pod="ai-tenant-e2e-shared-a-w0-8f8aa8/rate-test-model-8f8aa8-kserve-7786f67f6f-mbztf" Nov 03 13:38:10.990410 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:10.990362 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5fabbac7-721a-4f35-bd22-e86df0cecc62-home\") pod \"rate-test-model-8f8aa8-kserve-7786f67f6f-mbztf\" (UID: \"5fabbac7-721a-4f35-bd22-e86df0cecc62\") " pod="ai-tenant-e2e-shared-a-w0-8f8aa8/rate-test-model-8f8aa8-kserve-7786f67f6f-mbztf" Nov 03 13:38:10.990573 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:10.990553 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5fabbac7-721a-4f35-bd22-e86df0cecc62-model-cache\") pod \"rate-test-model-8f8aa8-kserve-7786f67f6f-mbztf\" (UID: \"5fabbac7-721a-4f35-bd22-e86df0cecc62\") " pod="ai-tenant-e2e-shared-a-w0-8f8aa8/rate-test-model-8f8aa8-kserve-7786f67f6f-mbztf" Nov 03 13:38:11.092524 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:11.092483 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5fabbac7-721a-4f35-bd22-e86df0cecc62-model-cache\") pod \"rate-test-model-8f8aa8-kserve-7786f67f6f-mbztf\" (UID: \"5fabbac7-721a-4f35-bd22-e86df0cecc62\") " pod="ai-tenant-e2e-shared-a-w0-8f8aa8/rate-test-model-8f8aa8-kserve-7786f67f6f-mbztf" Nov 03 13:38:11.092727 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:11.092577 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5fabbac7-721a-4f35-bd22-e86df0cecc62-tmp-dir\") pod \"rate-test-model-8f8aa8-kserve-7786f67f6f-mbztf\" (UID: \"5fabbac7-721a-4f35-bd22-e86df0cecc62\") " pod="ai-tenant-e2e-shared-a-w0-8f8aa8/rate-test-model-8f8aa8-kserve-7786f67f6f-mbztf" Nov 03 13:38:11.092727 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:11.092679 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5fabbac7-721a-4f35-bd22-e86df0cecc62-tls-certs\") pod \"rate-test-model-8f8aa8-kserve-7786f67f6f-mbztf\" (UID: \"5fabbac7-721a-4f35-bd22-e86df0cecc62\") " pod="ai-tenant-e2e-shared-a-w0-8f8aa8/rate-test-model-8f8aa8-kserve-7786f67f6f-mbztf" Nov 03 13:38:11.092853 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:11.092751 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5fabbac7-721a-4f35-bd22-e86df0cecc62-dshm\") pod \"rate-test-model-8f8aa8-kserve-7786f67f6f-mbztf\" (UID: \"5fabbac7-721a-4f35-bd22-e86df0cecc62\") " pod="ai-tenant-e2e-shared-a-w0-8f8aa8/rate-test-model-8f8aa8-kserve-7786f67f6f-mbztf" Nov 03 13:38:11.092853 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:11.092833 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dmvw6\" (UniqueName: \"kubernetes.io/projected/5fabbac7-721a-4f35-bd22-e86df0cecc62-kube-api-access-dmvw6\") pod \"rate-test-model-8f8aa8-kserve-7786f67f6f-mbztf\" (UID: \"5fabbac7-721a-4f35-bd22-e86df0cecc62\") " pod="ai-tenant-e2e-shared-a-w0-8f8aa8/rate-test-model-8f8aa8-kserve-7786f67f6f-mbztf" Nov 03 13:38:11.092973 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:11.092878 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5fabbac7-721a-4f35-bd22-e86df0cecc62-home\") pod \"rate-test-model-8f8aa8-kserve-7786f67f6f-mbztf\" (UID: \"5fabbac7-721a-4f35-bd22-e86df0cecc62\") " pod="ai-tenant-e2e-shared-a-w0-8f8aa8/rate-test-model-8f8aa8-kserve-7786f67f6f-mbztf" Nov 03 13:38:11.093026 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:11.093001 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5fabbac7-721a-4f35-bd22-e86df0cecc62-tmp-dir\") pod \"rate-test-model-8f8aa8-kserve-7786f67f6f-mbztf\" (UID: \"5fabbac7-721a-4f35-bd22-e86df0cecc62\") " pod="ai-tenant-e2e-shared-a-w0-8f8aa8/rate-test-model-8f8aa8-kserve-7786f67f6f-mbztf" Nov 03 13:38:11.094258 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:11.093586 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5fabbac7-721a-4f35-bd22-e86df0cecc62-model-cache\") pod \"rate-test-model-8f8aa8-kserve-7786f67f6f-mbztf\" (UID: \"5fabbac7-721a-4f35-bd22-e86df0cecc62\") " pod="ai-tenant-e2e-shared-a-w0-8f8aa8/rate-test-model-8f8aa8-kserve-7786f67f6f-mbztf" Nov 03 13:38:11.094585 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:11.093880 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5fabbac7-721a-4f35-bd22-e86df0cecc62-home\") pod \"rate-test-model-8f8aa8-kserve-7786f67f6f-mbztf\" (UID: \"5fabbac7-721a-4f35-bd22-e86df0cecc62\") " pod="ai-tenant-e2e-shared-a-w0-8f8aa8/rate-test-model-8f8aa8-kserve-7786f67f6f-mbztf" Nov 03 13:38:11.098352 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:11.096016 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5fabbac7-721a-4f35-bd22-e86df0cecc62-tls-certs\") pod \"rate-test-model-8f8aa8-kserve-7786f67f6f-mbztf\" (UID: \"5fabbac7-721a-4f35-bd22-e86df0cecc62\") " pod="ai-tenant-e2e-shared-a-w0-8f8aa8/rate-test-model-8f8aa8-kserve-7786f67f6f-mbztf" Nov 03 13:38:11.098352 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:11.098003 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5fabbac7-721a-4f35-bd22-e86df0cecc62-dshm\") pod \"rate-test-model-8f8aa8-kserve-7786f67f6f-mbztf\" (UID: \"5fabbac7-721a-4f35-bd22-e86df0cecc62\") " pod="ai-tenant-e2e-shared-a-w0-8f8aa8/rate-test-model-8f8aa8-kserve-7786f67f6f-mbztf" Nov 03 13:38:11.103988 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:11.103960 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dmvw6\" (UniqueName: \"kubernetes.io/projected/5fabbac7-721a-4f35-bd22-e86df0cecc62-kube-api-access-dmvw6\") pod \"rate-test-model-8f8aa8-kserve-7786f67f6f-mbztf\" (UID: \"5fabbac7-721a-4f35-bd22-e86df0cecc62\") " pod="ai-tenant-e2e-shared-a-w0-8f8aa8/rate-test-model-8f8aa8-kserve-7786f67f6f-mbztf" Nov 03 13:38:11.282371 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:11.279613 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-8f8aa8/rate-test-model-8f8aa8-kserve-7786f67f6f-mbztf" Nov 03 13:38:11.802857 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:11.802812 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w0-8f8aa8/rate-test-model-8f8aa8-kserve-7786f67f6f-mbztf"] Nov 03 13:38:12.115911 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:12.115872 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c69ea47-20cb-40de-94fb-add95bd74c10" path="/var/lib/kubelet/pods/3c69ea47-20cb-40de-94fb-add95bd74c10/volumes" Nov 03 13:38:12.116526 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:12.116510 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ec0db42-ee6d-496c-8792-a17fe716e820" path="/var/lib/kubelet/pods/6ec0db42-ee6d-496c-8792-a17fe716e820/volumes" Nov 03 13:38:12.214486 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:12.214430 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-8f8aa8/rate-test-model-8f8aa8-kserve-7786f67f6f-mbztf" event={"ID":"5fabbac7-721a-4f35-bd22-e86df0cecc62","Type":"ContainerStarted","Data":"5605d9b3c7a996f21b2a1c473766ca440a976ce0ca56aa746b80040cca17721c"} Nov 03 13:38:12.214486 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:12.214491 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-8f8aa8/rate-test-model-8f8aa8-kserve-7786f67f6f-mbztf" event={"ID":"5fabbac7-721a-4f35-bd22-e86df0cecc62","Type":"ContainerStarted","Data":"e464dbcc4536d2ceaca73ff3194216de5519155e5750f7e28b3182f7835e4de6"} Nov 03 13:38:12.238308 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:12.238239 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w0-8f8aa8/rate-test-model-8f8aa8-kserve-7786f67f6f-mbztf" podStartSLOduration=2.238224623 podStartE2EDuration="2.238224623s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:38:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:38:12.23481179 +0000 UTC m=+1772.243003085" watchObservedRunningTime="2026-11-03 13:38:12.238224623 +0000 UTC m=+1772.246415899" Nov 03 13:38:12.280213 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:12.280174 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w0-8f8aa8/rate-test-model-8f8aa8-kserve-7786f67f6f-mbztf" Nov 03 13:38:12.562816 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:12.562705 5024 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-c47670/payload-processing-external-model-e2e-extmod-a-c47670-659497kbr" podUID="52b866fe-9c71-40c3-b14a-4674654d638e" containerName="payload-processing" containerID="cri-o://60c6eaad1b5bc456680e7e8b9b64b770d3cdaa8ddfbfa33bf837e796a954b48f" gracePeriod=25 Nov 03 13:38:12.881733 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:12.881703 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-c47670/payload-processing-external-model-e2e-extmod-a-c47670-659497kbr" Nov 03 13:38:12.944924 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:12.944890 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/52b866fe-9c71-40c3-b14a-4674654d638e-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/52b866fe-9c71-40c3-b14a-4674654d638e-routing-overlay-volume\") pod \"52b866fe-9c71-40c3-b14a-4674654d638e\" (UID: \"52b866fe-9c71-40c3-b14a-4674654d638e\") " Nov 03 13:38:12.945072 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:12.944980 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52b866fe-9c71-40c3-b14a-4674654d638e-provider-credentials\" (UniqueName: \"kubernetes.io/projected/52b866fe-9c71-40c3-b14a-4674654d638e-provider-credentials\") pod \"52b866fe-9c71-40c3-b14a-4674654d638e\" (UID: \"52b866fe-9c71-40c3-b14a-4674654d638e\") " Nov 03 13:38:12.945072 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:12.945023 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52b866fe-9c71-40c3-b14a-4674654d638e-kube-api-access-44r8k\" (UniqueName: \"kubernetes.io/projected/52b866fe-9c71-40c3-b14a-4674654d638e-kube-api-access-44r8k\") pod \"52b866fe-9c71-40c3-b14a-4674654d638e\" (UID: \"52b866fe-9c71-40c3-b14a-4674654d638e\") " Nov 03 13:38:12.945072 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:12.945058 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/52b866fe-9c71-40c3-b14a-4674654d638e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52b866fe-9c71-40c3-b14a-4674654d638e-plugins-config-volume\") pod \"52b866fe-9c71-40c3-b14a-4674654d638e\" (UID: \"52b866fe-9c71-40c3-b14a-4674654d638e\") " Nov 03 13:38:12.945460 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:12.945429 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52b866fe-9c71-40c3-b14a-4674654d638e-routing-overlay-volume" pod "52b866fe-9c71-40c3-b14a-4674654d638e" (UID: "52b866fe-9c71-40c3-b14a-4674654d638e"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:38:12.945564 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:12.945550 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52b866fe-9c71-40c3-b14a-4674654d638e-plugins-config-volume" pod "52b866fe-9c71-40c3-b14a-4674654d638e" (UID: "52b866fe-9c71-40c3-b14a-4674654d638e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:38:12.945934 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:12.945916 5024 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/52b866fe-9c71-40c3-b14a-4674654d638e-routing-overlay-volume\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:38:12.945934 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:12.945933 5024 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52b866fe-9c71-40c3-b14a-4674654d638e-plugins-config-volume\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:38:12.947480 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:12.947459 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52b866fe-9c71-40c3-b14a-4674654d638e-provider-credentials" pod "52b866fe-9c71-40c3-b14a-4674654d638e" (UID: "52b866fe-9c71-40c3-b14a-4674654d638e"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:38:12.947791 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:12.947763 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52b866fe-9c71-40c3-b14a-4674654d638e-kube-api-access-44r8k" pod "52b866fe-9c71-40c3-b14a-4674654d638e" (UID: "52b866fe-9c71-40c3-b14a-4674654d638e"). InnerVolumeSpecName "kube-api-access-44r8k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:38:13.047041 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:13.047006 5024 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/52b866fe-9c71-40c3-b14a-4674654d638e-provider-credentials\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:38:13.047041 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:13.047037 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-44r8k\" (UniqueName: \"kubernetes.io/projected/52b866fe-9c71-40c3-b14a-4674654d638e-kube-api-access-44r8k\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:38:13.276384 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:13.272871 5024 generic.go:356] "Generic (PLEG): container finished" podID="52b866fe-9c71-40c3-b14a-4674654d638e" containerID="60c6eaad1b5bc456680e7e8b9b64b770d3cdaa8ddfbfa33bf837e796a954b48f" exitCode=0 Nov 03 13:38:13.276384 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:13.274280 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-c47670/payload-processing-external-model-e2e-extmod-a-c47670-659497kbr" Nov 03 13:38:13.281418 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:13.281381 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-c47670/payload-processing-external-model-e2e-extmod-a-c47670-659497kbr" event={"ID":"52b866fe-9c71-40c3-b14a-4674654d638e","Type":"ContainerDied","Data":"60c6eaad1b5bc456680e7e8b9b64b770d3cdaa8ddfbfa33bf837e796a954b48f"} Nov 03 13:38:13.282731 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:13.282704 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-c47670/payload-processing-external-model-e2e-extmod-a-c47670-659497kbr" event={"ID":"52b866fe-9c71-40c3-b14a-4674654d638e","Type":"ContainerDied","Data":"f6ae5e17f85b67f0aaf77e89b03af59040bc4bc58112755ed2761d1a8a096023"} Nov 03 13:38:13.282897 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:13.282882 5024 scope.go:122] "RemoveContainer" containerID="60c6eaad1b5bc456680e7e8b9b64b770d3cdaa8ddfbfa33bf837e796a954b48f" Nov 03 13:38:13.324198 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:13.324164 5024 scope.go:122] "RemoveContainer" containerID="60c6eaad1b5bc456680e7e8b9b64b770d3cdaa8ddfbfa33bf837e796a954b48f" Nov 03 13:38:13.324639 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:38:13.324609 5024 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"60c6eaad1b5bc456680e7e8b9b64b770d3cdaa8ddfbfa33bf837e796a954b48f\": container with ID starting with 60c6eaad1b5bc456680e7e8b9b64b770d3cdaa8ddfbfa33bf837e796a954b48f not found: ID does not exist" containerID="60c6eaad1b5bc456680e7e8b9b64b770d3cdaa8ddfbfa33bf837e796a954b48f" Nov 03 13:38:13.324743 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:13.324653 5024 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60c6eaad1b5bc456680e7e8b9b64b770d3cdaa8ddfbfa33bf837e796a954b48f"} err="failed to get container status \"60c6eaad1b5bc456680e7e8b9b64b770d3cdaa8ddfbfa33bf837e796a954b48f\": rpc error: code = NotFound desc = could not find container \"60c6eaad1b5bc456680e7e8b9b64b770d3cdaa8ddfbfa33bf837e796a954b48f\": container with ID starting with 60c6eaad1b5bc456680e7e8b9b64b770d3cdaa8ddfbfa33bf837e796a954b48f not found: ID does not exist" Nov 03 13:38:13.324958 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:13.324941 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-c47670/payload-processing-external-model-e2e-extmod-a-c47670-659497kbr"] Nov 03 13:38:13.338114 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:13.338068 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-c47670/payload-processing-external-model-e2e-extmod-a-c47670-659497kbr"] Nov 03 13:38:14.111336 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:14.111299 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="52b866fe-9c71-40c3-b14a-4674654d638e" path="/var/lib/kubelet/pods/52b866fe-9c71-40c3-b14a-4674654d638e/volumes" Nov 03 13:38:15.184028 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:15.183302 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-7e439d04-ffd9b45f6-lt8m6"] Nov 03 13:38:15.184028 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:15.183555 5024 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7e439d04-ffd9b45f6-lt8m6" podUID="cfb86542-1303-482c-a6ce-bee9bc81da27" containerName="maas-api" containerID="cri-o://4dcb14124747e301c972f6b37212b6e22ed87aa250ec6801cc9799cf045faea7" gracePeriod=30 Nov 03 13:38:15.898565 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:15.898529 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7e439d04-ffd9b45f6-lt8m6" Nov 03 13:38:15.966054 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:15.966018 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cfb86542-1303-482c-a6ce-bee9bc81da27-kube-api-access-zf7cg\" (UniqueName: \"kubernetes.io/projected/cfb86542-1303-482c-a6ce-bee9bc81da27-kube-api-access-zf7cg\") pod \"cfb86542-1303-482c-a6ce-bee9bc81da27\" (UID: \"cfb86542-1303-482c-a6ce-bee9bc81da27\") " Nov 03 13:38:15.966231 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:15.966163 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cfb86542-1303-482c-a6ce-bee9bc81da27-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cfb86542-1303-482c-a6ce-bee9bc81da27-openshift-service-ca\") pod \"cfb86542-1303-482c-a6ce-bee9bc81da27\" (UID: \"cfb86542-1303-482c-a6ce-bee9bc81da27\") " Nov 03 13:38:15.966270 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:15.966234 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cfb86542-1303-482c-a6ce-bee9bc81da27-metrics-certs\" (UniqueName: \"kubernetes.io/secret/cfb86542-1303-482c-a6ce-bee9bc81da27-metrics-certs\") pod \"cfb86542-1303-482c-a6ce-bee9bc81da27\" (UID: \"cfb86542-1303-482c-a6ce-bee9bc81da27\") " Nov 03 13:38:15.966395 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:15.966378 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cfb86542-1303-482c-a6ce-bee9bc81da27-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cfb86542-1303-482c-a6ce-bee9bc81da27-maas-api-tls\") pod \"cfb86542-1303-482c-a6ce-bee9bc81da27\" (UID: \"cfb86542-1303-482c-a6ce-bee9bc81da27\") " Nov 03 13:38:15.966710 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:15.966674 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cfb86542-1303-482c-a6ce-bee9bc81da27-openshift-service-ca" pod "cfb86542-1303-482c-a6ce-bee9bc81da27" (UID: "cfb86542-1303-482c-a6ce-bee9bc81da27"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:38:15.967081 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:15.967060 5024 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cfb86542-1303-482c-a6ce-bee9bc81da27-openshift-service-ca\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:38:15.968253 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:15.968227 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cfb86542-1303-482c-a6ce-bee9bc81da27-kube-api-access-zf7cg" pod "cfb86542-1303-482c-a6ce-bee9bc81da27" (UID: "cfb86542-1303-482c-a6ce-bee9bc81da27"). InnerVolumeSpecName "kube-api-access-zf7cg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:38:15.968486 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:15.968457 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cfb86542-1303-482c-a6ce-bee9bc81da27-maas-api-tls" pod "cfb86542-1303-482c-a6ce-bee9bc81da27" (UID: "cfb86542-1303-482c-a6ce-bee9bc81da27"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 13:38:15.968531 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:15.968467 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cfb86542-1303-482c-a6ce-bee9bc81da27-metrics-certs" pod "cfb86542-1303-482c-a6ce-bee9bc81da27" (UID: "cfb86542-1303-482c-a6ce-bee9bc81da27"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 13:38:16.068985 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:16.068946 5024 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cfb86542-1303-482c-a6ce-bee9bc81da27-metrics-certs\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:38:16.068985 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:16.068979 5024 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cfb86542-1303-482c-a6ce-bee9bc81da27-maas-api-tls\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:38:16.068985 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:16.068989 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zf7cg\" (UniqueName: \"kubernetes.io/projected/cfb86542-1303-482c-a6ce-bee9bc81da27-kube-api-access-zf7cg\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:38:16.379364 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:16.378459 5024 generic.go:356] "Generic (PLEG): container finished" podID="cfb86542-1303-482c-a6ce-bee9bc81da27" containerID="4dcb14124747e301c972f6b37212b6e22ed87aa250ec6801cc9799cf045faea7" exitCode=0 Nov 03 13:38:16.379364 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:16.378620 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7e439d04-ffd9b45f6-lt8m6" event={"ID":"cfb86542-1303-482c-a6ce-bee9bc81da27","Type":"ContainerDied","Data":"4dcb14124747e301c972f6b37212b6e22ed87aa250ec6801cc9799cf045faea7"} Nov 03 13:38:16.379364 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:16.378668 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7e439d04-ffd9b45f6-lt8m6" event={"ID":"cfb86542-1303-482c-a6ce-bee9bc81da27","Type":"ContainerDied","Data":"379075d5f585fb7d526e85720b5a4ccda1c02e11d0085d32b195fa2e10219136"} Nov 03 13:38:16.379364 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:16.378695 5024 scope.go:122] "RemoveContainer" containerID="4dcb14124747e301c972f6b37212b6e22ed87aa250ec6801cc9799cf045faea7" Nov 03 13:38:16.379364 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:16.378839 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7e439d04-ffd9b45f6-lt8m6" Nov 03 13:38:16.470396 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:16.470356 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-7e439d04-ffd9b45f6-lt8m6"] Nov 03 13:38:16.474897 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:16.474864 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-7e439d04-ffd9b45f6-lt8m6"] Nov 03 13:38:16.496907 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:16.496881 5024 scope.go:122] "RemoveContainer" containerID="4dcb14124747e301c972f6b37212b6e22ed87aa250ec6801cc9799cf045faea7" Nov 03 13:38:16.497457 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:38:16.497425 5024 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4dcb14124747e301c972f6b37212b6e22ed87aa250ec6801cc9799cf045faea7\": container with ID starting with 4dcb14124747e301c972f6b37212b6e22ed87aa250ec6801cc9799cf045faea7 not found: ID does not exist" containerID="4dcb14124747e301c972f6b37212b6e22ed87aa250ec6801cc9799cf045faea7" Nov 03 13:38:16.497630 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:16.497601 5024 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4dcb14124747e301c972f6b37212b6e22ed87aa250ec6801cc9799cf045faea7"} err="failed to get container status \"4dcb14124747e301c972f6b37212b6e22ed87aa250ec6801cc9799cf045faea7\": rpc error: code = NotFound desc = could not find container \"4dcb14124747e301c972f6b37212b6e22ed87aa250ec6801cc9799cf045faea7\": container with ID starting with 4dcb14124747e301c972f6b37212b6e22ed87aa250ec6801cc9799cf045faea7 not found: ID does not exist" Nov 03 13:38:16.721963 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:16.721881 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-6f33a0-d8c4dd6f6-zxfcz"] Nov 03 13:38:16.722938 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:16.722911 5024 cpu_manager.go:414] "containerMap: removing container" podUID="3c69ea47-20cb-40de-94fb-add95bd74c10" containerName="istio-proxy" Nov 03 13:38:16.722938 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:16.722937 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c69ea47-20cb-40de-94fb-add95bd74c10" containerName="istio-proxy" Nov 03 13:38:16.723107 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:16.722978 5024 cpu_manager.go:414] "containerMap: removing container" podUID="cfb86542-1303-482c-a6ce-bee9bc81da27" containerName="maas-api" Nov 03 13:38:16.723107 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:16.722986 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cfb86542-1303-482c-a6ce-bee9bc81da27" containerName="maas-api" Nov 03 13:38:16.723107 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:16.723043 5024 cpu_manager.go:414] "containerMap: removing container" podUID="52b866fe-9c71-40c3-b14a-4674654d638e" containerName="payload-processing" Nov 03 13:38:16.723107 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:16.723053 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52b866fe-9c71-40c3-b14a-4674654d638e" containerName="payload-processing" Nov 03 13:38:16.723340 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:16.723226 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="cfb86542-1303-482c-a6ce-bee9bc81da27" containerName="maas-api" Nov 03 13:38:16.723340 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:16.723247 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c69ea47-20cb-40de-94fb-add95bd74c10" containerName="istio-proxy" Nov 03 13:38:16.723340 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:16.723259 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="52b866fe-9c71-40c3-b14a-4674654d638e" containerName="payload-processing" Nov 03 13:38:16.732369 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:16.732335 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-6f33a0-d8c4dd6f6-zxfcz" Nov 03 13:38:16.747964 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:16.746904 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-6f33a0-d8c4dd6f6-zxfcz"] Nov 03 13:38:16.756467 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:16.756432 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-explicit-6f33a0-serving-cert\"" Nov 03 13:38:16.811769 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:16.811725 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l66vz\" (UniqueName: \"kubernetes.io/projected/9d4a421b-18cf-4c67-8b5e-31ad51dd01aa-kube-api-access-l66vz\") pod \"maas-api-e2e-explicit-6f33a0-d8c4dd6f6-zxfcz\" (UID: \"9d4a421b-18cf-4c67-8b5e-31ad51dd01aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-6f33a0-d8c4dd6f6-zxfcz" Nov 03 13:38:16.811963 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:16.811786 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9d4a421b-18cf-4c67-8b5e-31ad51dd01aa-maas-api-tls\") pod \"maas-api-e2e-explicit-6f33a0-d8c4dd6f6-zxfcz\" (UID: \"9d4a421b-18cf-4c67-8b5e-31ad51dd01aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-6f33a0-d8c4dd6f6-zxfcz" Nov 03 13:38:16.811963 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:16.811873 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d4a421b-18cf-4c67-8b5e-31ad51dd01aa-metrics-certs\") pod \"maas-api-e2e-explicit-6f33a0-d8c4dd6f6-zxfcz\" (UID: \"9d4a421b-18cf-4c67-8b5e-31ad51dd01aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-6f33a0-d8c4dd6f6-zxfcz" Nov 03 13:38:16.812045 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:16.811999 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d4a421b-18cf-4c67-8b5e-31ad51dd01aa-openshift-service-ca\") pod \"maas-api-e2e-explicit-6f33a0-d8c4dd6f6-zxfcz\" (UID: \"9d4a421b-18cf-4c67-8b5e-31ad51dd01aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-6f33a0-d8c4dd6f6-zxfcz" Nov 03 13:38:16.915457 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:16.915406 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l66vz\" (UniqueName: \"kubernetes.io/projected/9d4a421b-18cf-4c67-8b5e-31ad51dd01aa-kube-api-access-l66vz\") pod \"maas-api-e2e-explicit-6f33a0-d8c4dd6f6-zxfcz\" (UID: \"9d4a421b-18cf-4c67-8b5e-31ad51dd01aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-6f33a0-d8c4dd6f6-zxfcz" Nov 03 13:38:16.915650 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:16.915517 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9d4a421b-18cf-4c67-8b5e-31ad51dd01aa-maas-api-tls\") pod \"maas-api-e2e-explicit-6f33a0-d8c4dd6f6-zxfcz\" (UID: \"9d4a421b-18cf-4c67-8b5e-31ad51dd01aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-6f33a0-d8c4dd6f6-zxfcz" Nov 03 13:38:16.915650 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:16.915592 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d4a421b-18cf-4c67-8b5e-31ad51dd01aa-metrics-certs\") pod \"maas-api-e2e-explicit-6f33a0-d8c4dd6f6-zxfcz\" (UID: \"9d4a421b-18cf-4c67-8b5e-31ad51dd01aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-6f33a0-d8c4dd6f6-zxfcz" Nov 03 13:38:16.915817 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:16.915784 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d4a421b-18cf-4c67-8b5e-31ad51dd01aa-openshift-service-ca\") pod \"maas-api-e2e-explicit-6f33a0-d8c4dd6f6-zxfcz\" (UID: \"9d4a421b-18cf-4c67-8b5e-31ad51dd01aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-6f33a0-d8c4dd6f6-zxfcz" Nov 03 13:38:16.929894 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:16.919414 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d4a421b-18cf-4c67-8b5e-31ad51dd01aa-openshift-service-ca\") pod \"maas-api-e2e-explicit-6f33a0-d8c4dd6f6-zxfcz\" (UID: \"9d4a421b-18cf-4c67-8b5e-31ad51dd01aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-6f33a0-d8c4dd6f6-zxfcz" Nov 03 13:38:16.983194 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:16.983101 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l66vz\" (UniqueName: \"kubernetes.io/projected/9d4a421b-18cf-4c67-8b5e-31ad51dd01aa-kube-api-access-l66vz\") pod \"maas-api-e2e-explicit-6f33a0-d8c4dd6f6-zxfcz\" (UID: \"9d4a421b-18cf-4c67-8b5e-31ad51dd01aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-6f33a0-d8c4dd6f6-zxfcz" Nov 03 13:38:16.983815 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:16.983780 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9d4a421b-18cf-4c67-8b5e-31ad51dd01aa-maas-api-tls\") pod \"maas-api-e2e-explicit-6f33a0-d8c4dd6f6-zxfcz\" (UID: \"9d4a421b-18cf-4c67-8b5e-31ad51dd01aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-6f33a0-d8c4dd6f6-zxfcz" Nov 03 13:38:16.983950 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:16.983916 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d4a421b-18cf-4c67-8b5e-31ad51dd01aa-metrics-certs\") pod \"maas-api-e2e-explicit-6f33a0-d8c4dd6f6-zxfcz\" (UID: \"9d4a421b-18cf-4c67-8b5e-31ad51dd01aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-6f33a0-d8c4dd6f6-zxfcz" Nov 03 13:38:17.056578 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:17.056544 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-6f33a0-d8c4dd6f6-zxfcz" Nov 03 13:38:17.391803 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:17.391770 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-6f33a0-d8c4dd6f6-zxfcz"] Nov 03 13:38:17.421886 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:38:17.394104 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d4a421b_18cf_4c67_8b5e_31ad51dd01aa.slice/crio-d8bc91092733d0b5678994f474b206aa4c04c3f4ac32e227484a3f89378cba9a WatchSource:0}: Error finding container d8bc91092733d0b5678994f474b206aa4c04c3f4ac32e227484a3f89378cba9a: Status 404 returned error can't find the container with id d8bc91092733d0b5678994f474b206aa4c04c3f4ac32e227484a3f89378cba9a Nov 03 13:38:18.113713 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:18.113669 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cfb86542-1303-482c-a6ce-bee9bc81da27" path="/var/lib/kubelet/pods/cfb86542-1303-482c-a6ce-bee9bc81da27/volumes" Nov 03 13:38:18.400366 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:18.400324 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-6f33a0-d8c4dd6f6-zxfcz" event={"ID":"9d4a421b-18cf-4c67-8b5e-31ad51dd01aa","Type":"ContainerStarted","Data":"4059ef754e5e73dc76630e3d492f3c99dae2ee0d646dbc6ac762dc91ae91a7da"} Nov 03 13:38:18.400793 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:18.400399 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-6f33a0-d8c4dd6f6-zxfcz" Nov 03 13:38:18.400793 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:18.400424 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-6f33a0-d8c4dd6f6-zxfcz" event={"ID":"9d4a421b-18cf-4c67-8b5e-31ad51dd01aa","Type":"ContainerStarted","Data":"d8bc91092733d0b5678994f474b206aa4c04c3f4ac32e227484a3f89378cba9a"} Nov 03 13:38:18.429232 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:18.429175 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-6f33a0-d8c4dd6f6-zxfcz" podStartSLOduration=1.788643493 podStartE2EDuration="2.429159734s" totalImagesPullingTime="640.516241ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:38:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:38:18.428061102 +0000 UTC m=+1778.436252378" watchObservedRunningTime="2026-11-03 13:38:18.429159734 +0000 UTC m=+1778.437351009" Nov 03 13:38:21.280204 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:21.280161 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w0-8f8aa8/rate-test-model-8f8aa8-kserve-7786f67f6f-mbztf" Nov 03 13:38:21.293256 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:21.293222 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w0-8f8aa8/rate-test-model-8f8aa8-kserve-7786f67f6f-mbztf" Nov 03 13:38:21.461727 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:21.461589 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w0-8f8aa8/rate-test-model-8f8aa8-kserve-7786f67f6f-mbztf" Nov 03 13:38:21.877086 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:21.877044 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-7e439d04-openshift-default-6ff95548d9-nh4cq"] Nov 03 13:38:21.877374 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:21.877324 5024 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-7e439d04-openshift-default-6ff95548d9-nh4cq" podUID="d803a251-b682-4e3e-b636-92fbe8d99c64" containerName="istio-proxy" containerID="cri-o://f6b31f5d82254e5c941ba580acabd1e344fbb1eb1048bc02fca7c08a8bae2b22" gracePeriod=30 Nov 03 13:38:24.439768 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:24.439732 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-6f33a0-d8c4dd6f6-zxfcz" Nov 03 13:38:26.429869 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:26.429831 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-zzmwf"] Nov 03 13:38:26.430432 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:26.430044 5024 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-zzmwf" podUID="e65de701-9327-4af6-a7f7-e0d6adf6ffcd" containerName="istio-proxy" containerID="cri-o://e3ff084f3e19c0fb25357ff26d959383a28737282564937224b3eb7cb3ef14d6" gracePeriod=30 Nov 03 13:38:27.249059 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:27.249028 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-7e439d04-openshift-default-6ff95548d9-nh4cq" Nov 03 13:38:27.382496 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:27.382448 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d803a251-b682-4e3e-b636-92fbe8d99c64-kube-api-access-tmmc7\" (UniqueName: \"kubernetes.io/projected/d803a251-b682-4e3e-b636-92fbe8d99c64-kube-api-access-tmmc7\") pod \"d803a251-b682-4e3e-b636-92fbe8d99c64\" (UID: \"d803a251-b682-4e3e-b636-92fbe8d99c64\") " Nov 03 13:38:27.382721 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:27.382543 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d803a251-b682-4e3e-b636-92fbe8d99c64-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d803a251-b682-4e3e-b636-92fbe8d99c64-workload-socket\") pod \"d803a251-b682-4e3e-b636-92fbe8d99c64\" (UID: \"d803a251-b682-4e3e-b636-92fbe8d99c64\") " Nov 03 13:38:27.382859 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:27.382842 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d803a251-b682-4e3e-b636-92fbe8d99c64-istio-token\" (UniqueName: \"kubernetes.io/projected/d803a251-b682-4e3e-b636-92fbe8d99c64-istio-token\") pod \"d803a251-b682-4e3e-b636-92fbe8d99c64\" (UID: \"d803a251-b682-4e3e-b636-92fbe8d99c64\") " Nov 03 13:38:27.382950 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:27.382935 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d803a251-b682-4e3e-b636-92fbe8d99c64-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d803a251-b682-4e3e-b636-92fbe8d99c64-istio-envoy\") pod \"d803a251-b682-4e3e-b636-92fbe8d99c64\" (UID: \"d803a251-b682-4e3e-b636-92fbe8d99c64\") " Nov 03 13:38:27.383021 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:27.382986 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d803a251-b682-4e3e-b636-92fbe8d99c64-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d803a251-b682-4e3e-b636-92fbe8d99c64-istio-data\") pod \"d803a251-b682-4e3e-b636-92fbe8d99c64\" (UID: \"d803a251-b682-4e3e-b636-92fbe8d99c64\") " Nov 03 13:38:27.383079 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:27.383038 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d803a251-b682-4e3e-b636-92fbe8d99c64-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d803a251-b682-4e3e-b636-92fbe8d99c64-credential-socket\") pod \"d803a251-b682-4e3e-b636-92fbe8d99c64\" (UID: \"d803a251-b682-4e3e-b636-92fbe8d99c64\") " Nov 03 13:38:27.383132 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:27.383107 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d803a251-b682-4e3e-b636-92fbe8d99c64-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d803a251-b682-4e3e-b636-92fbe8d99c64-istio-podinfo\") pod \"d803a251-b682-4e3e-b636-92fbe8d99c64\" (UID: \"d803a251-b682-4e3e-b636-92fbe8d99c64\") " Nov 03 13:38:27.383183 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:27.383143 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d803a251-b682-4e3e-b636-92fbe8d99c64-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d803a251-b682-4e3e-b636-92fbe8d99c64-workload-certs\") pod \"d803a251-b682-4e3e-b636-92fbe8d99c64\" (UID: \"d803a251-b682-4e3e-b636-92fbe8d99c64\") " Nov 03 13:38:27.383183 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:27.383174 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d803a251-b682-4e3e-b636-92fbe8d99c64-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d803a251-b682-4e3e-b636-92fbe8d99c64-istiod-ca-cert\") pod \"d803a251-b682-4e3e-b636-92fbe8d99c64\" (UID: \"d803a251-b682-4e3e-b636-92fbe8d99c64\") " Nov 03 13:38:27.384318 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:27.384255 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d803a251-b682-4e3e-b636-92fbe8d99c64-istiod-ca-cert" pod "d803a251-b682-4e3e-b636-92fbe8d99c64" (UID: "d803a251-b682-4e3e-b636-92fbe8d99c64"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:38:27.384542 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:27.384522 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d803a251-b682-4e3e-b636-92fbe8d99c64-istio-data" pod "d803a251-b682-4e3e-b636-92fbe8d99c64" (UID: "d803a251-b682-4e3e-b636-92fbe8d99c64"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:38:27.384712 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:27.384695 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d803a251-b682-4e3e-b636-92fbe8d99c64-credential-socket" pod "d803a251-b682-4e3e-b636-92fbe8d99c64" (UID: "d803a251-b682-4e3e-b636-92fbe8d99c64"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:38:27.385331 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:27.385228 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d803a251-b682-4e3e-b636-92fbe8d99c64-workload-socket" pod "d803a251-b682-4e3e-b636-92fbe8d99c64" (UID: "d803a251-b682-4e3e-b636-92fbe8d99c64"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:38:27.385972 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:27.385588 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d803a251-b682-4e3e-b636-92fbe8d99c64-workload-certs" pod "d803a251-b682-4e3e-b636-92fbe8d99c64" (UID: "d803a251-b682-4e3e-b636-92fbe8d99c64"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:38:27.390708 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:27.390456 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d803a251-b682-4e3e-b636-92fbe8d99c64-istio-podinfo" pod "d803a251-b682-4e3e-b636-92fbe8d99c64" (UID: "d803a251-b682-4e3e-b636-92fbe8d99c64"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 13:38:27.390708 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:27.390456 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d803a251-b682-4e3e-b636-92fbe8d99c64-istio-envoy" pod "d803a251-b682-4e3e-b636-92fbe8d99c64" (UID: "d803a251-b682-4e3e-b636-92fbe8d99c64"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:38:27.390708 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:27.390633 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d803a251-b682-4e3e-b636-92fbe8d99c64-kube-api-access-tmmc7" pod "d803a251-b682-4e3e-b636-92fbe8d99c64" (UID: "d803a251-b682-4e3e-b636-92fbe8d99c64"). InnerVolumeSpecName "kube-api-access-tmmc7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:38:27.390708 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:27.390685 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d803a251-b682-4e3e-b636-92fbe8d99c64-istio-token" pod "d803a251-b682-4e3e-b636-92fbe8d99c64" (UID: "d803a251-b682-4e3e-b636-92fbe8d99c64"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:38:27.506570 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:27.505891 5024 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d803a251-b682-4e3e-b636-92fbe8d99c64-credential-socket\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:38:27.506570 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:27.505958 5024 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d803a251-b682-4e3e-b636-92fbe8d99c64-istio-podinfo\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:38:27.506570 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:27.505976 5024 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d803a251-b682-4e3e-b636-92fbe8d99c64-workload-certs\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:38:27.506570 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:27.505990 5024 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d803a251-b682-4e3e-b636-92fbe8d99c64-istiod-ca-cert\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:38:27.506570 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:27.506005 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tmmc7\" (UniqueName: \"kubernetes.io/projected/d803a251-b682-4e3e-b636-92fbe8d99c64-kube-api-access-tmmc7\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:38:27.506570 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:27.506032 5024 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d803a251-b682-4e3e-b636-92fbe8d99c64-workload-socket\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:38:27.506570 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:27.506045 5024 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d803a251-b682-4e3e-b636-92fbe8d99c64-istio-token\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:38:27.506570 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:27.506057 5024 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d803a251-b682-4e3e-b636-92fbe8d99c64-istio-envoy\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:38:27.506570 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:27.506069 5024 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d803a251-b682-4e3e-b636-92fbe8d99c64-istio-data\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:38:27.528347 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:27.527255 5024 generic.go:356] "Generic (PLEG): container finished" podID="d803a251-b682-4e3e-b636-92fbe8d99c64" containerID="f6b31f5d82254e5c941ba580acabd1e344fbb1eb1048bc02fca7c08a8bae2b22" exitCode=0 Nov 03 13:38:27.528347 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:27.527353 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7e439d04-openshift-default-6ff95548d9-nh4cq" event={"ID":"d803a251-b682-4e3e-b636-92fbe8d99c64","Type":"ContainerDied","Data":"f6b31f5d82254e5c941ba580acabd1e344fbb1eb1048bc02fca7c08a8bae2b22"} Nov 03 13:38:27.528347 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:27.527407 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7e439d04-openshift-default-6ff95548d9-nh4cq" event={"ID":"d803a251-b682-4e3e-b636-92fbe8d99c64","Type":"ContainerDied","Data":"da27de2a8641b82b03433b9c19996cab401abe49ed6fb8922b77c191921c9309"} Nov 03 13:38:27.528347 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:27.527453 5024 scope.go:122] "RemoveContainer" containerID="f6b31f5d82254e5c941ba580acabd1e344fbb1eb1048bc02fca7c08a8bae2b22" Nov 03 13:38:27.528347 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:27.527360 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-7e439d04-openshift-default-6ff95548d9-nh4cq" Nov 03 13:38:27.542924 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:27.542895 5024 scope.go:122] "RemoveContainer" containerID="4dec6fb29c63ccf03e05c3d8d8c49f7e33af7c51bae00ab63511a676b90329c8" Nov 03 13:38:27.557248 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:27.557225 5024 scope.go:122] "RemoveContainer" containerID="f6b31f5d82254e5c941ba580acabd1e344fbb1eb1048bc02fca7c08a8bae2b22" Nov 03 13:38:27.557606 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:38:27.557582 5024 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6b31f5d82254e5c941ba580acabd1e344fbb1eb1048bc02fca7c08a8bae2b22\": container with ID starting with f6b31f5d82254e5c941ba580acabd1e344fbb1eb1048bc02fca7c08a8bae2b22 not found: ID does not exist" containerID="f6b31f5d82254e5c941ba580acabd1e344fbb1eb1048bc02fca7c08a8bae2b22" Nov 03 13:38:27.557695 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:27.557620 5024 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6b31f5d82254e5c941ba580acabd1e344fbb1eb1048bc02fca7c08a8bae2b22"} err="failed to get container status \"f6b31f5d82254e5c941ba580acabd1e344fbb1eb1048bc02fca7c08a8bae2b22\": rpc error: code = NotFound desc = could not find container \"f6b31f5d82254e5c941ba580acabd1e344fbb1eb1048bc02fca7c08a8bae2b22\": container with ID starting with f6b31f5d82254e5c941ba580acabd1e344fbb1eb1048bc02fca7c08a8bae2b22 not found: ID does not exist" Nov 03 13:38:27.557695 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:27.557645 5024 scope.go:122] "RemoveContainer" containerID="4dec6fb29c63ccf03e05c3d8d8c49f7e33af7c51bae00ab63511a676b90329c8" Nov 03 13:38:27.557935 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:38:27.557913 5024 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4dec6fb29c63ccf03e05c3d8d8c49f7e33af7c51bae00ab63511a676b90329c8\": container with ID starting with 4dec6fb29c63ccf03e05c3d8d8c49f7e33af7c51bae00ab63511a676b90329c8 not found: ID does not exist" containerID="4dec6fb29c63ccf03e05c3d8d8c49f7e33af7c51bae00ab63511a676b90329c8" Nov 03 13:38:27.557980 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:27.557942 5024 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4dec6fb29c63ccf03e05c3d8d8c49f7e33af7c51bae00ab63511a676b90329c8"} err="failed to get container status \"4dec6fb29c63ccf03e05c3d8d8c49f7e33af7c51bae00ab63511a676b90329c8\": rpc error: code = NotFound desc = could not find container \"4dec6fb29c63ccf03e05c3d8d8c49f7e33af7c51bae00ab63511a676b90329c8\": container with ID starting with 4dec6fb29c63ccf03e05c3d8d8c49f7e33af7c51bae00ab63511a676b90329c8 not found: ID does not exist" Nov 03 13:38:27.570906 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:27.570873 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-7e439d04-openshift-default-6ff95548d9-nh4cq"] Nov 03 13:38:27.574827 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:27.574795 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-7e439d04-openshift-default-6ff95548d9-nh4cq"] Nov 03 13:38:28.152553 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:28.151218 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d803a251-b682-4e3e-b636-92fbe8d99c64" path="/var/lib/kubelet/pods/d803a251-b682-4e3e-b636-92fbe8d99c64/volumes" Nov 03 13:38:31.629335 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:31.628496 5024 generic.go:356] "Generic (PLEG): container finished" podID="e65de701-9327-4af6-a7f7-e0d6adf6ffcd" containerID="e3ff084f3e19c0fb25357ff26d959383a28737282564937224b3eb7cb3ef14d6" exitCode=0 Nov 03 13:38:31.629335 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:31.628581 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-zzmwf" event={"ID":"e65de701-9327-4af6-a7f7-e0d6adf6ffcd","Type":"ContainerDied","Data":"e3ff084f3e19c0fb25357ff26d959383a28737282564937224b3eb7cb3ef14d6"} Nov 03 13:38:31.721241 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:31.721211 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-zzmwf" Nov 03 13:38:31.837827 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:31.837787 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e65de701-9327-4af6-a7f7-e0d6adf6ffcd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e65de701-9327-4af6-a7f7-e0d6adf6ffcd-istiod-ca-cert\") pod \"e65de701-9327-4af6-a7f7-e0d6adf6ffcd\" (UID: \"e65de701-9327-4af6-a7f7-e0d6adf6ffcd\") " Nov 03 13:38:31.837827 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:31.837839 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e65de701-9327-4af6-a7f7-e0d6adf6ffcd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e65de701-9327-4af6-a7f7-e0d6adf6ffcd-istio-data\") pod \"e65de701-9327-4af6-a7f7-e0d6adf6ffcd\" (UID: \"e65de701-9327-4af6-a7f7-e0d6adf6ffcd\") " Nov 03 13:38:31.838095 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:31.837881 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e65de701-9327-4af6-a7f7-e0d6adf6ffcd-kube-api-access-cdzt6\" (UniqueName: \"kubernetes.io/projected/e65de701-9327-4af6-a7f7-e0d6adf6ffcd-kube-api-access-cdzt6\") pod \"e65de701-9327-4af6-a7f7-e0d6adf6ffcd\" (UID: \"e65de701-9327-4af6-a7f7-e0d6adf6ffcd\") " Nov 03 13:38:31.838095 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:31.837953 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e65de701-9327-4af6-a7f7-e0d6adf6ffcd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e65de701-9327-4af6-a7f7-e0d6adf6ffcd-workload-certs\") pod \"e65de701-9327-4af6-a7f7-e0d6adf6ffcd\" (UID: \"e65de701-9327-4af6-a7f7-e0d6adf6ffcd\") " Nov 03 13:38:31.838095 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:31.837999 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e65de701-9327-4af6-a7f7-e0d6adf6ffcd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e65de701-9327-4af6-a7f7-e0d6adf6ffcd-credential-socket\") pod \"e65de701-9327-4af6-a7f7-e0d6adf6ffcd\" (UID: \"e65de701-9327-4af6-a7f7-e0d6adf6ffcd\") " Nov 03 13:38:31.838095 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:31.838030 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e65de701-9327-4af6-a7f7-e0d6adf6ffcd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e65de701-9327-4af6-a7f7-e0d6adf6ffcd-workload-socket\") pod \"e65de701-9327-4af6-a7f7-e0d6adf6ffcd\" (UID: \"e65de701-9327-4af6-a7f7-e0d6adf6ffcd\") " Nov 03 13:38:31.838095 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:31.838077 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e65de701-9327-4af6-a7f7-e0d6adf6ffcd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e65de701-9327-4af6-a7f7-e0d6adf6ffcd-istio-envoy\") pod \"e65de701-9327-4af6-a7f7-e0d6adf6ffcd\" (UID: \"e65de701-9327-4af6-a7f7-e0d6adf6ffcd\") " Nov 03 13:38:31.838370 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:31.838152 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e65de701-9327-4af6-a7f7-e0d6adf6ffcd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e65de701-9327-4af6-a7f7-e0d6adf6ffcd-istio-podinfo\") pod \"e65de701-9327-4af6-a7f7-e0d6adf6ffcd\" (UID: \"e65de701-9327-4af6-a7f7-e0d6adf6ffcd\") " Nov 03 13:38:31.838370 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:31.838223 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e65de701-9327-4af6-a7f7-e0d6adf6ffcd-istio-data" pod "e65de701-9327-4af6-a7f7-e0d6adf6ffcd" (UID: "e65de701-9327-4af6-a7f7-e0d6adf6ffcd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:38:31.838370 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:31.838299 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e65de701-9327-4af6-a7f7-e0d6adf6ffcd-istiod-ca-cert" pod "e65de701-9327-4af6-a7f7-e0d6adf6ffcd" (UID: "e65de701-9327-4af6-a7f7-e0d6adf6ffcd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:38:31.838370 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:31.838359 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e65de701-9327-4af6-a7f7-e0d6adf6ffcd-istio-token\" (UniqueName: \"kubernetes.io/projected/e65de701-9327-4af6-a7f7-e0d6adf6ffcd-istio-token\") pod \"e65de701-9327-4af6-a7f7-e0d6adf6ffcd\" (UID: \"e65de701-9327-4af6-a7f7-e0d6adf6ffcd\") " Nov 03 13:38:31.838574 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:31.838370 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e65de701-9327-4af6-a7f7-e0d6adf6ffcd-workload-certs" pod "e65de701-9327-4af6-a7f7-e0d6adf6ffcd" (UID: "e65de701-9327-4af6-a7f7-e0d6adf6ffcd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:38:31.838574 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:31.838397 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e65de701-9327-4af6-a7f7-e0d6adf6ffcd-credential-socket" pod "e65de701-9327-4af6-a7f7-e0d6adf6ffcd" (UID: "e65de701-9327-4af6-a7f7-e0d6adf6ffcd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:38:31.838574 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:31.838455 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e65de701-9327-4af6-a7f7-e0d6adf6ffcd-workload-socket" pod "e65de701-9327-4af6-a7f7-e0d6adf6ffcd" (UID: "e65de701-9327-4af6-a7f7-e0d6adf6ffcd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:38:31.839189 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:31.839143 5024 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e65de701-9327-4af6-a7f7-e0d6adf6ffcd-istiod-ca-cert\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:38:31.839189 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:31.839172 5024 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e65de701-9327-4af6-a7f7-e0d6adf6ffcd-istio-data\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:38:31.839189 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:31.839187 5024 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e65de701-9327-4af6-a7f7-e0d6adf6ffcd-workload-certs\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:38:31.839454 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:31.839204 5024 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e65de701-9327-4af6-a7f7-e0d6adf6ffcd-credential-socket\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:38:31.839454 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:31.839218 5024 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e65de701-9327-4af6-a7f7-e0d6adf6ffcd-workload-socket\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:38:31.840493 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:31.840464 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e65de701-9327-4af6-a7f7-e0d6adf6ffcd-kube-api-access-cdzt6" pod "e65de701-9327-4af6-a7f7-e0d6adf6ffcd" (UID: "e65de701-9327-4af6-a7f7-e0d6adf6ffcd"). InnerVolumeSpecName "kube-api-access-cdzt6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:38:31.840592 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:31.840560 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e65de701-9327-4af6-a7f7-e0d6adf6ffcd-istio-envoy" pod "e65de701-9327-4af6-a7f7-e0d6adf6ffcd" (UID: "e65de701-9327-4af6-a7f7-e0d6adf6ffcd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:38:31.840670 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:31.840656 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e65de701-9327-4af6-a7f7-e0d6adf6ffcd-istio-token" pod "e65de701-9327-4af6-a7f7-e0d6adf6ffcd" (UID: "e65de701-9327-4af6-a7f7-e0d6adf6ffcd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:38:31.840728 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:31.840693 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e65de701-9327-4af6-a7f7-e0d6adf6ffcd-istio-podinfo" pod "e65de701-9327-4af6-a7f7-e0d6adf6ffcd" (UID: "e65de701-9327-4af6-a7f7-e0d6adf6ffcd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 13:38:31.946820 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:31.946657 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cdzt6\" (UniqueName: \"kubernetes.io/projected/e65de701-9327-4af6-a7f7-e0d6adf6ffcd-kube-api-access-cdzt6\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:38:31.946820 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:31.946702 5024 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e65de701-9327-4af6-a7f7-e0d6adf6ffcd-istio-envoy\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:38:31.946820 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:31.946718 5024 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e65de701-9327-4af6-a7f7-e0d6adf6ffcd-istio-podinfo\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:38:31.946820 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:31.946734 5024 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e65de701-9327-4af6-a7f7-e0d6adf6ffcd-istio-token\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:38:32.637138 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:32.637107 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-zzmwf" Nov 03 13:38:32.637633 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:32.637142 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-zzmwf" event={"ID":"e65de701-9327-4af6-a7f7-e0d6adf6ffcd","Type":"ContainerDied","Data":"8b4aa5d2e99cb6411b5939e6fd157f5262bc7e1d5ed8b15e3ee684c25a19a950"} Nov 03 13:38:32.637633 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:32.637211 5024 scope.go:122] "RemoveContainer" containerID="e3ff084f3e19c0fb25357ff26d959383a28737282564937224b3eb7cb3ef14d6" Nov 03 13:38:32.656561 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:32.656523 5024 patch_prober.go:29] interesting pod/e2e-extmod-b-7c192b-openshift-default-5c496b978f-zzmwf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 13:38:32.656719 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:32.656627 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-zzmwf" podUID="e65de701-9327-4af6-a7f7-e0d6adf6ffcd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 13:38:32.663861 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:32.663831 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-zzmwf"] Nov 03 13:38:32.668427 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:32.668399 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-zzmwf"] Nov 03 13:38:33.257733 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:33.257692 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-cf696"] Nov 03 13:38:33.258959 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:33.258932 5024 cpu_manager.go:414] "containerMap: removing container" podUID="d803a251-b682-4e3e-b636-92fbe8d99c64" containerName="istio-proxy" Nov 03 13:38:33.259049 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:33.258961 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d803a251-b682-4e3e-b636-92fbe8d99c64" containerName="istio-proxy" Nov 03 13:38:33.259049 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:33.258987 5024 cpu_manager.go:414] "containerMap: removing container" podUID="e65de701-9327-4af6-a7f7-e0d6adf6ffcd" containerName="istio-proxy" Nov 03 13:38:33.259049 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:33.258996 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e65de701-9327-4af6-a7f7-e0d6adf6ffcd" containerName="istio-proxy" Nov 03 13:38:33.259200 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:33.259183 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="d803a251-b682-4e3e-b636-92fbe8d99c64" containerName="istio-proxy" Nov 03 13:38:33.259233 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:33.259207 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="e65de701-9327-4af6-a7f7-e0d6adf6ffcd" containerName="istio-proxy" Nov 03 13:38:33.259233 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:33.259223 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="d803a251-b682-4e3e-b636-92fbe8d99c64" containerName="istio-proxy" Nov 03 13:38:33.265025 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:33.264999 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-cf696" Nov 03 13:38:33.288353 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:33.288319 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-cf696"] Nov 03 13:38:33.389194 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:33.387746 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a69c6f3-da97-4a27-9809-951395987c7c-workload-socket\") pod \"e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-cf696\" (UID: \"2a69c6f3-da97-4a27-9809-951395987c7c\") " pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-cf696" Nov 03 13:38:33.389194 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:33.387926 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a69c6f3-da97-4a27-9809-951395987c7c-istiod-ca-cert\") pod \"e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-cf696\" (UID: \"2a69c6f3-da97-4a27-9809-951395987c7c\") " pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-cf696" Nov 03 13:38:33.389194 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:33.388035 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a69c6f3-da97-4a27-9809-951395987c7c-istio-token\") pod \"e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-cf696\" (UID: \"2a69c6f3-da97-4a27-9809-951395987c7c\") " pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-cf696" Nov 03 13:38:33.389194 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:33.388171 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a69c6f3-da97-4a27-9809-951395987c7c-istio-data\") pod \"e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-cf696\" (UID: \"2a69c6f3-da97-4a27-9809-951395987c7c\") " pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-cf696" Nov 03 13:38:33.389194 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:33.388216 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a69c6f3-da97-4a27-9809-951395987c7c-istio-envoy\") pod \"e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-cf696\" (UID: \"2a69c6f3-da97-4a27-9809-951395987c7c\") " pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-cf696" Nov 03 13:38:33.389194 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:33.388385 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9m9zw\" (UniqueName: \"kubernetes.io/projected/2a69c6f3-da97-4a27-9809-951395987c7c-kube-api-access-9m9zw\") pod \"e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-cf696\" (UID: \"2a69c6f3-da97-4a27-9809-951395987c7c\") " pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-cf696" Nov 03 13:38:33.389194 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:33.388526 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a69c6f3-da97-4a27-9809-951395987c7c-workload-certs\") pod \"e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-cf696\" (UID: \"2a69c6f3-da97-4a27-9809-951395987c7c\") " pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-cf696" Nov 03 13:38:33.389194 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:33.388614 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a69c6f3-da97-4a27-9809-951395987c7c-istio-podinfo\") pod \"e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-cf696\" (UID: \"2a69c6f3-da97-4a27-9809-951395987c7c\") " pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-cf696" Nov 03 13:38:33.389194 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:33.388661 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a69c6f3-da97-4a27-9809-951395987c7c-credential-socket\") pod \"e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-cf696\" (UID: \"2a69c6f3-da97-4a27-9809-951395987c7c\") " pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-cf696" Nov 03 13:38:33.490339 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:33.490253 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a69c6f3-da97-4a27-9809-951395987c7c-istiod-ca-cert\") pod \"e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-cf696\" (UID: \"2a69c6f3-da97-4a27-9809-951395987c7c\") " pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-cf696" Nov 03 13:38:33.490580 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:33.490416 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a69c6f3-da97-4a27-9809-951395987c7c-istio-token\") pod \"e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-cf696\" (UID: \"2a69c6f3-da97-4a27-9809-951395987c7c\") " pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-cf696" Nov 03 13:38:33.490580 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:33.490499 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a69c6f3-da97-4a27-9809-951395987c7c-istio-data\") pod \"e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-cf696\" (UID: \"2a69c6f3-da97-4a27-9809-951395987c7c\") " pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-cf696" Nov 03 13:38:33.490580 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:33.490547 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a69c6f3-da97-4a27-9809-951395987c7c-istio-envoy\") pod \"e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-cf696\" (UID: \"2a69c6f3-da97-4a27-9809-951395987c7c\") " pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-cf696" Nov 03 13:38:33.490754 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:33.490615 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9m9zw\" (UniqueName: \"kubernetes.io/projected/2a69c6f3-da97-4a27-9809-951395987c7c-kube-api-access-9m9zw\") pod \"e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-cf696\" (UID: \"2a69c6f3-da97-4a27-9809-951395987c7c\") " pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-cf696" Nov 03 13:38:33.490754 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:33.490690 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a69c6f3-da97-4a27-9809-951395987c7c-workload-certs\") pod \"e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-cf696\" (UID: \"2a69c6f3-da97-4a27-9809-951395987c7c\") " pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-cf696" Nov 03 13:38:33.490860 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:33.490776 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a69c6f3-da97-4a27-9809-951395987c7c-istio-podinfo\") pod \"e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-cf696\" (UID: \"2a69c6f3-da97-4a27-9809-951395987c7c\") " pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-cf696" Nov 03 13:38:33.490860 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:33.490822 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a69c6f3-da97-4a27-9809-951395987c7c-credential-socket\") pod \"e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-cf696\" (UID: \"2a69c6f3-da97-4a27-9809-951395987c7c\") " pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-cf696" Nov 03 13:38:33.490970 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:33.490898 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a69c6f3-da97-4a27-9809-951395987c7c-workload-socket\") pod \"e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-cf696\" (UID: \"2a69c6f3-da97-4a27-9809-951395987c7c\") " pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-cf696" Nov 03 13:38:33.491399 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:33.491373 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a69c6f3-da97-4a27-9809-951395987c7c-workload-socket\") pod \"e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-cf696\" (UID: \"2a69c6f3-da97-4a27-9809-951395987c7c\") " pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-cf696" Nov 03 13:38:33.492062 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:33.491425 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a69c6f3-da97-4a27-9809-951395987c7c-istio-data\") pod \"e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-cf696\" (UID: \"2a69c6f3-da97-4a27-9809-951395987c7c\") " pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-cf696" Nov 03 13:38:33.492062 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:33.491441 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a69c6f3-da97-4a27-9809-951395987c7c-istiod-ca-cert\") pod \"e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-cf696\" (UID: \"2a69c6f3-da97-4a27-9809-951395987c7c\") " pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-cf696" Nov 03 13:38:33.492062 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:33.491637 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a69c6f3-da97-4a27-9809-951395987c7c-workload-certs\") pod \"e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-cf696\" (UID: \"2a69c6f3-da97-4a27-9809-951395987c7c\") " pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-cf696" Nov 03 13:38:33.492356 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:33.492129 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a69c6f3-da97-4a27-9809-951395987c7c-credential-socket\") pod \"e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-cf696\" (UID: \"2a69c6f3-da97-4a27-9809-951395987c7c\") " pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-cf696" Nov 03 13:38:33.494201 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:33.494162 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a69c6f3-da97-4a27-9809-951395987c7c-istio-envoy\") pod \"e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-cf696\" (UID: \"2a69c6f3-da97-4a27-9809-951395987c7c\") " pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-cf696" Nov 03 13:38:33.494588 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:33.494559 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a69c6f3-da97-4a27-9809-951395987c7c-istio-podinfo\") pod \"e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-cf696\" (UID: \"2a69c6f3-da97-4a27-9809-951395987c7c\") " pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-cf696" Nov 03 13:38:33.511366 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:33.506437 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9m9zw\" (UniqueName: \"kubernetes.io/projected/2a69c6f3-da97-4a27-9809-951395987c7c-kube-api-access-9m9zw\") pod \"e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-cf696\" (UID: \"2a69c6f3-da97-4a27-9809-951395987c7c\") " pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-cf696" Nov 03 13:38:33.527044 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:33.525827 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a69c6f3-da97-4a27-9809-951395987c7c-istio-token\") pod \"e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-cf696\" (UID: \"2a69c6f3-da97-4a27-9809-951395987c7c\") " pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-cf696" Nov 03 13:38:33.592046 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:33.592010 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-cf696" Nov 03 13:38:33.907365 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:38:33.906347 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a69c6f3_da97_4a27_9809_951395987c7c.slice/crio-eac38ca0c29438663a7b42c7c715a6b74b76043efff869d90f12db8aa03210df WatchSource:0}: Error finding container eac38ca0c29438663a7b42c7c715a6b74b76043efff869d90f12db8aa03210df: Status 404 returned error can't find the container with id eac38ca0c29438663a7b42c7c715a6b74b76043efff869d90f12db8aa03210df Nov 03 13:38:33.907829 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:33.907692 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-cf696"] Nov 03 13:38:33.910808 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:33.910755 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:38:33.910918 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:33.910844 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:38:33.910918 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:33.910886 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:38:34.463309 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:34.463157 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e65de701-9327-4af6-a7f7-e0d6adf6ffcd" path="/var/lib/kubelet/pods/e65de701-9327-4af6-a7f7-e0d6adf6ffcd/volumes" Nov 03 13:38:34.741862 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:34.741750 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-cf696" event={"ID":"2a69c6f3-da97-4a27-9809-951395987c7c","Type":"ContainerStarted","Data":"a3d9337fc97b523e5eb2e7cbb8c9477d8cb12da0c5c4f282784bc9d62b52283c"} Nov 03 13:38:34.741862 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:34.741827 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-cf696" event={"ID":"2a69c6f3-da97-4a27-9809-951395987c7c","Type":"ContainerStarted","Data":"eac38ca0c29438663a7b42c7c715a6b74b76043efff869d90f12db8aa03210df"} Nov 03 13:38:34.770209 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:34.770142 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-cf696" podStartSLOduration=1.770122002 podStartE2EDuration="1.770122002s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:38:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:38:34.767530044 +0000 UTC m=+1794.775721352" watchObservedRunningTime="2026-11-03 13:38:34.770122002 +0000 UTC m=+1794.778313279" Nov 03 13:38:35.595308 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:35.594447 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-cf696" Nov 03 13:38:35.614368 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:35.612463 5024 patch_prober.go:29] interesting pod/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-cf696 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body= Nov 03 13:38:35.614368 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:35.612575 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-cf696" podUID="2a69c6f3-da97-4a27-9809-951395987c7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" Nov 03 13:38:36.599689 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:36.599640 5024 patch_prober.go:29] interesting pod/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-cf696 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body= Nov 03 13:38:36.600369 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:36.600324 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-cf696" podUID="2a69c6f3-da97-4a27-9809-951395987c7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" Nov 03 13:38:37.261662 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:37.261622 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-explicit-6f33a0/explicit-model-6f33a0-kserve-5478dcfc45-5gr5q"] Nov 03 13:38:37.262698 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:37.262670 5024 cpu_manager.go:414] "containerMap: removing container" podUID="d803a251-b682-4e3e-b636-92fbe8d99c64" containerName="istio-proxy" Nov 03 13:38:37.262698 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:37.262696 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d803a251-b682-4e3e-b636-92fbe8d99c64" containerName="istio-proxy" Nov 03 13:38:37.268895 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:37.268862 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-6f33a0/explicit-model-6f33a0-kserve-5478dcfc45-5gr5q" Nov 03 13:38:37.306783 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:37.306639 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-6f33a0/explicit-model-6f33a0-kserve-5478dcfc45-5gr5q"] Nov 03 13:38:37.332309 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:37.319940 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-6f33a0\"/\"kube-root-ca.crt\"" Nov 03 13:38:37.332309 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:37.320054 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-6f33a0\"/\"default-dockercfg-4g2f2\"" Nov 03 13:38:37.332309 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:37.320097 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-6f33a0\"/\"openshift-service-ca.crt\"" Nov 03 13:38:37.332309 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:37.320197 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-6f33a0\"/\"explicit-model-6f33a0-kserve-self-signed-certs\"" Nov 03 13:38:37.472322 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:37.457252 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-659tl\" (UniqueName: \"kubernetes.io/projected/9138d4ce-ec66-4112-88be-6b6c4d9444b9-kube-api-access-659tl\") pod \"explicit-model-6f33a0-kserve-5478dcfc45-5gr5q\" (UID: \"9138d4ce-ec66-4112-88be-6b6c4d9444b9\") " pod="ai-tenant-e2e-explicit-6f33a0/explicit-model-6f33a0-kserve-5478dcfc45-5gr5q" Nov 03 13:38:37.472745 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:37.472708 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9138d4ce-ec66-4112-88be-6b6c4d9444b9-model-cache\") pod \"explicit-model-6f33a0-kserve-5478dcfc45-5gr5q\" (UID: \"9138d4ce-ec66-4112-88be-6b6c4d9444b9\") " pod="ai-tenant-e2e-explicit-6f33a0/explicit-model-6f33a0-kserve-5478dcfc45-5gr5q" Nov 03 13:38:37.473019 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:37.472996 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9138d4ce-ec66-4112-88be-6b6c4d9444b9-home\") pod \"explicit-model-6f33a0-kserve-5478dcfc45-5gr5q\" (UID: \"9138d4ce-ec66-4112-88be-6b6c4d9444b9\") " pod="ai-tenant-e2e-explicit-6f33a0/explicit-model-6f33a0-kserve-5478dcfc45-5gr5q" Nov 03 13:38:37.473207 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:37.473188 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9138d4ce-ec66-4112-88be-6b6c4d9444b9-dshm\") pod \"explicit-model-6f33a0-kserve-5478dcfc45-5gr5q\" (UID: \"9138d4ce-ec66-4112-88be-6b6c4d9444b9\") " pod="ai-tenant-e2e-explicit-6f33a0/explicit-model-6f33a0-kserve-5478dcfc45-5gr5q" Nov 03 13:38:37.473359 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:37.473343 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9138d4ce-ec66-4112-88be-6b6c4d9444b9-tls-certs\") pod \"explicit-model-6f33a0-kserve-5478dcfc45-5gr5q\" (UID: \"9138d4ce-ec66-4112-88be-6b6c4d9444b9\") " pod="ai-tenant-e2e-explicit-6f33a0/explicit-model-6f33a0-kserve-5478dcfc45-5gr5q" Nov 03 13:38:37.473688 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:37.473669 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9138d4ce-ec66-4112-88be-6b6c4d9444b9-tmp-dir\") pod \"explicit-model-6f33a0-kserve-5478dcfc45-5gr5q\" (UID: \"9138d4ce-ec66-4112-88be-6b6c4d9444b9\") " pod="ai-tenant-e2e-explicit-6f33a0/explicit-model-6f33a0-kserve-5478dcfc45-5gr5q" Nov 03 13:38:37.596208 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:37.596116 5024 patch_prober.go:29] interesting pod/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-cf696 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body= Nov 03 13:38:37.616438 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:37.616388 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-cf696" podUID="2a69c6f3-da97-4a27-9809-951395987c7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" Nov 03 13:38:37.658053 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:37.658004 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-659tl\" (UniqueName: \"kubernetes.io/projected/9138d4ce-ec66-4112-88be-6b6c4d9444b9-kube-api-access-659tl\") pod \"explicit-model-6f33a0-kserve-5478dcfc45-5gr5q\" (UID: \"9138d4ce-ec66-4112-88be-6b6c4d9444b9\") " pod="ai-tenant-e2e-explicit-6f33a0/explicit-model-6f33a0-kserve-5478dcfc45-5gr5q" Nov 03 13:38:37.689699 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:37.689661 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9138d4ce-ec66-4112-88be-6b6c4d9444b9-model-cache\") pod \"explicit-model-6f33a0-kserve-5478dcfc45-5gr5q\" (UID: \"9138d4ce-ec66-4112-88be-6b6c4d9444b9\") " pod="ai-tenant-e2e-explicit-6f33a0/explicit-model-6f33a0-kserve-5478dcfc45-5gr5q" Nov 03 13:38:37.690006 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:37.689986 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9138d4ce-ec66-4112-88be-6b6c4d9444b9-home\") pod \"explicit-model-6f33a0-kserve-5478dcfc45-5gr5q\" (UID: \"9138d4ce-ec66-4112-88be-6b6c4d9444b9\") " pod="ai-tenant-e2e-explicit-6f33a0/explicit-model-6f33a0-kserve-5478dcfc45-5gr5q" Nov 03 13:38:37.690171 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:37.690158 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9138d4ce-ec66-4112-88be-6b6c4d9444b9-dshm\") pod \"explicit-model-6f33a0-kserve-5478dcfc45-5gr5q\" (UID: \"9138d4ce-ec66-4112-88be-6b6c4d9444b9\") " pod="ai-tenant-e2e-explicit-6f33a0/explicit-model-6f33a0-kserve-5478dcfc45-5gr5q" Nov 03 13:38:37.690309 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:37.690278 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9138d4ce-ec66-4112-88be-6b6c4d9444b9-tls-certs\") pod \"explicit-model-6f33a0-kserve-5478dcfc45-5gr5q\" (UID: \"9138d4ce-ec66-4112-88be-6b6c4d9444b9\") " pod="ai-tenant-e2e-explicit-6f33a0/explicit-model-6f33a0-kserve-5478dcfc45-5gr5q" Nov 03 13:38:37.690613 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:37.690592 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9138d4ce-ec66-4112-88be-6b6c4d9444b9-tmp-dir\") pod \"explicit-model-6f33a0-kserve-5478dcfc45-5gr5q\" (UID: \"9138d4ce-ec66-4112-88be-6b6c4d9444b9\") " pod="ai-tenant-e2e-explicit-6f33a0/explicit-model-6f33a0-kserve-5478dcfc45-5gr5q" Nov 03 13:38:37.691486 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:37.691459 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9138d4ce-ec66-4112-88be-6b6c4d9444b9-tmp-dir\") pod \"explicit-model-6f33a0-kserve-5478dcfc45-5gr5q\" (UID: \"9138d4ce-ec66-4112-88be-6b6c4d9444b9\") " pod="ai-tenant-e2e-explicit-6f33a0/explicit-model-6f33a0-kserve-5478dcfc45-5gr5q" Nov 03 13:38:37.691934 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:37.691912 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9138d4ce-ec66-4112-88be-6b6c4d9444b9-model-cache\") pod \"explicit-model-6f33a0-kserve-5478dcfc45-5gr5q\" (UID: \"9138d4ce-ec66-4112-88be-6b6c4d9444b9\") " pod="ai-tenant-e2e-explicit-6f33a0/explicit-model-6f33a0-kserve-5478dcfc45-5gr5q" Nov 03 13:38:37.721336 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:37.695215 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9138d4ce-ec66-4112-88be-6b6c4d9444b9-home\") pod \"explicit-model-6f33a0-kserve-5478dcfc45-5gr5q\" (UID: \"9138d4ce-ec66-4112-88be-6b6c4d9444b9\") " pod="ai-tenant-e2e-explicit-6f33a0/explicit-model-6f33a0-kserve-5478dcfc45-5gr5q" Nov 03 13:38:37.789667 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:37.789626 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9138d4ce-ec66-4112-88be-6b6c4d9444b9-dshm\") pod \"explicit-model-6f33a0-kserve-5478dcfc45-5gr5q\" (UID: \"9138d4ce-ec66-4112-88be-6b6c4d9444b9\") " pod="ai-tenant-e2e-explicit-6f33a0/explicit-model-6f33a0-kserve-5478dcfc45-5gr5q" Nov 03 13:38:37.826311 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:37.822810 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9138d4ce-ec66-4112-88be-6b6c4d9444b9-tls-certs\") pod \"explicit-model-6f33a0-kserve-5478dcfc45-5gr5q\" (UID: \"9138d4ce-ec66-4112-88be-6b6c4d9444b9\") " pod="ai-tenant-e2e-explicit-6f33a0/explicit-model-6f33a0-kserve-5478dcfc45-5gr5q" Nov 03 13:38:37.837031 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:37.836992 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-659tl\" (UniqueName: \"kubernetes.io/projected/9138d4ce-ec66-4112-88be-6b6c4d9444b9-kube-api-access-659tl\") pod \"explicit-model-6f33a0-kserve-5478dcfc45-5gr5q\" (UID: \"9138d4ce-ec66-4112-88be-6b6c4d9444b9\") " pod="ai-tenant-e2e-explicit-6f33a0/explicit-model-6f33a0-kserve-5478dcfc45-5gr5q" Nov 03 13:38:37.965488 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:37.965450 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-6f33a0/explicit-model-6f33a0-kserve-5478dcfc45-5gr5q" Nov 03 13:38:38.615378 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:38.606367 5024 patch_prober.go:29] interesting pod/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-cf696 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body= Nov 03 13:38:38.615378 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:38.606446 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-cf696" podUID="2a69c6f3-da97-4a27-9809-951395987c7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" Nov 03 13:38:38.757212 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:38.757168 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-6f33a0/explicit-model-6f33a0-kserve-5478dcfc45-5gr5q"] Nov 03 13:38:38.768806 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:38:38.768773 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9138d4ce_ec66_4112_88be_6b6c4d9444b9.slice/crio-086f17e64f2d74cf58d7623c7e4ba1c24fa23b2cf3cdcf398ee76e0ee108993c WatchSource:0}: Error finding container 086f17e64f2d74cf58d7623c7e4ba1c24fa23b2cf3cdcf398ee76e0ee108993c: Status 404 returned error can't find the container with id 086f17e64f2d74cf58d7623c7e4ba1c24fa23b2cf3cdcf398ee76e0ee108993c Nov 03 13:38:38.870259 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:38.870093 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-6f33a0/explicit-model-6f33a0-kserve-5478dcfc45-5gr5q" event={"ID":"9138d4ce-ec66-4112-88be-6b6c4d9444b9","Type":"ContainerStarted","Data":"086f17e64f2d74cf58d7623c7e4ba1c24fa23b2cf3cdcf398ee76e0ee108993c"} Nov 03 13:38:39.594365 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:39.592837 5024 patch_prober.go:29] interesting pod/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-cf696 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body= Nov 03 13:38:39.594365 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:39.592923 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-cf696" podUID="2a69c6f3-da97-4a27-9809-951395987c7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" Nov 03 13:38:39.611312 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:38:39.605994 5024 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d7mwz" containerName="extractor" Nov 03 13:38:39.936307 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:39.935626 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-6f33a0/explicit-model-6f33a0-kserve-5478dcfc45-5gr5q" event={"ID":"9138d4ce-ec66-4112-88be-6b6c4d9444b9","Type":"ContainerStarted","Data":"cf2b42e6390f08d29ab36231bd665d6a93c11a38ffea4fb7e714e95e7992fcef"} Nov 03 13:38:39.962018 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:39.961965 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-explicit-6f33a0/explicit-model-6f33a0-kserve-5478dcfc45-5gr5q" podStartSLOduration=2.961950655 podStartE2EDuration="2.961950655s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:38:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:38:39.959981245 +0000 UTC m=+1799.968172522" watchObservedRunningTime="2026-11-03 13:38:39.961950655 +0000 UTC m=+1799.970141930" Nov 03 13:38:39.966248 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:39.966217 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-explicit-6f33a0/explicit-model-6f33a0-kserve-5478dcfc45-5gr5q" Nov 03 13:38:40.596312 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:40.595323 5024 patch_prober.go:29] interesting pod/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-cf696 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body= Nov 03 13:38:40.596312 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:40.595390 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-cf696" podUID="2a69c6f3-da97-4a27-9809-951395987c7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" Nov 03 13:38:41.260458 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:41.260431 5024 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6htht_cd91de08-084e-4667-b13f-2e07b3cff624/console-operator/1.log" Nov 03 13:38:41.279313 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:41.269799 5024 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6htht_cd91de08-084e-4667-b13f-2e07b3cff624/console-operator/1.log" Nov 03 13:38:41.642312 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:41.641246 5024 patch_prober.go:29] interesting pod/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-cf696 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body= Nov 03 13:38:41.642312 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:41.641430 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-cf696" podUID="2a69c6f3-da97-4a27-9809-951395987c7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" Nov 03 13:38:43.125797 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:43.125762 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-cf696" Nov 03 13:38:43.126319 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:43.126135 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-cf696" Nov 03 13:38:43.127167 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:43.127138 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-cf696" Nov 03 13:38:44.429557 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:44.429520 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-6f33a0/explicit-model-6f33a0-kserve-5478dcfc45-5gr5q"] Nov 03 13:38:45.026007 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:45.025966 5024 scope.go:122] "RemoveContainer" containerID="6c3dc16a75a1544367fe596cd5d106649ccf4e7a165aa80d943b388a325928d7" Nov 03 13:38:47.920813 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:47.920757 5024 patch_prober.go:29] interesting pod/e2e-explicit-6f33a0-openshift-default-649df665d5-rrthx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 13:38:47.921244 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:47.920845 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-explicit-6f33a0-openshift-default-649df665d5-rrthx" podUID="a0bde2d4-7fae-41d0-960a-20a0e664d597" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 13:38:48.201154 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:48.201057 5024 generic.go:356] "Generic (PLEG): container finished" podID="a0bde2d4-7fae-41d0-960a-20a0e664d597" containerID="0f0b1f5e44436747e56dfcf0213c92ce11a6c6f6a5e4882be66718fb8f5c3618" exitCode=0 Nov 03 13:38:48.201154 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:48.201137 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-6f33a0-openshift-default-649df665d5-rrthx" event={"ID":"a0bde2d4-7fae-41d0-960a-20a0e664d597","Type":"ContainerDied","Data":"0f0b1f5e44436747e56dfcf0213c92ce11a6c6f6a5e4882be66718fb8f5c3618"} Nov 03 13:38:48.201795 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:48.201772 5024 scope.go:122] "RemoveContainer" containerID="0f0b1f5e44436747e56dfcf0213c92ce11a6c6f6a5e4882be66718fb8f5c3618" Nov 03 13:38:48.203759 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:48.203728 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:38:48.203854 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:48.203812 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:38:48.203854 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:48.203846 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:38:49.213631 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:49.213589 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-6f33a0-openshift-default-649df665d5-rrthx" event={"ID":"a0bde2d4-7fae-41d0-960a-20a0e664d597","Type":"ContainerStarted","Data":"1cbe9f529d95b28dde5fdf6b456a1fd108a228d1446ca60dd7195648719c5531"} Nov 03 13:38:49.367771 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:49.367735 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-6f33a0-openshift-default-649df665d5-rrthx" Nov 03 13:38:49.368027 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:49.368001 5024 patch_prober.go:29] interesting pod/e2e-explicit-6f33a0-openshift-default-649df665d5-rrthx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body= Nov 03 13:38:49.368077 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:49.368054 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-6f33a0-openshift-default-649df665d5-rrthx" podUID="a0bde2d4-7fae-41d0-960a-20a0e664d597" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" Nov 03 13:38:50.368249 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:50.368197 5024 patch_prober.go:29] interesting pod/e2e-explicit-6f33a0-openshift-default-649df665d5-rrthx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body= Nov 03 13:38:50.368869 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:50.368260 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-6f33a0-openshift-default-649df665d5-rrthx" podUID="a0bde2d4-7fae-41d0-960a-20a0e664d597" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" Nov 03 13:38:51.368907 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:51.368864 5024 patch_prober.go:29] interesting pod/e2e-explicit-6f33a0-openshift-default-649df665d5-rrthx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body= Nov 03 13:38:51.369429 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:51.368933 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-6f33a0-openshift-default-649df665d5-rrthx" podUID="a0bde2d4-7fae-41d0-960a-20a0e664d597" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" Nov 03 13:38:52.368534 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:52.368494 5024 patch_prober.go:29] interesting pod/e2e-explicit-6f33a0-openshift-default-649df665d5-rrthx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body= Nov 03 13:38:52.368736 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:52.368550 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-6f33a0-openshift-default-649df665d5-rrthx" podUID="a0bde2d4-7fae-41d0-960a-20a0e664d597" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" Nov 03 13:38:53.368379 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:53.368337 5024 patch_prober.go:29] interesting pod/e2e-explicit-6f33a0-openshift-default-649df665d5-rrthx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body= Nov 03 13:38:53.368871 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:53.368397 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-6f33a0-openshift-default-649df665d5-rrthx" podUID="a0bde2d4-7fae-41d0-960a-20a0e664d597" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" Nov 03 13:38:54.368843 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:54.368803 5024 patch_prober.go:29] interesting pod/e2e-explicit-6f33a0-openshift-default-649df665d5-rrthx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body= Nov 03 13:38:54.369309 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:54.368866 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-6f33a0-openshift-default-649df665d5-rrthx" podUID="a0bde2d4-7fae-41d0-960a-20a0e664d597" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" Nov 03 13:38:55.373216 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:55.373185 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-6f33a0-openshift-default-649df665d5-rrthx" Nov 03 13:38:55.373723 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:55.373698 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-6f33a0-openshift-default-649df665d5-rrthx" Nov 03 13:38:55.375057 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:55.375032 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-6f33a0-openshift-default-649df665d5-rrthx" Nov 03 13:38:59.491204 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:59.491136 5024 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-explicit-6f33a0/explicit-model-6f33a0-kserve-5478dcfc45-5gr5q" podUID="9138d4ce-ec66-4112-88be-6b6c4d9444b9" containerName="main" containerID="cri-o://cf2b42e6390f08d29ab36231bd665d6a93c11a38ffea4fb7e714e95e7992fcef" gracePeriod=45 Nov 03 13:38:59.868265 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:38:59.868232 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-6f33a0/explicit-model-6f33a0-kserve-5478dcfc45-5gr5q" Nov 03 13:39:00.014213 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:39:00.014164 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9138d4ce-ec66-4112-88be-6b6c4d9444b9-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9138d4ce-ec66-4112-88be-6b6c4d9444b9-dshm\") pod \"9138d4ce-ec66-4112-88be-6b6c4d9444b9\" (UID: \"9138d4ce-ec66-4112-88be-6b6c4d9444b9\") " Nov 03 13:39:00.014437 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:39:00.014228 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9138d4ce-ec66-4112-88be-6b6c4d9444b9-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9138d4ce-ec66-4112-88be-6b6c4d9444b9-tmp-dir\") pod \"9138d4ce-ec66-4112-88be-6b6c4d9444b9\" (UID: \"9138d4ce-ec66-4112-88be-6b6c4d9444b9\") " Nov 03 13:39:00.014437 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:39:00.014255 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9138d4ce-ec66-4112-88be-6b6c4d9444b9-tls-certs\" (UniqueName: \"kubernetes.io/secret/9138d4ce-ec66-4112-88be-6b6c4d9444b9-tls-certs\") pod \"9138d4ce-ec66-4112-88be-6b6c4d9444b9\" (UID: \"9138d4ce-ec66-4112-88be-6b6c4d9444b9\") " Nov 03 13:39:00.014437 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:39:00.014317 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9138d4ce-ec66-4112-88be-6b6c4d9444b9-kube-api-access-659tl\" (UniqueName: \"kubernetes.io/projected/9138d4ce-ec66-4112-88be-6b6c4d9444b9-kube-api-access-659tl\") pod \"9138d4ce-ec66-4112-88be-6b6c4d9444b9\" (UID: \"9138d4ce-ec66-4112-88be-6b6c4d9444b9\") " Nov 03 13:39:00.014437 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:39:00.014420 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9138d4ce-ec66-4112-88be-6b6c4d9444b9-home\" (UniqueName: \"kubernetes.io/empty-dir/9138d4ce-ec66-4112-88be-6b6c4d9444b9-home\") pod \"9138d4ce-ec66-4112-88be-6b6c4d9444b9\" (UID: \"9138d4ce-ec66-4112-88be-6b6c4d9444b9\") " Nov 03 13:39:00.014671 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:39:00.014539 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9138d4ce-ec66-4112-88be-6b6c4d9444b9-tmp-dir" pod "9138d4ce-ec66-4112-88be-6b6c4d9444b9" (UID: "9138d4ce-ec66-4112-88be-6b6c4d9444b9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:39:00.014671 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:39:00.014618 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9138d4ce-ec66-4112-88be-6b6c4d9444b9-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9138d4ce-ec66-4112-88be-6b6c4d9444b9-model-cache\") pod \"9138d4ce-ec66-4112-88be-6b6c4d9444b9\" (UID: \"9138d4ce-ec66-4112-88be-6b6c4d9444b9\") " Nov 03 13:39:00.014897 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:39:00.014866 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9138d4ce-ec66-4112-88be-6b6c4d9444b9-home" pod "9138d4ce-ec66-4112-88be-6b6c4d9444b9" (UID: "9138d4ce-ec66-4112-88be-6b6c4d9444b9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:39:00.015157 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:39:00.015119 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9138d4ce-ec66-4112-88be-6b6c4d9444b9-model-cache" pod "9138d4ce-ec66-4112-88be-6b6c4d9444b9" (UID: "9138d4ce-ec66-4112-88be-6b6c4d9444b9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:39:00.015715 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:39:00.015687 5024 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9138d4ce-ec66-4112-88be-6b6c4d9444b9-tmp-dir\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:39:00.015801 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:39:00.015720 5024 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9138d4ce-ec66-4112-88be-6b6c4d9444b9-home\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:39:00.015801 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:39:00.015736 5024 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9138d4ce-ec66-4112-88be-6b6c4d9444b9-model-cache\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:39:00.017235 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:39:00.017164 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9138d4ce-ec66-4112-88be-6b6c4d9444b9-kube-api-access-659tl" pod "9138d4ce-ec66-4112-88be-6b6c4d9444b9" (UID: "9138d4ce-ec66-4112-88be-6b6c4d9444b9"). InnerVolumeSpecName "kube-api-access-659tl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:39:00.017372 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:39:00.017251 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9138d4ce-ec66-4112-88be-6b6c4d9444b9-dshm" pod "9138d4ce-ec66-4112-88be-6b6c4d9444b9" (UID: "9138d4ce-ec66-4112-88be-6b6c4d9444b9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:39:00.017372 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:39:00.017277 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9138d4ce-ec66-4112-88be-6b6c4d9444b9-tls-certs" pod "9138d4ce-ec66-4112-88be-6b6c4d9444b9" (UID: "9138d4ce-ec66-4112-88be-6b6c4d9444b9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 13:39:00.117248 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:39:00.117204 5024 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9138d4ce-ec66-4112-88be-6b6c4d9444b9-dshm\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:39:00.117248 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:39:00.117244 5024 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9138d4ce-ec66-4112-88be-6b6c4d9444b9-tls-certs\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:39:00.117248 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:39:00.117257 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-659tl\" (UniqueName: \"kubernetes.io/projected/9138d4ce-ec66-4112-88be-6b6c4d9444b9-kube-api-access-659tl\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:39:00.334558 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:39:00.334517 5024 generic.go:356] "Generic (PLEG): container finished" podID="9138d4ce-ec66-4112-88be-6b6c4d9444b9" containerID="cf2b42e6390f08d29ab36231bd665d6a93c11a38ffea4fb7e714e95e7992fcef" exitCode=0 Nov 03 13:39:00.334740 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:39:00.334581 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-6f33a0/explicit-model-6f33a0-kserve-5478dcfc45-5gr5q" Nov 03 13:39:00.334740 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:39:00.334607 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-6f33a0/explicit-model-6f33a0-kserve-5478dcfc45-5gr5q" event={"ID":"9138d4ce-ec66-4112-88be-6b6c4d9444b9","Type":"ContainerDied","Data":"cf2b42e6390f08d29ab36231bd665d6a93c11a38ffea4fb7e714e95e7992fcef"} Nov 03 13:39:00.334740 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:39:00.334658 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-6f33a0/explicit-model-6f33a0-kserve-5478dcfc45-5gr5q" event={"ID":"9138d4ce-ec66-4112-88be-6b6c4d9444b9","Type":"ContainerDied","Data":"086f17e64f2d74cf58d7623c7e4ba1c24fa23b2cf3cdcf398ee76e0ee108993c"} Nov 03 13:39:00.334740 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:39:00.334681 5024 scope.go:122] "RemoveContainer" containerID="cf2b42e6390f08d29ab36231bd665d6a93c11a38ffea4fb7e714e95e7992fcef" Nov 03 13:39:00.345512 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:39:00.345481 5024 scope.go:122] "RemoveContainer" containerID="cf2b42e6390f08d29ab36231bd665d6a93c11a38ffea4fb7e714e95e7992fcef" Nov 03 13:39:00.345872 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:39:00.345851 5024 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf2b42e6390f08d29ab36231bd665d6a93c11a38ffea4fb7e714e95e7992fcef\": container with ID starting with cf2b42e6390f08d29ab36231bd665d6a93c11a38ffea4fb7e714e95e7992fcef not found: ID does not exist" containerID="cf2b42e6390f08d29ab36231bd665d6a93c11a38ffea4fb7e714e95e7992fcef" Nov 03 13:39:00.345935 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:39:00.345881 5024 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf2b42e6390f08d29ab36231bd665d6a93c11a38ffea4fb7e714e95e7992fcef"} err="failed to get container status \"cf2b42e6390f08d29ab36231bd665d6a93c11a38ffea4fb7e714e95e7992fcef\": rpc error: code = NotFound desc = could not find container \"cf2b42e6390f08d29ab36231bd665d6a93c11a38ffea4fb7e714e95e7992fcef\": container with ID starting with cf2b42e6390f08d29ab36231bd665d6a93c11a38ffea4fb7e714e95e7992fcef not found: ID does not exist" Nov 03 13:39:00.359763 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:39:00.359720 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-6f33a0/explicit-model-6f33a0-kserve-5478dcfc45-5gr5q"] Nov 03 13:39:00.364774 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:39:00.364741 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-explicit-6f33a0/explicit-model-6f33a0-kserve-5478dcfc45-5gr5q"] Nov 03 13:39:02.111853 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:39:02.111816 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9138d4ce-ec66-4112-88be-6b6c4d9444b9" path="/var/lib/kubelet/pods/9138d4ce-ec66-4112-88be-6b6c4d9444b9/volumes" Nov 03 13:39:05.584809 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:39:05.584755 5024 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-08111b/premium-w1-08111b-kserve-57555d598-dx9g5" podUID="e45c569a-243f-49ab-8d74-dd08cbb6127c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.134:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 13:39:05.735448 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:39:05.735392 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d3ca48/premium-w3-d3ca48-kserve-75ffc959b-zvjbc" podUID="add1e8b9-e7fd-4e4d-8d10-f97d0e2612ba" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 13:39:05.932747 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:39:05.932697 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w2-a724a6/sub-test-model-a724a6-kserve-57d7c88d9f-5nk9m" podUID="3e6246a7-8c0e-4712-8f67-d834948a5673" containerName="main" probeResult="failure" output="Get \"https://10.132.0.153:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 13:39:06.090584 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:39:06.090519 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-08111b/premium-w1-08111b-kserve-57555d598-dx9g5" podUID="e45c569a-243f-49ab-8d74-dd08cbb6127c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.134:8000/ready\": context deadline exceeded" Nov 03 13:39:06.462437 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:39:06.462386 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w0-8f8aa8/rate-test-model-8f8aa8-kserve-7786f67f6f-mbztf" podUID="5fabbac7-721a-4f35-bd22-e86df0cecc62" containerName="main" probeResult="failure" output="Get \"https://10.132.0.160:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 13:39:06.737369 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:39:06.736392 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d3ca48/premium-w3-d3ca48-kserve-75ffc959b-zvjbc" podUID="add1e8b9-e7fd-4e4d-8d10-f97d0e2612ba" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded" Nov 03 13:39:06.737369 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:39:06.737251 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-d3ca48/premium-w3-d3ca48-kserve-75ffc959b-zvjbc" Nov 03 13:39:06.797468 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:39:06.797404 5024 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-d3ca48/premium-w3-d3ca48-kserve-75ffc959b-zvjbc" podUID="add1e8b9-e7fd-4e4d-8d10-f97d0e2612ba" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/health\": context deadline exceeded" Nov 03 13:39:06.923219 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:39:06.923165 5024 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-h2ns8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 13:39:06.923555 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:39:06.923506 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-h2ns8" podUID="155ba865-dcfd-4203-82b9-5399ca58d988" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 13:39:07.096533 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:39:07.096424 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-08111b/premium-w1-08111b-kserve-57555d598-dx9g5" podUID="e45c569a-243f-49ab-8d74-dd08cbb6127c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.134:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 13:39:07.097602 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:39:07.097574 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-08111b/premium-w1-08111b-kserve-57555d598-dx9g5" Nov 03 13:39:07.476260 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:39:07.470894 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w0-8f8aa8/rate-test-model-8f8aa8-kserve-7786f67f6f-mbztf" podUID="5fabbac7-721a-4f35-bd22-e86df0cecc62" containerName="main" probeResult="failure" output="Get \"https://10.132.0.160:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 13:39:07.477943 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:39:07.477891 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-08111b/simulator-w1-08111b-kserve-7f65c5867-btf5b" podUID="e8aba665-e4cc-4e67-8cdd-d09ef7542e15" containerName="main" probeResult="failure" output="Get \"https://10.132.0.133:8000/ready\": context deadline exceeded" Nov 03 13:39:07.479739 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:39:07.479613 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w0-8f8aa8/rate-test-model-8f8aa8-kserve-7786f67f6f-mbztf" Nov 03 13:39:07.594468 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:39:07.594433 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w0-8f8aa8/rate-test-model-8f8aa8-kserve-7786f67f6f-mbztf" Nov 03 13:39:07.754145 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:39:07.738392 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d3ca48/premium-w3-d3ca48-kserve-75ffc959b-zvjbc" podUID="add1e8b9-e7fd-4e4d-8d10-f97d0e2612ba" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded" Nov 03 13:39:08.092150 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:39:08.090420 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-08111b/premium-w1-08111b-kserve-57555d598-dx9g5" Nov 03 13:39:08.277544 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:39:08.275904 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-d3ca48/premium-w3-d3ca48-kserve-75ffc959b-zvjbc" Nov 03 13:39:09.526921 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:39:09.526874 5024 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d7mwz" containerName="extractor" Nov 03 13:39:20.272786 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:39:20.272747 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-c381a7-558f76676f-98tbx"] Nov 03 13:39:25.273801 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:39:25.273732 5024 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w4-c381a7-558f76676f-98tbx" podUID="17976293-ef64-4977-a6a2-d872ac7b9b4a" containerName="payload-processing" containerID="cri-o://2e449a1cdee3b0a465913cc39dfa89ba62a396f845fe3224528fcc0fe9c9a9b7" gracePeriod=25 Nov 03 13:39:27.532829 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:39:27.532788 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-6f33a0-d8c4dd6f6-zxfcz"] Nov 03 13:39:27.533356 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:39:27.533074 5024 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-6f33a0-d8c4dd6f6-zxfcz" podUID="9d4a421b-18cf-4c67-8b5e-31ad51dd01aa" containerName="maas-api" containerID="cri-o://4059ef754e5e73dc76630e3d492f3c99dae2ee0d646dbc6ac762dc91ae91a7da" gracePeriod=30 Nov 03 13:39:28.058358 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:39:28.058318 5024 generic.go:356] "Generic (PLEG): container finished" podID="9d4a421b-18cf-4c67-8b5e-31ad51dd01aa" containerID="4059ef754e5e73dc76630e3d492f3c99dae2ee0d646dbc6ac762dc91ae91a7da" exitCode=0 Nov 03 13:39:28.058621 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:39:28.058599 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-6f33a0-d8c4dd6f6-zxfcz" event={"ID":"9d4a421b-18cf-4c67-8b5e-31ad51dd01aa","Type":"ContainerDied","Data":"4059ef754e5e73dc76630e3d492f3c99dae2ee0d646dbc6ac762dc91ae91a7da"} Nov 03 13:39:28.193455 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:39:28.193425 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-6f33a0-d8c4dd6f6-zxfcz" Nov 03 13:39:28.291998 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:39:28.291956 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9d4a421b-18cf-4c67-8b5e-31ad51dd01aa-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d4a421b-18cf-4c67-8b5e-31ad51dd01aa-metrics-certs\") pod \"9d4a421b-18cf-4c67-8b5e-31ad51dd01aa\" (UID: \"9d4a421b-18cf-4c67-8b5e-31ad51dd01aa\") " Nov 03 13:39:28.292188 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:39:28.292014 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9d4a421b-18cf-4c67-8b5e-31ad51dd01aa-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9d4a421b-18cf-4c67-8b5e-31ad51dd01aa-maas-api-tls\") pod \"9d4a421b-18cf-4c67-8b5e-31ad51dd01aa\" (UID: \"9d4a421b-18cf-4c67-8b5e-31ad51dd01aa\") " Nov 03 13:39:28.292273 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:39:28.292219 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d4a421b-18cf-4c67-8b5e-31ad51dd01aa-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d4a421b-18cf-4c67-8b5e-31ad51dd01aa-openshift-service-ca\") pod \"9d4a421b-18cf-4c67-8b5e-31ad51dd01aa\" (UID: \"9d4a421b-18cf-4c67-8b5e-31ad51dd01aa\") " Nov 03 13:39:28.292422 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:39:28.292401 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d4a421b-18cf-4c67-8b5e-31ad51dd01aa-kube-api-access-l66vz\" (UniqueName: \"kubernetes.io/projected/9d4a421b-18cf-4c67-8b5e-31ad51dd01aa-kube-api-access-l66vz\") pod \"9d4a421b-18cf-4c67-8b5e-31ad51dd01aa\" (UID: \"9d4a421b-18cf-4c67-8b5e-31ad51dd01aa\") " Nov 03 13:39:28.292773 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:39:28.292731 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d4a421b-18cf-4c67-8b5e-31ad51dd01aa-openshift-service-ca" pod "9d4a421b-18cf-4c67-8b5e-31ad51dd01aa" (UID: "9d4a421b-18cf-4c67-8b5e-31ad51dd01aa"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:39:28.293589 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:39:28.293542 5024 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d4a421b-18cf-4c67-8b5e-31ad51dd01aa-openshift-service-ca\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:39:28.294690 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:39:28.294661 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d4a421b-18cf-4c67-8b5e-31ad51dd01aa-kube-api-access-l66vz" pod "9d4a421b-18cf-4c67-8b5e-31ad51dd01aa" (UID: "9d4a421b-18cf-4c67-8b5e-31ad51dd01aa"). InnerVolumeSpecName "kube-api-access-l66vz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:39:28.294794 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:39:28.294763 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d4a421b-18cf-4c67-8b5e-31ad51dd01aa-maas-api-tls" pod "9d4a421b-18cf-4c67-8b5e-31ad51dd01aa" (UID: "9d4a421b-18cf-4c67-8b5e-31ad51dd01aa"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 13:39:28.294871 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:39:28.294851 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d4a421b-18cf-4c67-8b5e-31ad51dd01aa-metrics-certs" pod "9d4a421b-18cf-4c67-8b5e-31ad51dd01aa" (UID: "9d4a421b-18cf-4c67-8b5e-31ad51dd01aa"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 13:39:28.395714 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:39:28.395665 5024 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d4a421b-18cf-4c67-8b5e-31ad51dd01aa-metrics-certs\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:39:28.395714 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:39:28.395709 5024 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9d4a421b-18cf-4c67-8b5e-31ad51dd01aa-maas-api-tls\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:39:28.395953 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:39:28.395725 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l66vz\" (UniqueName: \"kubernetes.io/projected/9d4a421b-18cf-4c67-8b5e-31ad51dd01aa-kube-api-access-l66vz\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:39:29.066976 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:39:29.066941 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-6f33a0-d8c4dd6f6-zxfcz" Nov 03 13:39:29.067472 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:39:29.066977 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-6f33a0-d8c4dd6f6-zxfcz" event={"ID":"9d4a421b-18cf-4c67-8b5e-31ad51dd01aa","Type":"ContainerDied","Data":"d8bc91092733d0b5678994f474b206aa4c04c3f4ac32e227484a3f89378cba9a"} Nov 03 13:39:29.067472 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:39:29.067047 5024 scope.go:122] "RemoveContainer" containerID="4059ef754e5e73dc76630e3d492f3c99dae2ee0d646dbc6ac762dc91ae91a7da" Nov 03 13:39:29.095332 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:39:29.095249 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-6f33a0-d8c4dd6f6-zxfcz"] Nov 03 13:39:29.098981 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:39:29.098950 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-6f33a0-d8c4dd6f6-zxfcz"] Nov 03 13:39:30.115243 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:39:30.115204 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d4a421b-18cf-4c67-8b5e-31ad51dd01aa" path="/var/lib/kubelet/pods/9d4a421b-18cf-4c67-8b5e-31ad51dd01aa/volumes" Nov 03 13:39:33.795551 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:39:33.795502 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-c381a7-558f76676f-98tbx" podUID="17976293-ef64-4977-a6a2-d872ac7b9b4a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 13:39:37.555686 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:39:37.555641 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-6f33a0-openshift-default-649df665d5-rrthx"] Nov 03 13:39:37.556249 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:39:37.555929 5024 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-explicit-6f33a0-openshift-default-649df665d5-rrthx" podUID="a0bde2d4-7fae-41d0-960a-20a0e664d597" containerName="istio-proxy" containerID="cri-o://1cbe9f529d95b28dde5fdf6b456a1fd108a228d1446ca60dd7195648719c5531" gracePeriod=30 Nov 03 13:39:39.511226 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:39:39.511183 5024 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d7mwz" containerName="extractor" Nov 03 13:39:40.376521 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:39:40.376474 5024 patch_prober.go:29] interesting pod/e2e-explicit-6f33a0-openshift-default-649df665d5-rrthx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 13:39:40.376695 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:39:40.376562 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-explicit-6f33a0-openshift-default-649df665d5-rrthx" podUID="a0bde2d4-7fae-41d0-960a-20a0e664d597" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 13:39:42.854463 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:39:42.854427 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-6f33a0-openshift-default-649df665d5-rrthx" Nov 03 13:39:42.926377 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:39:42.926333 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0bde2d4-7fae-41d0-960a-20a0e664d597-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0bde2d4-7fae-41d0-960a-20a0e664d597-istio-data\") pod \"a0bde2d4-7fae-41d0-960a-20a0e664d597\" (UID: \"a0bde2d4-7fae-41d0-960a-20a0e664d597\") " Nov 03 13:39:42.926588 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:39:42.926389 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0bde2d4-7fae-41d0-960a-20a0e664d597-istio-token\" (UniqueName: \"kubernetes.io/projected/a0bde2d4-7fae-41d0-960a-20a0e664d597-istio-token\") pod \"a0bde2d4-7fae-41d0-960a-20a0e664d597\" (UID: \"a0bde2d4-7fae-41d0-960a-20a0e664d597\") " Nov 03 13:39:42.926588 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:39:42.926417 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0bde2d4-7fae-41d0-960a-20a0e664d597-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0bde2d4-7fae-41d0-960a-20a0e664d597-credential-socket\") pod \"a0bde2d4-7fae-41d0-960a-20a0e664d597\" (UID: \"a0bde2d4-7fae-41d0-960a-20a0e664d597\") " Nov 03 13:39:42.926588 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:39:42.926494 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a0bde2d4-7fae-41d0-960a-20a0e664d597-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0bde2d4-7fae-41d0-960a-20a0e664d597-istiod-ca-cert\") pod \"a0bde2d4-7fae-41d0-960a-20a0e664d597\" (UID: \"a0bde2d4-7fae-41d0-960a-20a0e664d597\") " Nov 03 13:39:42.926764 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:39:42.926638 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0bde2d4-7fae-41d0-960a-20a0e664d597-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0bde2d4-7fae-41d0-960a-20a0e664d597-workload-socket\") pod \"a0bde2d4-7fae-41d0-960a-20a0e664d597\" (UID: \"a0bde2d4-7fae-41d0-960a-20a0e664d597\") " Nov 03 13:39:42.926764 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:39:42.926668 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0bde2d4-7fae-41d0-960a-20a0e664d597-istio-data" pod "a0bde2d4-7fae-41d0-960a-20a0e664d597" (UID: "a0bde2d4-7fae-41d0-960a-20a0e664d597"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:39:42.926764 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:39:42.926740 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0bde2d4-7fae-41d0-960a-20a0e664d597-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0bde2d4-7fae-41d0-960a-20a0e664d597-workload-certs\") pod \"a0bde2d4-7fae-41d0-960a-20a0e664d597\" (UID: \"a0bde2d4-7fae-41d0-960a-20a0e664d597\") " Nov 03 13:39:42.926909 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:39:42.926861 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a0bde2d4-7fae-41d0-960a-20a0e664d597-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0bde2d4-7fae-41d0-960a-20a0e664d597-istio-podinfo\") pod \"a0bde2d4-7fae-41d0-960a-20a0e664d597\" (UID: \"a0bde2d4-7fae-41d0-960a-20a0e664d597\") " Nov 03 13:39:42.926909 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:39:42.926869 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0bde2d4-7fae-41d0-960a-20a0e664d597-credential-socket" pod "a0bde2d4-7fae-41d0-960a-20a0e664d597" (UID: "a0bde2d4-7fae-41d0-960a-20a0e664d597"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:39:42.927015 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:39:42.926971 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0bde2d4-7fae-41d0-960a-20a0e664d597-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0bde2d4-7fae-41d0-960a-20a0e664d597-istio-envoy\") pod \"a0bde2d4-7fae-41d0-960a-20a0e664d597\" (UID: \"a0bde2d4-7fae-41d0-960a-20a0e664d597\") " Nov 03 13:39:42.927069 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:39:42.927010 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0bde2d4-7fae-41d0-960a-20a0e664d597-kube-api-access-n5c7c\" (UniqueName: \"kubernetes.io/projected/a0bde2d4-7fae-41d0-960a-20a0e664d597-kube-api-access-n5c7c\") pod \"a0bde2d4-7fae-41d0-960a-20a0e664d597\" (UID: \"a0bde2d4-7fae-41d0-960a-20a0e664d597\") " Nov 03 13:39:42.927185 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:39:42.927164 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0bde2d4-7fae-41d0-960a-20a0e664d597-istiod-ca-cert" pod "a0bde2d4-7fae-41d0-960a-20a0e664d597" (UID: "a0bde2d4-7fae-41d0-960a-20a0e664d597"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:39:42.927410 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:39:42.927357 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0bde2d4-7fae-41d0-960a-20a0e664d597-workload-certs" pod "a0bde2d4-7fae-41d0-960a-20a0e664d597" (UID: "a0bde2d4-7fae-41d0-960a-20a0e664d597"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:39:42.928555 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:39:42.928464 5024 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0bde2d4-7fae-41d0-960a-20a0e664d597-istio-data\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:39:42.928555 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:39:42.928491 5024 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0bde2d4-7fae-41d0-960a-20a0e664d597-credential-socket\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:39:42.928555 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:39:42.928506 5024 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0bde2d4-7fae-41d0-960a-20a0e664d597-istiod-ca-cert\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:39:42.928555 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:39:42.928521 5024 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0bde2d4-7fae-41d0-960a-20a0e664d597-workload-certs\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:39:42.929817 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:39:42.929786 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0bde2d4-7fae-41d0-960a-20a0e664d597-istio-envoy" pod "a0bde2d4-7fae-41d0-960a-20a0e664d597" (UID: "a0bde2d4-7fae-41d0-960a-20a0e664d597"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:39:42.930760 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:39:42.930734 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0bde2d4-7fae-41d0-960a-20a0e664d597-workload-socket" pod "a0bde2d4-7fae-41d0-960a-20a0e664d597" (UID: "a0bde2d4-7fae-41d0-960a-20a0e664d597"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:39:42.932050 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:39:42.931264 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a0bde2d4-7fae-41d0-960a-20a0e664d597-istio-podinfo" pod "a0bde2d4-7fae-41d0-960a-20a0e664d597" (UID: "a0bde2d4-7fae-41d0-960a-20a0e664d597"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 13:39:42.932050 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:39:42.932015 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0bde2d4-7fae-41d0-960a-20a0e664d597-istio-token" pod "a0bde2d4-7fae-41d0-960a-20a0e664d597" (UID: "a0bde2d4-7fae-41d0-960a-20a0e664d597"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:39:42.932645 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:39:42.932617 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0bde2d4-7fae-41d0-960a-20a0e664d597-kube-api-access-n5c7c" pod "a0bde2d4-7fae-41d0-960a-20a0e664d597" (UID: "a0bde2d4-7fae-41d0-960a-20a0e664d597"). InnerVolumeSpecName "kube-api-access-n5c7c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:39:43.030233 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:39:43.030191 5024 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0bde2d4-7fae-41d0-960a-20a0e664d597-istio-envoy\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:39:43.030233 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:39:43.030229 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n5c7c\" (UniqueName: \"kubernetes.io/projected/a0bde2d4-7fae-41d0-960a-20a0e664d597-kube-api-access-n5c7c\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:39:43.030501 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:39:43.030242 5024 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0bde2d4-7fae-41d0-960a-20a0e664d597-istio-token\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:39:43.030501 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:39:43.030261 5024 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0bde2d4-7fae-41d0-960a-20a0e664d597-workload-socket\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:39:43.030501 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:39:43.030270 5024 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0bde2d4-7fae-41d0-960a-20a0e664d597-istio-podinfo\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:39:43.309087 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:39:43.309002 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-noten-cd80b3/noten-model-cd80b3-kserve-6789465645-pr6c4"] Nov 03 13:39:43.311551 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:39:43.311522 5024 cpu_manager.go:414] "containerMap: removing container" podUID="9138d4ce-ec66-4112-88be-6b6c4d9444b9" containerName="main" Nov 03 13:39:43.311756 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:39:43.311736 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9138d4ce-ec66-4112-88be-6b6c4d9444b9" containerName="main" Nov 03 13:39:43.311880 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:39:43.311865 5024 cpu_manager.go:414] "containerMap: removing container" podUID="a0bde2d4-7fae-41d0-960a-20a0e664d597" containerName="istio-proxy" Nov 03 13:39:43.311969 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:39:43.311957 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0bde2d4-7fae-41d0-960a-20a0e664d597" containerName="istio-proxy" Nov 03 13:39:43.312227 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:39:43.312201 5024 cpu_manager.go:414] "containerMap: removing container" podUID="a0bde2d4-7fae-41d0-960a-20a0e664d597" containerName="istio-proxy" Nov 03 13:39:43.312227 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:39:43.312224 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0bde2d4-7fae-41d0-960a-20a0e664d597" containerName="istio-proxy" Nov 03 13:39:43.314199 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:39:43.312236 5024 cpu_manager.go:414] "containerMap: removing container" podUID="9d4a421b-18cf-4c67-8b5e-31ad51dd01aa" containerName="maas-api" Nov 03 13:39:43.314199 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:39:43.312244 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d4a421b-18cf-4c67-8b5e-31ad51dd01aa" containerName="maas-api" Nov 03 13:39:43.314199 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:39:43.312482 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0bde2d4-7fae-41d0-960a-20a0e664d597" containerName="istio-proxy" Nov 03 13:39:43.314199 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:39:43.312498 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d4a421b-18cf-4c67-8b5e-31ad51dd01aa" containerName="maas-api" Nov 03 13:39:43.314199 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:39:43.312512 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="9138d4ce-ec66-4112-88be-6b6c4d9444b9" containerName="main" Nov 03 13:39:43.319774 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:39:43.319745 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-cd80b3/noten-model-cd80b3-kserve-6789465645-pr6c4" Nov 03 13:39:43.322198 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:39:43.322167 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-cd80b3/noten-model-cd80b3-kserve-6789465645-pr6c4"] Nov 03 13:39:43.332398 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:39:43.332358 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-cd80b3\"/\"kube-root-ca.crt\"" Nov 03 13:39:43.332619 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:39:43.332479 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-cd80b3\"/\"openshift-service-ca.crt\"" Nov 03 13:39:43.334732 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:39:43.334705 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-cd80b3\"/\"default-dockercfg-wxdh8\"" Nov 03 13:39:43.334998 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:39:43.334982 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-cd80b3\"/\"noten-model-cd80b3-kserve-self-signed-certs\"" Nov 03 13:39:43.344413 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:39:43.344382 5024 generic.go:356] "Generic (PLEG): container finished" podID="a0bde2d4-7fae-41d0-960a-20a0e664d597" containerID="1cbe9f529d95b28dde5fdf6b456a1fd108a228d1446ca60dd7195648719c5531" exitCode=0 Nov 03 13:39:43.344593 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:39:43.344467 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-6f33a0-openshift-default-649df665d5-rrthx" event={"ID":"a0bde2d4-7fae-41d0-960a-20a0e664d597","Type":"ContainerDied","Data":"1cbe9f529d95b28dde5fdf6b456a1fd108a228d1446ca60dd7195648719c5531"} Nov 03 13:39:43.344593 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:39:43.344464 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-6f33a0-openshift-default-649df665d5-rrthx" Nov 03 13:39:43.344593 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:39:43.344556 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-6f33a0-openshift-default-649df665d5-rrthx" event={"ID":"a0bde2d4-7fae-41d0-960a-20a0e664d597","Type":"ContainerDied","Data":"b18bff99869fa2e2275ed08b90973ab221190e1acc6ebebc960c34af52aebf9c"} Nov 03 13:39:43.344759 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:39:43.344598 5024 scope.go:122] "RemoveContainer" containerID="1cbe9f529d95b28dde5fdf6b456a1fd108a228d1446ca60dd7195648719c5531" Nov 03 13:39:43.355694 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:39:43.355654 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c27e059d-8901-486f-9e7f-60fc2c2e7416-dshm\") pod \"noten-model-cd80b3-kserve-6789465645-pr6c4\" (UID: \"c27e059d-8901-486f-9e7f-60fc2c2e7416\") " pod="ai-tenant-e2e-noten-cd80b3/noten-model-cd80b3-kserve-6789465645-pr6c4" Nov 03 13:39:43.355839 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:39:43.355822 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c27e059d-8901-486f-9e7f-60fc2c2e7416-tmp-dir\") pod \"noten-model-cd80b3-kserve-6789465645-pr6c4\" (UID: \"c27e059d-8901-486f-9e7f-60fc2c2e7416\") " pod="ai-tenant-e2e-noten-cd80b3/noten-model-cd80b3-kserve-6789465645-pr6c4" Nov 03 13:39:43.355915 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:39:43.355895 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c27e059d-8901-486f-9e7f-60fc2c2e7416-tls-certs\") pod \"noten-model-cd80b3-kserve-6789465645-pr6c4\" (UID: \"c27e059d-8901-486f-9e7f-60fc2c2e7416\") " pod="ai-tenant-e2e-noten-cd80b3/noten-model-cd80b3-kserve-6789465645-pr6c4" Nov 03 13:39:43.355975 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:39:43.355947 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c27e059d-8901-486f-9e7f-60fc2c2e7416-home\") pod \"noten-model-cd80b3-kserve-6789465645-pr6c4\" (UID: \"c27e059d-8901-486f-9e7f-60fc2c2e7416\") " pod="ai-tenant-e2e-noten-cd80b3/noten-model-cd80b3-kserve-6789465645-pr6c4" Nov 03 13:39:43.356040 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:39:43.356018 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fk7jk\" (UniqueName: \"kubernetes.io/projected/c27e059d-8901-486f-9e7f-60fc2c2e7416-kube-api-access-fk7jk\") pod \"noten-model-cd80b3-kserve-6789465645-pr6c4\" (UID: \"c27e059d-8901-486f-9e7f-60fc2c2e7416\") " pod="ai-tenant-e2e-noten-cd80b3/noten-model-cd80b3-kserve-6789465645-pr6c4" Nov 03 13:39:43.356116 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:39:43.356099 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c27e059d-8901-486f-9e7f-60fc2c2e7416-model-cache\") pod \"noten-model-cd80b3-kserve-6789465645-pr6c4\" (UID: \"c27e059d-8901-486f-9e7f-60fc2c2e7416\") " pod="ai-tenant-e2e-noten-cd80b3/noten-model-cd80b3-kserve-6789465645-pr6c4" Nov 03 13:39:43.367037 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:39:43.367011 5024 scope.go:122] "RemoveContainer" containerID="0f0b1f5e44436747e56dfcf0213c92ce11a6c6f6a5e4882be66718fb8f5c3618" Nov 03 13:39:43.381477 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:39:43.381449 5024 scope.go:122] "RemoveContainer" containerID="1cbe9f529d95b28dde5fdf6b456a1fd108a228d1446ca60dd7195648719c5531" Nov 03 13:39:43.381899 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:39:43.381871 5024 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1cbe9f529d95b28dde5fdf6b456a1fd108a228d1446ca60dd7195648719c5531\": container with ID starting with 1cbe9f529d95b28dde5fdf6b456a1fd108a228d1446ca60dd7195648719c5531 not found: ID does not exist" containerID="1cbe9f529d95b28dde5fdf6b456a1fd108a228d1446ca60dd7195648719c5531" Nov 03 13:39:43.381978 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:39:43.381913 5024 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1cbe9f529d95b28dde5fdf6b456a1fd108a228d1446ca60dd7195648719c5531"} err="failed to get container status \"1cbe9f529d95b28dde5fdf6b456a1fd108a228d1446ca60dd7195648719c5531\": rpc error: code = NotFound desc = could not find container \"1cbe9f529d95b28dde5fdf6b456a1fd108a228d1446ca60dd7195648719c5531\": container with ID starting with 1cbe9f529d95b28dde5fdf6b456a1fd108a228d1446ca60dd7195648719c5531 not found: ID does not exist" Nov 03 13:39:43.381978 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:39:43.381950 5024 scope.go:122] "RemoveContainer" containerID="0f0b1f5e44436747e56dfcf0213c92ce11a6c6f6a5e4882be66718fb8f5c3618" Nov 03 13:39:43.382362 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:39:43.382271 5024 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f0b1f5e44436747e56dfcf0213c92ce11a6c6f6a5e4882be66718fb8f5c3618\": container with ID starting with 0f0b1f5e44436747e56dfcf0213c92ce11a6c6f6a5e4882be66718fb8f5c3618 not found: ID does not exist" containerID="0f0b1f5e44436747e56dfcf0213c92ce11a6c6f6a5e4882be66718fb8f5c3618" Nov 03 13:39:43.382502 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:39:43.382359 5024 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f0b1f5e44436747e56dfcf0213c92ce11a6c6f6a5e4882be66718fb8f5c3618"} err="failed to get container status \"0f0b1f5e44436747e56dfcf0213c92ce11a6c6f6a5e4882be66718fb8f5c3618\": rpc error: code = NotFound desc = could not find container \"0f0b1f5e44436747e56dfcf0213c92ce11a6c6f6a5e4882be66718fb8f5c3618\": container with ID starting with 0f0b1f5e44436747e56dfcf0213c92ce11a6c6f6a5e4882be66718fb8f5c3618 not found: ID does not exist" Nov 03 13:39:43.403049 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:39:43.403013 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-6f33a0-openshift-default-649df665d5-rrthx"] Nov 03 13:39:43.420893 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:39:43.420858 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-explicit-6f33a0-openshift-default-649df665d5-rrthx"] Nov 03 13:39:43.456890 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:39:43.456853 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c27e059d-8901-486f-9e7f-60fc2c2e7416-model-cache\") pod \"noten-model-cd80b3-kserve-6789465645-pr6c4\" (UID: \"c27e059d-8901-486f-9e7f-60fc2c2e7416\") " pod="ai-tenant-e2e-noten-cd80b3/noten-model-cd80b3-kserve-6789465645-pr6c4" Nov 03 13:39:43.457105 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:39:43.456976 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c27e059d-8901-486f-9e7f-60fc2c2e7416-dshm\") pod \"noten-model-cd80b3-kserve-6789465645-pr6c4\" (UID: \"c27e059d-8901-486f-9e7f-60fc2c2e7416\") " pod="ai-tenant-e2e-noten-cd80b3/noten-model-cd80b3-kserve-6789465645-pr6c4" Nov 03 13:39:43.457177 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:39:43.457142 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c27e059d-8901-486f-9e7f-60fc2c2e7416-tmp-dir\") pod \"noten-model-cd80b3-kserve-6789465645-pr6c4\" (UID: \"c27e059d-8901-486f-9e7f-60fc2c2e7416\") " pod="ai-tenant-e2e-noten-cd80b3/noten-model-cd80b3-kserve-6789465645-pr6c4" Nov 03 13:39:43.457237 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:39:43.457220 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c27e059d-8901-486f-9e7f-60fc2c2e7416-tls-certs\") pod \"noten-model-cd80b3-kserve-6789465645-pr6c4\" (UID: \"c27e059d-8901-486f-9e7f-60fc2c2e7416\") " pod="ai-tenant-e2e-noten-cd80b3/noten-model-cd80b3-kserve-6789465645-pr6c4" Nov 03 13:39:43.457349 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:39:43.457254 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c27e059d-8901-486f-9e7f-60fc2c2e7416-model-cache\") pod \"noten-model-cd80b3-kserve-6789465645-pr6c4\" (UID: \"c27e059d-8901-486f-9e7f-60fc2c2e7416\") " pod="ai-tenant-e2e-noten-cd80b3/noten-model-cd80b3-kserve-6789465645-pr6c4" Nov 03 13:39:43.457349 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:39:43.457277 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c27e059d-8901-486f-9e7f-60fc2c2e7416-home\") pod \"noten-model-cd80b3-kserve-6789465645-pr6c4\" (UID: \"c27e059d-8901-486f-9e7f-60fc2c2e7416\") " pod="ai-tenant-e2e-noten-cd80b3/noten-model-cd80b3-kserve-6789465645-pr6c4" Nov 03 13:39:43.457474 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:39:43.457411 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fk7jk\" (UniqueName: \"kubernetes.io/projected/c27e059d-8901-486f-9e7f-60fc2c2e7416-kube-api-access-fk7jk\") pod \"noten-model-cd80b3-kserve-6789465645-pr6c4\" (UID: \"c27e059d-8901-486f-9e7f-60fc2c2e7416\") " pod="ai-tenant-e2e-noten-cd80b3/noten-model-cd80b3-kserve-6789465645-pr6c4" Nov 03 13:39:43.457552 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:39:43.457529 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c27e059d-8901-486f-9e7f-60fc2c2e7416-tmp-dir\") pod \"noten-model-cd80b3-kserve-6789465645-pr6c4\" (UID: \"c27e059d-8901-486f-9e7f-60fc2c2e7416\") " pod="ai-tenant-e2e-noten-cd80b3/noten-model-cd80b3-kserve-6789465645-pr6c4" Nov 03 13:39:43.457620 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:39:43.457599 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c27e059d-8901-486f-9e7f-60fc2c2e7416-home\") pod \"noten-model-cd80b3-kserve-6789465645-pr6c4\" (UID: \"c27e059d-8901-486f-9e7f-60fc2c2e7416\") " pod="ai-tenant-e2e-noten-cd80b3/noten-model-cd80b3-kserve-6789465645-pr6c4" Nov 03 13:39:43.459259 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:39:43.459236 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c27e059d-8901-486f-9e7f-60fc2c2e7416-dshm\") pod \"noten-model-cd80b3-kserve-6789465645-pr6c4\" (UID: \"c27e059d-8901-486f-9e7f-60fc2c2e7416\") " pod="ai-tenant-e2e-noten-cd80b3/noten-model-cd80b3-kserve-6789465645-pr6c4" Nov 03 13:39:43.459627 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:39:43.459609 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c27e059d-8901-486f-9e7f-60fc2c2e7416-tls-certs\") pod \"noten-model-cd80b3-kserve-6789465645-pr6c4\" (UID: \"c27e059d-8901-486f-9e7f-60fc2c2e7416\") " pod="ai-tenant-e2e-noten-cd80b3/noten-model-cd80b3-kserve-6789465645-pr6c4" Nov 03 13:39:43.468277 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:39:43.468251 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fk7jk\" (UniqueName: \"kubernetes.io/projected/c27e059d-8901-486f-9e7f-60fc2c2e7416-kube-api-access-fk7jk\") pod \"noten-model-cd80b3-kserve-6789465645-pr6c4\" (UID: \"c27e059d-8901-486f-9e7f-60fc2c2e7416\") " pod="ai-tenant-e2e-noten-cd80b3/noten-model-cd80b3-kserve-6789465645-pr6c4" Nov 03 13:39:43.651311 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:39:43.646903 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-cd80b3/noten-model-cd80b3-kserve-6789465645-pr6c4" Nov 03 13:39:43.794179 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:39:43.794135 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-c381a7-558f76676f-98tbx" podUID="17976293-ef64-4977-a6a2-d872ac7b9b4a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 13:39:44.009998 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:39:44.009946 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-cd80b3/noten-model-cd80b3-kserve-6789465645-pr6c4"] Nov 03 13:39:44.010489 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:39:44.010170 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc27e059d_8901_486f_9e7f_60fc2c2e7416.slice/crio-3dc59b1f30a661cf8aeee41c152243cb5f0af3369fbff2a34bc1a71cdc9424e4 WatchSource:0}: Error finding container 3dc59b1f30a661cf8aeee41c152243cb5f0af3369fbff2a34bc1a71cdc9424e4: Status 404 returned error can't find the container with id 3dc59b1f30a661cf8aeee41c152243cb5f0af3369fbff2a34bc1a71cdc9424e4 Nov 03 13:39:44.112064 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:39:44.112020 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a0bde2d4-7fae-41d0-960a-20a0e664d597" path="/var/lib/kubelet/pods/a0bde2d4-7fae-41d0-960a-20a0e664d597/volumes" Nov 03 13:39:44.355661 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:39:44.355622 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-cd80b3/noten-model-cd80b3-kserve-6789465645-pr6c4" event={"ID":"c27e059d-8901-486f-9e7f-60fc2c2e7416","Type":"ContainerStarted","Data":"5ba82370b3e521710eaea9b2482ecb5db28888fa4b273b37f9558128d55e3d95"} Nov 03 13:39:44.355832 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:39:44.355668 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-cd80b3/noten-model-cd80b3-kserve-6789465645-pr6c4" event={"ID":"c27e059d-8901-486f-9e7f-60fc2c2e7416","Type":"ContainerStarted","Data":"3dc59b1f30a661cf8aeee41c152243cb5f0af3369fbff2a34bc1a71cdc9424e4"} Nov 03 13:39:44.382012 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:39:44.381953 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-noten-cd80b3/noten-model-cd80b3-kserve-6789465645-pr6c4" podStartSLOduration=1.3819383140000001 podStartE2EDuration="1.381938314s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:39:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:39:44.377534877 +0000 UTC m=+1864.385726161" watchObservedRunningTime="2026-11-03 13:39:44.381938314 +0000 UTC m=+1864.390129631" Nov 03 13:39:44.648412 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:39:44.648323 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-noten-cd80b3/noten-model-cd80b3-kserve-6789465645-pr6c4" Nov 03 13:39:45.676149 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:39:45.676112 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-c381a7-558f76676f-98tbx" Nov 03 13:39:45.700494 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:39:45.700452 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/17976293-ef64-4977-a6a2-d872ac7b9b4a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17976293-ef64-4977-a6a2-d872ac7b9b4a-plugins-config-volume\") pod \"17976293-ef64-4977-a6a2-d872ac7b9b4a\" (UID: \"17976293-ef64-4977-a6a2-d872ac7b9b4a\") " Nov 03 13:39:45.700742 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:39:45.700727 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17976293-ef64-4977-a6a2-d872ac7b9b4a-kube-api-access-94hzp\" (UniqueName: \"kubernetes.io/projected/17976293-ef64-4977-a6a2-d872ac7b9b4a-kube-api-access-94hzp\") pod \"17976293-ef64-4977-a6a2-d872ac7b9b4a\" (UID: \"17976293-ef64-4977-a6a2-d872ac7b9b4a\") " Nov 03 13:39:45.700866 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:39:45.700838 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17976293-ef64-4977-a6a2-d872ac7b9b4a-plugins-config-volume" pod "17976293-ef64-4977-a6a2-d872ac7b9b4a" (UID: "17976293-ef64-4977-a6a2-d872ac7b9b4a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:39:45.701205 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:39:45.701190 5024 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17976293-ef64-4977-a6a2-d872ac7b9b4a-plugins-config-volume\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:39:45.703467 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:39:45.703431 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17976293-ef64-4977-a6a2-d872ac7b9b4a-kube-api-access-94hzp" pod "17976293-ef64-4977-a6a2-d872ac7b9b4a" (UID: "17976293-ef64-4977-a6a2-d872ac7b9b4a"). InnerVolumeSpecName "kube-api-access-94hzp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:39:45.802179 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:39:45.802138 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-94hzp\" (UniqueName: \"kubernetes.io/projected/17976293-ef64-4977-a6a2-d872ac7b9b4a-kube-api-access-94hzp\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:39:46.372719 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:39:46.372620 5024 generic.go:356] "Generic (PLEG): container finished" podID="17976293-ef64-4977-a6a2-d872ac7b9b4a" containerID="2e449a1cdee3b0a465913cc39dfa89ba62a396f845fe3224528fcc0fe9c9a9b7" exitCode=0 Nov 03 13:39:46.372719 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:39:46.372690 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-c381a7-558f76676f-98tbx" Nov 03 13:39:46.372719 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:39:46.372697 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-c381a7-558f76676f-98tbx" event={"ID":"17976293-ef64-4977-a6a2-d872ac7b9b4a","Type":"ContainerDied","Data":"2e449a1cdee3b0a465913cc39dfa89ba62a396f845fe3224528fcc0fe9c9a9b7"} Nov 03 13:39:46.372981 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:39:46.372774 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-c381a7-558f76676f-98tbx" event={"ID":"17976293-ef64-4977-a6a2-d872ac7b9b4a","Type":"ContainerDied","Data":"b00ccff9d66e7fc2c879b952bbeda82c65251e2d1365ef63c394b00d123109e5"} Nov 03 13:39:46.372981 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:39:46.372803 5024 scope.go:122] "RemoveContainer" containerID="2e449a1cdee3b0a465913cc39dfa89ba62a396f845fe3224528fcc0fe9c9a9b7" Nov 03 13:39:46.384171 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:39:46.384143 5024 scope.go:122] "RemoveContainer" containerID="2e449a1cdee3b0a465913cc39dfa89ba62a396f845fe3224528fcc0fe9c9a9b7" Nov 03 13:39:46.384627 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:39:46.384597 5024 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e449a1cdee3b0a465913cc39dfa89ba62a396f845fe3224528fcc0fe9c9a9b7\": container with ID starting with 2e449a1cdee3b0a465913cc39dfa89ba62a396f845fe3224528fcc0fe9c9a9b7 not found: ID does not exist" containerID="2e449a1cdee3b0a465913cc39dfa89ba62a396f845fe3224528fcc0fe9c9a9b7" Nov 03 13:39:46.384723 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:39:46.384640 5024 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e449a1cdee3b0a465913cc39dfa89ba62a396f845fe3224528fcc0fe9c9a9b7"} err="failed to get container status \"2e449a1cdee3b0a465913cc39dfa89ba62a396f845fe3224528fcc0fe9c9a9b7\": rpc error: code = NotFound desc = could not find container \"2e449a1cdee3b0a465913cc39dfa89ba62a396f845fe3224528fcc0fe9c9a9b7\": container with ID starting with 2e449a1cdee3b0a465913cc39dfa89ba62a396f845fe3224528fcc0fe9c9a9b7 not found: ID does not exist" Nov 03 13:39:46.400545 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:39:46.400512 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-c381a7-558f76676f-98tbx"] Nov 03 13:39:46.417584 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:39:46.417549 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-c381a7-558f76676f-98tbx"] Nov 03 13:39:47.329035 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:39:47.329000 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-c381a7-7f58d8cb94-nbflc"] Nov 03 13:39:47.329824 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:39:47.329787 5024 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c381a7-7f58d8cb94-nbflc" podUID="a5bccdb6-fde7-475e-bb9e-a256e83a83d1" containerName="maas-api" containerID="cri-o://0040d1b0f3a658cca2dac30bb14fdb049d2aa37804e308dc50d38ba9f38eba62" gracePeriod=30 Nov 03 13:39:47.983787 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:39:47.983756 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c381a7-7f58d8cb94-nbflc" Nov 03 13:39:48.023786 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:39:48.023747 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a5bccdb6-fde7-475e-bb9e-a256e83a83d1-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a5bccdb6-fde7-475e-bb9e-a256e83a83d1-metrics-certs\") pod \"a5bccdb6-fde7-475e-bb9e-a256e83a83d1\" (UID: \"a5bccdb6-fde7-475e-bb9e-a256e83a83d1\") " Nov 03 13:39:48.024095 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:39:48.024071 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a5bccdb6-fde7-475e-bb9e-a256e83a83d1-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a5bccdb6-fde7-475e-bb9e-a256e83a83d1-maas-api-tls\") pod \"a5bccdb6-fde7-475e-bb9e-a256e83a83d1\" (UID: \"a5bccdb6-fde7-475e-bb9e-a256e83a83d1\") " Nov 03 13:39:48.024197 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:39:48.024180 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5bccdb6-fde7-475e-bb9e-a256e83a83d1-kube-api-access-9g72s\" (UniqueName: \"kubernetes.io/projected/a5bccdb6-fde7-475e-bb9e-a256e83a83d1-kube-api-access-9g72s\") pod \"a5bccdb6-fde7-475e-bb9e-a256e83a83d1\" (UID: \"a5bccdb6-fde7-475e-bb9e-a256e83a83d1\") " Nov 03 13:39:48.024261 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:39:48.024235 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a5bccdb6-fde7-475e-bb9e-a256e83a83d1-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a5bccdb6-fde7-475e-bb9e-a256e83a83d1-openshift-service-ca\") pod \"a5bccdb6-fde7-475e-bb9e-a256e83a83d1\" (UID: \"a5bccdb6-fde7-475e-bb9e-a256e83a83d1\") " Nov 03 13:39:48.025477 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:39:48.025448 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5bccdb6-fde7-475e-bb9e-a256e83a83d1-openshift-service-ca" pod "a5bccdb6-fde7-475e-bb9e-a256e83a83d1" (UID: "a5bccdb6-fde7-475e-bb9e-a256e83a83d1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:39:48.027234 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:39:48.027203 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a5bccdb6-fde7-475e-bb9e-a256e83a83d1-maas-api-tls" pod "a5bccdb6-fde7-475e-bb9e-a256e83a83d1" (UID: "a5bccdb6-fde7-475e-bb9e-a256e83a83d1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 13:39:48.027362 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:39:48.027311 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5bccdb6-fde7-475e-bb9e-a256e83a83d1-kube-api-access-9g72s" pod "a5bccdb6-fde7-475e-bb9e-a256e83a83d1" (UID: "a5bccdb6-fde7-475e-bb9e-a256e83a83d1"). InnerVolumeSpecName "kube-api-access-9g72s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:39:48.027506 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:39:48.027482 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a5bccdb6-fde7-475e-bb9e-a256e83a83d1-metrics-certs" pod "a5bccdb6-fde7-475e-bb9e-a256e83a83d1" (UID: "a5bccdb6-fde7-475e-bb9e-a256e83a83d1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 13:39:48.111783 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:39:48.111740 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="17976293-ef64-4977-a6a2-d872ac7b9b4a" path="/var/lib/kubelet/pods/17976293-ef64-4977-a6a2-d872ac7b9b4a/volumes" Nov 03 13:39:48.131039 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:39:48.130834 5024 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a5bccdb6-fde7-475e-bb9e-a256e83a83d1-maas-api-tls\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:39:48.131356 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:39:48.131330 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9g72s\" (UniqueName: \"kubernetes.io/projected/a5bccdb6-fde7-475e-bb9e-a256e83a83d1-kube-api-access-9g72s\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:39:48.131633 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:39:48.131592 5024 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a5bccdb6-fde7-475e-bb9e-a256e83a83d1-openshift-service-ca\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:39:48.131764 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:39:48.131749 5024 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a5bccdb6-fde7-475e-bb9e-a256e83a83d1-metrics-certs\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:39:48.396036 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:39:48.396002 5024 generic.go:356] "Generic (PLEG): container finished" podID="a5bccdb6-fde7-475e-bb9e-a256e83a83d1" containerID="0040d1b0f3a658cca2dac30bb14fdb049d2aa37804e308dc50d38ba9f38eba62" exitCode=0 Nov 03 13:39:48.396495 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:39:48.396046 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c381a7-7f58d8cb94-nbflc" Nov 03 13:39:48.396495 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:39:48.396083 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c381a7-7f58d8cb94-nbflc" event={"ID":"a5bccdb6-fde7-475e-bb9e-a256e83a83d1","Type":"ContainerDied","Data":"0040d1b0f3a658cca2dac30bb14fdb049d2aa37804e308dc50d38ba9f38eba62"} Nov 03 13:39:48.396495 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:39:48.396138 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c381a7-7f58d8cb94-nbflc" event={"ID":"a5bccdb6-fde7-475e-bb9e-a256e83a83d1","Type":"ContainerDied","Data":"86f18dce1beabd90cf4c3c85311089bbe6cbb65604c4f4bdfca4e7a8ada2449d"} Nov 03 13:39:48.396495 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:39:48.396162 5024 scope.go:122] "RemoveContainer" containerID="0040d1b0f3a658cca2dac30bb14fdb049d2aa37804e308dc50d38ba9f38eba62" Nov 03 13:39:48.407432 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:39:48.407409 5024 scope.go:122] "RemoveContainer" containerID="0040d1b0f3a658cca2dac30bb14fdb049d2aa37804e308dc50d38ba9f38eba62" Nov 03 13:39:48.407783 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:39:48.407761 5024 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0040d1b0f3a658cca2dac30bb14fdb049d2aa37804e308dc50d38ba9f38eba62\": container with ID starting with 0040d1b0f3a658cca2dac30bb14fdb049d2aa37804e308dc50d38ba9f38eba62 not found: ID does not exist" containerID="0040d1b0f3a658cca2dac30bb14fdb049d2aa37804e308dc50d38ba9f38eba62" Nov 03 13:39:48.407841 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:39:48.407794 5024 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0040d1b0f3a658cca2dac30bb14fdb049d2aa37804e308dc50d38ba9f38eba62"} err="failed to get container status \"0040d1b0f3a658cca2dac30bb14fdb049d2aa37804e308dc50d38ba9f38eba62\": rpc error: code = NotFound desc = could not find container \"0040d1b0f3a658cca2dac30bb14fdb049d2aa37804e308dc50d38ba9f38eba62\": container with ID starting with 0040d1b0f3a658cca2dac30bb14fdb049d2aa37804e308dc50d38ba9f38eba62 not found: ID does not exist" Nov 03 13:39:48.422927 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:39:48.422889 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-c381a7-7f58d8cb94-nbflc"] Nov 03 13:39:48.427472 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:39:48.427441 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-c381a7-7f58d8cb94-nbflc"] Nov 03 13:39:50.112350 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:39:50.112304 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a5bccdb6-fde7-475e-bb9e-a256e83a83d1" path="/var/lib/kubelet/pods/a5bccdb6-fde7-475e-bb9e-a256e83a83d1/volumes" Nov 03 13:39:52.696318 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:39:52.696123 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-cd80b3/noten-model-cd80b3-kserve-6789465645-pr6c4"] Nov 03 13:39:56.049527 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:39:56.049486 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-7tc2j"] Nov 03 13:39:56.049950 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:39:56.049752 5024 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-7tc2j" podUID="b61db32a-7a3b-4aff-b331-f7888a1a4e0e" containerName="istio-proxy" containerID="cri-o://c18e004cc03ecc2eb605c781789855e30f789cdb09d599e4ef0ce83f11b920a6" gracePeriod=30 Nov 03 13:39:56.053422 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:39:56.053392 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-fglkk"] Nov 03 13:39:56.053650 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:39:56.053621 5024 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-fglkk" podUID="e6d115b8-3b33-43c6-890b-a87bb608444b" containerName="istio-proxy" containerID="cri-o://e3d4317b51fa508f0711383ce8b044eb1bf9c9b7003d31e0cd4f1526b551615a" gracePeriod=30 Nov 03 13:39:58.503848 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:39:58.503801 5024 patch_prober.go:29] interesting pod/e2e-worker-w4-c381a7-openshift-default-78bcd4695-7tc2j container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 13:39:58.504261 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:39:58.503891 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-7tc2j" podUID="b61db32a-7a3b-4aff-b331-f7888a1a4e0e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 13:40:01.405620 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:40:01.405587 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-fglkk" Nov 03 13:40:01.487214 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:40:01.487176 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-7tc2j" Nov 03 13:40:01.530311 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:40:01.530073 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6d115b8-3b33-43c6-890b-a87bb608444b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6d115b8-3b33-43c6-890b-a87bb608444b-credential-socket\") pod \"e6d115b8-3b33-43c6-890b-a87bb608444b\" (UID: \"e6d115b8-3b33-43c6-890b-a87bb608444b\") " Nov 03 13:40:01.530311 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:40:01.530213 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6d115b8-3b33-43c6-890b-a87bb608444b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6d115b8-3b33-43c6-890b-a87bb608444b-istio-data\") pod \"e6d115b8-3b33-43c6-890b-a87bb608444b\" (UID: \"e6d115b8-3b33-43c6-890b-a87bb608444b\") " Nov 03 13:40:01.530311 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:40:01.530302 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e6d115b8-3b33-43c6-890b-a87bb608444b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6d115b8-3b33-43c6-890b-a87bb608444b-istio-podinfo\") pod \"e6d115b8-3b33-43c6-890b-a87bb608444b\" (UID: \"e6d115b8-3b33-43c6-890b-a87bb608444b\") " Nov 03 13:40:01.530633 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:40:01.530370 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6d115b8-3b33-43c6-890b-a87bb608444b-istio-token\" (UniqueName: \"kubernetes.io/projected/e6d115b8-3b33-43c6-890b-a87bb608444b-istio-token\") pod \"e6d115b8-3b33-43c6-890b-a87bb608444b\" (UID: \"e6d115b8-3b33-43c6-890b-a87bb608444b\") " Nov 03 13:40:01.530633 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:40:01.530465 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e6d115b8-3b33-43c6-890b-a87bb608444b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6d115b8-3b33-43c6-890b-a87bb608444b-istiod-ca-cert\") pod \"e6d115b8-3b33-43c6-890b-a87bb608444b\" (UID: \"e6d115b8-3b33-43c6-890b-a87bb608444b\") " Nov 03 13:40:01.530633 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:40:01.530514 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6d115b8-3b33-43c6-890b-a87bb608444b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6d115b8-3b33-43c6-890b-a87bb608444b-workload-certs\") pod \"e6d115b8-3b33-43c6-890b-a87bb608444b\" (UID: \"e6d115b8-3b33-43c6-890b-a87bb608444b\") " Nov 03 13:40:01.530789 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:40:01.530652 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6d115b8-3b33-43c6-890b-a87bb608444b-kube-api-access-bn7nf\" (UniqueName: \"kubernetes.io/projected/e6d115b8-3b33-43c6-890b-a87bb608444b-kube-api-access-bn7nf\") pod \"e6d115b8-3b33-43c6-890b-a87bb608444b\" (UID: \"e6d115b8-3b33-43c6-890b-a87bb608444b\") " Nov 03 13:40:01.530789 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:40:01.530697 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6d115b8-3b33-43c6-890b-a87bb608444b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6d115b8-3b33-43c6-890b-a87bb608444b-workload-socket\") pod \"e6d115b8-3b33-43c6-890b-a87bb608444b\" (UID: \"e6d115b8-3b33-43c6-890b-a87bb608444b\") " Nov 03 13:40:01.530789 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:40:01.530776 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6d115b8-3b33-43c6-890b-a87bb608444b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6d115b8-3b33-43c6-890b-a87bb608444b-istio-envoy\") pod \"e6d115b8-3b33-43c6-890b-a87bb608444b\" (UID: \"e6d115b8-3b33-43c6-890b-a87bb608444b\") " Nov 03 13:40:01.544887 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:40:01.541623 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6d115b8-3b33-43c6-890b-a87bb608444b-workload-certs" pod "e6d115b8-3b33-43c6-890b-a87bb608444b" (UID: "e6d115b8-3b33-43c6-890b-a87bb608444b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:40:01.544887 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:40:01.542022 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6d115b8-3b33-43c6-890b-a87bb608444b-credential-socket" pod "e6d115b8-3b33-43c6-890b-a87bb608444b" (UID: "e6d115b8-3b33-43c6-890b-a87bb608444b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:40:01.544887 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:40:01.542467 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6d115b8-3b33-43c6-890b-a87bb608444b-istiod-ca-cert" pod "e6d115b8-3b33-43c6-890b-a87bb608444b" (UID: "e6d115b8-3b33-43c6-890b-a87bb608444b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:40:01.544887 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:40:01.544822 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6d115b8-3b33-43c6-890b-a87bb608444b-workload-socket" pod "e6d115b8-3b33-43c6-890b-a87bb608444b" (UID: "e6d115b8-3b33-43c6-890b-a87bb608444b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:40:01.545440 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:40:01.545401 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6d115b8-3b33-43c6-890b-a87bb608444b-istio-data" pod "e6d115b8-3b33-43c6-890b-a87bb608444b" (UID: "e6d115b8-3b33-43c6-890b-a87bb608444b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:40:01.564573 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:40:01.563406 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e6d115b8-3b33-43c6-890b-a87bb608444b-istio-podinfo" pod "e6d115b8-3b33-43c6-890b-a87bb608444b" (UID: "e6d115b8-3b33-43c6-890b-a87bb608444b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 13:40:01.575957 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:40:01.575891 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6d115b8-3b33-43c6-890b-a87bb608444b-kube-api-access-bn7nf" pod "e6d115b8-3b33-43c6-890b-a87bb608444b" (UID: "e6d115b8-3b33-43c6-890b-a87bb608444b"). InnerVolumeSpecName "kube-api-access-bn7nf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:40:01.580496 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:40:01.577151 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6d115b8-3b33-43c6-890b-a87bb608444b-istio-token" pod "e6d115b8-3b33-43c6-890b-a87bb608444b" (UID: "e6d115b8-3b33-43c6-890b-a87bb608444b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:40:01.594407 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:40:01.593384 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6d115b8-3b33-43c6-890b-a87bb608444b-istio-envoy" pod "e6d115b8-3b33-43c6-890b-a87bb608444b" (UID: "e6d115b8-3b33-43c6-890b-a87bb608444b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:40:01.632827 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:40:01.632737 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b61db32a-7a3b-4aff-b331-f7888a1a4e0e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b61db32a-7a3b-4aff-b331-f7888a1a4e0e-credential-socket\") pod \"b61db32a-7a3b-4aff-b331-f7888a1a4e0e\" (UID: \"b61db32a-7a3b-4aff-b331-f7888a1a4e0e\") " Nov 03 13:40:01.633010 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:40:01.632843 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b61db32a-7a3b-4aff-b331-f7888a1a4e0e-istio-token\" (UniqueName: \"kubernetes.io/projected/b61db32a-7a3b-4aff-b331-f7888a1a4e0e-istio-token\") pod \"b61db32a-7a3b-4aff-b331-f7888a1a4e0e\" (UID: \"b61db32a-7a3b-4aff-b331-f7888a1a4e0e\") " Nov 03 13:40:01.633010 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:40:01.632881 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b61db32a-7a3b-4aff-b331-f7888a1a4e0e-kube-api-access-qnjq6\" (UniqueName: \"kubernetes.io/projected/b61db32a-7a3b-4aff-b331-f7888a1a4e0e-kube-api-access-qnjq6\") pod \"b61db32a-7a3b-4aff-b331-f7888a1a4e0e\" (UID: \"b61db32a-7a3b-4aff-b331-f7888a1a4e0e\") " Nov 03 13:40:01.633138 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:40:01.633008 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b61db32a-7a3b-4aff-b331-f7888a1a4e0e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b61db32a-7a3b-4aff-b331-f7888a1a4e0e-workload-socket\") pod \"b61db32a-7a3b-4aff-b331-f7888a1a4e0e\" (UID: \"b61db32a-7a3b-4aff-b331-f7888a1a4e0e\") " Nov 03 13:40:01.633202 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:40:01.633137 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b61db32a-7a3b-4aff-b331-f7888a1a4e0e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b61db32a-7a3b-4aff-b331-f7888a1a4e0e-istio-podinfo\") pod \"b61db32a-7a3b-4aff-b331-f7888a1a4e0e\" (UID: \"b61db32a-7a3b-4aff-b331-f7888a1a4e0e\") " Nov 03 13:40:01.633202 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:40:01.633144 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b61db32a-7a3b-4aff-b331-f7888a1a4e0e-credential-socket" pod "b61db32a-7a3b-4aff-b331-f7888a1a4e0e" (UID: "b61db32a-7a3b-4aff-b331-f7888a1a4e0e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:40:01.633202 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:40:01.633186 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b61db32a-7a3b-4aff-b331-f7888a1a4e0e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b61db32a-7a3b-4aff-b331-f7888a1a4e0e-istiod-ca-cert\") pod \"b61db32a-7a3b-4aff-b331-f7888a1a4e0e\" (UID: \"b61db32a-7a3b-4aff-b331-f7888a1a4e0e\") " Nov 03 13:40:01.633412 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:40:01.633269 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b61db32a-7a3b-4aff-b331-f7888a1a4e0e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b61db32a-7a3b-4aff-b331-f7888a1a4e0e-istio-envoy\") pod \"b61db32a-7a3b-4aff-b331-f7888a1a4e0e\" (UID: \"b61db32a-7a3b-4aff-b331-f7888a1a4e0e\") " Nov 03 13:40:01.633412 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:40:01.633322 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b61db32a-7a3b-4aff-b331-f7888a1a4e0e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b61db32a-7a3b-4aff-b331-f7888a1a4e0e-workload-certs\") pod \"b61db32a-7a3b-4aff-b331-f7888a1a4e0e\" (UID: \"b61db32a-7a3b-4aff-b331-f7888a1a4e0e\") " Nov 03 13:40:01.633412 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:40:01.633377 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b61db32a-7a3b-4aff-b331-f7888a1a4e0e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b61db32a-7a3b-4aff-b331-f7888a1a4e0e-istio-data\") pod \"b61db32a-7a3b-4aff-b331-f7888a1a4e0e\" (UID: \"b61db32a-7a3b-4aff-b331-f7888a1a4e0e\") " Nov 03 13:40:01.633672 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:40:01.633646 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b61db32a-7a3b-4aff-b331-f7888a1a4e0e-istiod-ca-cert" pod "b61db32a-7a3b-4aff-b331-f7888a1a4e0e" (UID: "b61db32a-7a3b-4aff-b331-f7888a1a4e0e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:40:01.633741 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:40:01.633664 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b61db32a-7a3b-4aff-b331-f7888a1a4e0e-workload-socket" pod "b61db32a-7a3b-4aff-b331-f7888a1a4e0e" (UID: "b61db32a-7a3b-4aff-b331-f7888a1a4e0e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:40:01.635420 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:40:01.634070 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b61db32a-7a3b-4aff-b331-f7888a1a4e0e-istio-data" pod "b61db32a-7a3b-4aff-b331-f7888a1a4e0e" (UID: "b61db32a-7a3b-4aff-b331-f7888a1a4e0e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:40:01.635420 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:40:01.634368 5024 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6d115b8-3b33-43c6-890b-a87bb608444b-istiod-ca-cert\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:40:01.635420 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:40:01.634390 5024 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6d115b8-3b33-43c6-890b-a87bb608444b-workload-certs\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:40:01.635420 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:40:01.634460 5024 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b61db32a-7a3b-4aff-b331-f7888a1a4e0e-workload-socket\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:40:01.635420 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:40:01.634613 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bn7nf\" (UniqueName: \"kubernetes.io/projected/e6d115b8-3b33-43c6-890b-a87bb608444b-kube-api-access-bn7nf\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:40:01.635420 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:40:01.634644 5024 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6d115b8-3b33-43c6-890b-a87bb608444b-workload-socket\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:40:01.635420 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:40:01.634677 5024 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6d115b8-3b33-43c6-890b-a87bb608444b-istio-envoy\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:40:01.635420 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:40:01.634691 5024 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b61db32a-7a3b-4aff-b331-f7888a1a4e0e-istiod-ca-cert\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:40:01.635420 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:40:01.635154 5024 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6d115b8-3b33-43c6-890b-a87bb608444b-credential-socket\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:40:01.635420 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:40:01.635232 5024 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b61db32a-7a3b-4aff-b331-f7888a1a4e0e-istio-data\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:40:01.635420 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:40:01.635252 5024 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6d115b8-3b33-43c6-890b-a87bb608444b-istio-data\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:40:01.635420 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:40:01.635266 5024 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b61db32a-7a3b-4aff-b331-f7888a1a4e0e-credential-socket\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:40:01.635420 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:40:01.635359 5024 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6d115b8-3b33-43c6-890b-a87bb608444b-istio-podinfo\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:40:01.635420 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:40:01.635380 5024 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6d115b8-3b33-43c6-890b-a87bb608444b-istio-token\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:40:01.636350 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:40:01.636314 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b61db32a-7a3b-4aff-b331-f7888a1a4e0e-workload-certs" pod "b61db32a-7a3b-4aff-b331-f7888a1a4e0e" (UID: "b61db32a-7a3b-4aff-b331-f7888a1a4e0e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:40:01.638551 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:40:01.638525 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b61db32a-7a3b-4aff-b331-f7888a1a4e0e-istio-envoy" pod "b61db32a-7a3b-4aff-b331-f7888a1a4e0e" (UID: "b61db32a-7a3b-4aff-b331-f7888a1a4e0e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:40:01.639774 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:40:01.639642 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b61db32a-7a3b-4aff-b331-f7888a1a4e0e-istio-podinfo" pod "b61db32a-7a3b-4aff-b331-f7888a1a4e0e" (UID: "b61db32a-7a3b-4aff-b331-f7888a1a4e0e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 13:40:01.639774 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:40:01.639743 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b61db32a-7a3b-4aff-b331-f7888a1a4e0e-istio-token" pod "b61db32a-7a3b-4aff-b331-f7888a1a4e0e" (UID: "b61db32a-7a3b-4aff-b331-f7888a1a4e0e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:40:01.640996 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:40:01.640974 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b61db32a-7a3b-4aff-b331-f7888a1a4e0e-kube-api-access-qnjq6" pod "b61db32a-7a3b-4aff-b331-f7888a1a4e0e" (UID: "b61db32a-7a3b-4aff-b331-f7888a1a4e0e"). InnerVolumeSpecName "kube-api-access-qnjq6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:40:01.736972 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:40:01.736931 5024 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b61db32a-7a3b-4aff-b331-f7888a1a4e0e-istio-podinfo\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:40:01.736972 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:40:01.736969 5024 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b61db32a-7a3b-4aff-b331-f7888a1a4e0e-istio-envoy\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:40:01.736972 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:40:01.736979 5024 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b61db32a-7a3b-4aff-b331-f7888a1a4e0e-workload-certs\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:40:01.737229 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:40:01.736990 5024 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b61db32a-7a3b-4aff-b331-f7888a1a4e0e-istio-token\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:40:01.737229 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:40:01.737000 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qnjq6\" (UniqueName: \"kubernetes.io/projected/b61db32a-7a3b-4aff-b331-f7888a1a4e0e-kube-api-access-qnjq6\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:40:01.903236 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:40:01.903139 5024 generic.go:356] "Generic (PLEG): container finished" podID="b61db32a-7a3b-4aff-b331-f7888a1a4e0e" containerID="c18e004cc03ecc2eb605c781789855e30f789cdb09d599e4ef0ce83f11b920a6" exitCode=0 Nov 03 13:40:01.903236 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:40:01.903200 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-7tc2j" Nov 03 13:40:01.903506 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:40:01.903230 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-7tc2j" event={"ID":"b61db32a-7a3b-4aff-b331-f7888a1a4e0e","Type":"ContainerDied","Data":"c18e004cc03ecc2eb605c781789855e30f789cdb09d599e4ef0ce83f11b920a6"} Nov 03 13:40:01.903506 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:40:01.903328 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-7tc2j" event={"ID":"b61db32a-7a3b-4aff-b331-f7888a1a4e0e","Type":"ContainerDied","Data":"784edf9c87f371fbe139fb67db023e3de59bf7bb65ff5a8ce8e04aa7fe62122b"} Nov 03 13:40:01.903506 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:40:01.903363 5024 scope.go:122] "RemoveContainer" containerID="c18e004cc03ecc2eb605c781789855e30f789cdb09d599e4ef0ce83f11b920a6" Nov 03 13:40:01.904994 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:40:01.904969 5024 generic.go:356] "Generic (PLEG): container finished" podID="e6d115b8-3b33-43c6-890b-a87bb608444b" containerID="e3d4317b51fa508f0711383ce8b044eb1bf9c9b7003d31e0cd4f1526b551615a" exitCode=0 Nov 03 13:40:01.905096 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:40:01.905026 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-fglkk" event={"ID":"e6d115b8-3b33-43c6-890b-a87bb608444b","Type":"ContainerDied","Data":"e3d4317b51fa508f0711383ce8b044eb1bf9c9b7003d31e0cd4f1526b551615a"} Nov 03 13:40:01.905096 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:40:01.905037 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-fglkk" Nov 03 13:40:01.905096 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:40:01.905060 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-fglkk" event={"ID":"e6d115b8-3b33-43c6-890b-a87bb608444b","Type":"ContainerDied","Data":"d9577aea3e9d0705d737162c904d6c84a9b8f4aef6e57356126a781ef383938c"} Nov 03 13:40:01.918244 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:40:01.918219 5024 scope.go:122] "RemoveContainer" containerID="c18e004cc03ecc2eb605c781789855e30f789cdb09d599e4ef0ce83f11b920a6" Nov 03 13:40:01.918599 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:40:01.918578 5024 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c18e004cc03ecc2eb605c781789855e30f789cdb09d599e4ef0ce83f11b920a6\": container with ID starting with c18e004cc03ecc2eb605c781789855e30f789cdb09d599e4ef0ce83f11b920a6 not found: ID does not exist" containerID="c18e004cc03ecc2eb605c781789855e30f789cdb09d599e4ef0ce83f11b920a6" Nov 03 13:40:01.918677 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:40:01.918607 5024 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c18e004cc03ecc2eb605c781789855e30f789cdb09d599e4ef0ce83f11b920a6"} err="failed to get container status \"c18e004cc03ecc2eb605c781789855e30f789cdb09d599e4ef0ce83f11b920a6\": rpc error: code = NotFound desc = could not find container \"c18e004cc03ecc2eb605c781789855e30f789cdb09d599e4ef0ce83f11b920a6\": container with ID starting with c18e004cc03ecc2eb605c781789855e30f789cdb09d599e4ef0ce83f11b920a6 not found: ID does not exist" Nov 03 13:40:01.918677 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:40:01.918626 5024 scope.go:122] "RemoveContainer" containerID="e3d4317b51fa508f0711383ce8b044eb1bf9c9b7003d31e0cd4f1526b551615a" Nov 03 13:40:01.931193 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:40:01.931166 5024 scope.go:122] "RemoveContainer" containerID="e3d4317b51fa508f0711383ce8b044eb1bf9c9b7003d31e0cd4f1526b551615a" Nov 03 13:40:01.931586 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:40:01.931559 5024 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e3d4317b51fa508f0711383ce8b044eb1bf9c9b7003d31e0cd4f1526b551615a\": container with ID starting with e3d4317b51fa508f0711383ce8b044eb1bf9c9b7003d31e0cd4f1526b551615a not found: ID does not exist" containerID="e3d4317b51fa508f0711383ce8b044eb1bf9c9b7003d31e0cd4f1526b551615a" Nov 03 13:40:01.931682 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:40:01.931591 5024 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3d4317b51fa508f0711383ce8b044eb1bf9c9b7003d31e0cd4f1526b551615a"} err="failed to get container status \"e3d4317b51fa508f0711383ce8b044eb1bf9c9b7003d31e0cd4f1526b551615a\": rpc error: code = NotFound desc = could not find container \"e3d4317b51fa508f0711383ce8b044eb1bf9c9b7003d31e0cd4f1526b551615a\": container with ID starting with e3d4317b51fa508f0711383ce8b044eb1bf9c9b7003d31e0cd4f1526b551615a not found: ID does not exist" Nov 03 13:40:01.939497 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:40:01.939464 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-7tc2j"] Nov 03 13:40:01.948311 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:40:01.948258 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-7tc2j"] Nov 03 13:40:01.959356 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:40:01.959316 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-fglkk"] Nov 03 13:40:01.963588 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:40:01.963553 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-c381a7-openshift-default-78bcd4695-fglkk"] Nov 03 13:40:02.191381 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:40:02.190266 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b61db32a-7a3b-4aff-b331-f7888a1a4e0e" path="/var/lib/kubelet/pods/b61db32a-7a3b-4aff-b331-f7888a1a4e0e/volumes" Nov 03 13:40:02.191381 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:40:02.191198 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e6d115b8-3b33-43c6-890b-a87bb608444b" path="/var/lib/kubelet/pods/e6d115b8-3b33-43c6-890b-a87bb608444b/volumes" Nov 03 13:40:07.913217 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:40:07.913161 5024 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-noten-cd80b3/noten-model-cd80b3-kserve-6789465645-pr6c4" podUID="c27e059d-8901-486f-9e7f-60fc2c2e7416" containerName="main" containerID="cri-o://5ba82370b3e521710eaea9b2482ecb5db28888fa4b273b37f9558128d55e3d95" gracePeriod=45 Nov 03 13:40:08.267850 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:40:08.267815 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-cd80b3/noten-model-cd80b3-kserve-6789465645-pr6c4" Nov 03 13:40:08.299962 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:40:08.299912 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c27e059d-8901-486f-9e7f-60fc2c2e7416-home\" (UniqueName: \"kubernetes.io/empty-dir/c27e059d-8901-486f-9e7f-60fc2c2e7416-home\") pod \"c27e059d-8901-486f-9e7f-60fc2c2e7416\" (UID: \"c27e059d-8901-486f-9e7f-60fc2c2e7416\") " Nov 03 13:40:08.300142 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:40:08.300005 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c27e059d-8901-486f-9e7f-60fc2c2e7416-tls-certs\" (UniqueName: \"kubernetes.io/secret/c27e059d-8901-486f-9e7f-60fc2c2e7416-tls-certs\") pod \"c27e059d-8901-486f-9e7f-60fc2c2e7416\" (UID: \"c27e059d-8901-486f-9e7f-60fc2c2e7416\") " Nov 03 13:40:08.300142 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:40:08.300100 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c27e059d-8901-486f-9e7f-60fc2c2e7416-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c27e059d-8901-486f-9e7f-60fc2c2e7416-tmp-dir\") pod \"c27e059d-8901-486f-9e7f-60fc2c2e7416\" (UID: \"c27e059d-8901-486f-9e7f-60fc2c2e7416\") " Nov 03 13:40:08.300271 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:40:08.300152 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c27e059d-8901-486f-9e7f-60fc2c2e7416-kube-api-access-fk7jk\" (UniqueName: \"kubernetes.io/projected/c27e059d-8901-486f-9e7f-60fc2c2e7416-kube-api-access-fk7jk\") pod \"c27e059d-8901-486f-9e7f-60fc2c2e7416\" (UID: \"c27e059d-8901-486f-9e7f-60fc2c2e7416\") " Nov 03 13:40:08.300271 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:40:08.300220 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c27e059d-8901-486f-9e7f-60fc2c2e7416-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c27e059d-8901-486f-9e7f-60fc2c2e7416-model-cache\") pod \"c27e059d-8901-486f-9e7f-60fc2c2e7416\" (UID: \"c27e059d-8901-486f-9e7f-60fc2c2e7416\") " Nov 03 13:40:08.300271 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:40:08.300254 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c27e059d-8901-486f-9e7f-60fc2c2e7416-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c27e059d-8901-486f-9e7f-60fc2c2e7416-dshm\") pod \"c27e059d-8901-486f-9e7f-60fc2c2e7416\" (UID: \"c27e059d-8901-486f-9e7f-60fc2c2e7416\") " Nov 03 13:40:08.301331 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:40:08.300751 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c27e059d-8901-486f-9e7f-60fc2c2e7416-model-cache" pod "c27e059d-8901-486f-9e7f-60fc2c2e7416" (UID: "c27e059d-8901-486f-9e7f-60fc2c2e7416"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:40:08.301331 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:40:08.301051 5024 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c27e059d-8901-486f-9e7f-60fc2c2e7416-model-cache\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:40:08.301331 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:40:08.301299 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c27e059d-8901-486f-9e7f-60fc2c2e7416-home" pod "c27e059d-8901-486f-9e7f-60fc2c2e7416" (UID: "c27e059d-8901-486f-9e7f-60fc2c2e7416"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:40:08.302038 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:40:08.302016 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c27e059d-8901-486f-9e7f-60fc2c2e7416-tmp-dir" pod "c27e059d-8901-486f-9e7f-60fc2c2e7416" (UID: "c27e059d-8901-486f-9e7f-60fc2c2e7416"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:40:08.304503 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:40:08.304466 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c27e059d-8901-486f-9e7f-60fc2c2e7416-dshm" pod "c27e059d-8901-486f-9e7f-60fc2c2e7416" (UID: "c27e059d-8901-486f-9e7f-60fc2c2e7416"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:40:08.305991 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:40:08.305962 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c27e059d-8901-486f-9e7f-60fc2c2e7416-kube-api-access-fk7jk" pod "c27e059d-8901-486f-9e7f-60fc2c2e7416" (UID: "c27e059d-8901-486f-9e7f-60fc2c2e7416"). InnerVolumeSpecName "kube-api-access-fk7jk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:40:08.318362 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:40:08.318320 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c27e059d-8901-486f-9e7f-60fc2c2e7416-tls-certs" pod "c27e059d-8901-486f-9e7f-60fc2c2e7416" (UID: "c27e059d-8901-486f-9e7f-60fc2c2e7416"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 13:40:08.402225 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:40:08.402172 5024 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c27e059d-8901-486f-9e7f-60fc2c2e7416-home\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:40:08.402225 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:40:08.402223 5024 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c27e059d-8901-486f-9e7f-60fc2c2e7416-tls-certs\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:40:08.402225 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:40:08.402234 5024 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c27e059d-8901-486f-9e7f-60fc2c2e7416-tmp-dir\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:40:08.402225 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:40:08.402244 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fk7jk\" (UniqueName: \"kubernetes.io/projected/c27e059d-8901-486f-9e7f-60fc2c2e7416-kube-api-access-fk7jk\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:40:08.402628 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:40:08.402256 5024 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c27e059d-8901-486f-9e7f-60fc2c2e7416-dshm\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:40:08.965941 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:40:08.965908 5024 generic.go:356] "Generic (PLEG): container finished" podID="c27e059d-8901-486f-9e7f-60fc2c2e7416" containerID="5ba82370b3e521710eaea9b2482ecb5db28888fa4b273b37f9558128d55e3d95" exitCode=0 Nov 03 13:40:08.966606 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:40:08.965956 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-cd80b3/noten-model-cd80b3-kserve-6789465645-pr6c4" event={"ID":"c27e059d-8901-486f-9e7f-60fc2c2e7416","Type":"ContainerDied","Data":"5ba82370b3e521710eaea9b2482ecb5db28888fa4b273b37f9558128d55e3d95"} Nov 03 13:40:08.966606 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:40:08.965984 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-cd80b3/noten-model-cd80b3-kserve-6789465645-pr6c4" Nov 03 13:40:08.966606 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:40:08.965996 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-cd80b3/noten-model-cd80b3-kserve-6789465645-pr6c4" event={"ID":"c27e059d-8901-486f-9e7f-60fc2c2e7416","Type":"ContainerDied","Data":"3dc59b1f30a661cf8aeee41c152243cb5f0af3369fbff2a34bc1a71cdc9424e4"} Nov 03 13:40:08.966606 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:40:08.966020 5024 scope.go:122] "RemoveContainer" containerID="5ba82370b3e521710eaea9b2482ecb5db28888fa4b273b37f9558128d55e3d95" Nov 03 13:40:08.978532 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:40:08.978461 5024 scope.go:122] "RemoveContainer" containerID="5ba82370b3e521710eaea9b2482ecb5db28888fa4b273b37f9558128d55e3d95" Nov 03 13:40:08.978967 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:40:08.978928 5024 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ba82370b3e521710eaea9b2482ecb5db28888fa4b273b37f9558128d55e3d95\": container with ID starting with 5ba82370b3e521710eaea9b2482ecb5db28888fa4b273b37f9558128d55e3d95 not found: ID does not exist" containerID="5ba82370b3e521710eaea9b2482ecb5db28888fa4b273b37f9558128d55e3d95" Nov 03 13:40:08.979053 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:40:08.978967 5024 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ba82370b3e521710eaea9b2482ecb5db28888fa4b273b37f9558128d55e3d95"} err="failed to get container status \"5ba82370b3e521710eaea9b2482ecb5db28888fa4b273b37f9558128d55e3d95\": rpc error: code = NotFound desc = could not find container \"5ba82370b3e521710eaea9b2482ecb5db28888fa4b273b37f9558128d55e3d95\": container with ID starting with 5ba82370b3e521710eaea9b2482ecb5db28888fa4b273b37f9558128d55e3d95 not found: ID does not exist" Nov 03 13:40:09.006430 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:40:09.006383 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-cd80b3/noten-model-cd80b3-kserve-6789465645-pr6c4"] Nov 03 13:40:09.010949 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:40:09.010914 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-noten-cd80b3/noten-model-cd80b3-kserve-6789465645-pr6c4"] Nov 03 13:40:09.512778 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:40:09.512739 5024 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d7mwz" containerName="extractor" Nov 03 13:40:10.128800 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:40:10.128756 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c27e059d-8901-486f-9e7f-60fc2c2e7416" path="/var/lib/kubelet/pods/c27e059d-8901-486f-9e7f-60fc2c2e7416/volumes" Nov 03 13:40:39.510452 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:40:39.510367 5024 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d7mwz" containerName="extractor" Nov 03 13:40:43.837235 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:40:43.837187 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-a62d22-6746df564f-wv77r"] Nov 03 13:40:43.841792 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:40:43.841760 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-a62d22-69fbf9644c-2svhb"] Nov 03 13:40:45.143011 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:40:45.142967 5024 scope.go:122] "RemoveContainer" containerID="856e798c4f8001afe2ebf9e60f45eab903bdd2fd33cfeffdaa7f430dd00e4129" Nov 03 13:40:47.294074 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:40:47.294020 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-08111b/premium-w1-08111b-kserve-57555d598-dx9g5" podUID="e45c569a-243f-49ab-8d74-dd08cbb6127c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.134:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 13:40:48.839206 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:40:48.839147 5024 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-a62d22-6746df564f-wv77r" podUID="327a26e7-752e-4271-a3eb-1100182124a0" containerName="payload-processing" containerID="cri-o://6b3106d0694a65dc70dcf23058c45ceff9ec6af61a7d112e09571410cfc2e036" gracePeriod=25 Nov 03 13:40:48.842752 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:40:48.842711 5024 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a62d22-69fbf9644c-2svhb" podUID="8a432763-9702-42b0-97e4-c2294b4fe866" containerName="payload-pre-processing" containerID="cri-o://39e97a01dd324a8ca31f278514a2a0cac2b1a5985ac9a20d076b011a62c45afe" gracePeriod=25 Nov 03 13:40:50.938214 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:40:50.938166 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-a62d22-openshift-default-d9c57bf65-pvtj7"] Nov 03 13:40:50.938752 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:40:50.938458 5024 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-a62d22-openshift-default-d9c57bf65-pvtj7" podUID="78398b98-5396-4f98-bce4-682198b80f4b" containerName="istio-proxy" containerID="cri-o://fd6a5b9b94d997fd6f479495e6afbe1a3fe56e9781b6deb4d2edb54647748d71" gracePeriod=30 Nov 03 13:40:51.938920 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:40:51.938873 5024 patch_prober.go:29] interesting pod/e2e-extmod-a-a62d22-openshift-default-d9c57bf65-pvtj7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 13:40:51.939402 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:40:51.938946 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-a62d22-openshift-default-d9c57bf65-pvtj7" podUID="78398b98-5396-4f98-bce4-682198b80f4b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 13:40:56.235436 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:40:56.235393 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-a62d22-openshift-default-d9c57bf65-pvtj7" Nov 03 13:40:56.277017 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:40:56.276974 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/78398b98-5396-4f98-bce4-682198b80f4b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/78398b98-5396-4f98-bce4-682198b80f4b-workload-certs\") pod \"78398b98-5396-4f98-bce4-682198b80f4b\" (UID: \"78398b98-5396-4f98-bce4-682198b80f4b\") " Nov 03 13:40:56.277219 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:40:56.277110 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/78398b98-5396-4f98-bce4-682198b80f4b-kube-api-access-4pc72\" (UniqueName: \"kubernetes.io/projected/78398b98-5396-4f98-bce4-682198b80f4b-kube-api-access-4pc72\") pod \"78398b98-5396-4f98-bce4-682198b80f4b\" (UID: \"78398b98-5396-4f98-bce4-682198b80f4b\") " Nov 03 13:40:56.277219 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:40:56.277192 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/78398b98-5396-4f98-bce4-682198b80f4b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/78398b98-5396-4f98-bce4-682198b80f4b-credential-socket\") pod \"78398b98-5396-4f98-bce4-682198b80f4b\" (UID: \"78398b98-5396-4f98-bce4-682198b80f4b\") " Nov 03 13:40:56.277362 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:40:56.277338 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/78398b98-5396-4f98-bce4-682198b80f4b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/78398b98-5396-4f98-bce4-682198b80f4b-istio-data\") pod \"78398b98-5396-4f98-bce4-682198b80f4b\" (UID: \"78398b98-5396-4f98-bce4-682198b80f4b\") " Nov 03 13:40:56.277485 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:40:56.277458 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78398b98-5396-4f98-bce4-682198b80f4b-workload-certs" pod "78398b98-5396-4f98-bce4-682198b80f4b" (UID: "78398b98-5396-4f98-bce4-682198b80f4b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:40:56.278335 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:40:56.278306 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78398b98-5396-4f98-bce4-682198b80f4b-istio-data" pod "78398b98-5396-4f98-bce4-682198b80f4b" (UID: "78398b98-5396-4f98-bce4-682198b80f4b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:40:56.278460 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:40:56.278378 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78398b98-5396-4f98-bce4-682198b80f4b-credential-socket" pod "78398b98-5396-4f98-bce4-682198b80f4b" (UID: "78398b98-5396-4f98-bce4-682198b80f4b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:40:56.278460 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:40:56.278420 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/78398b98-5396-4f98-bce4-682198b80f4b-istio-token\" (UniqueName: \"kubernetes.io/projected/78398b98-5396-4f98-bce4-682198b80f4b-istio-token\") pod \"78398b98-5396-4f98-bce4-682198b80f4b\" (UID: \"78398b98-5396-4f98-bce4-682198b80f4b\") " Nov 03 13:40:56.278582 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:40:56.278469 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/78398b98-5396-4f98-bce4-682198b80f4b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/78398b98-5396-4f98-bce4-682198b80f4b-istiod-ca-cert\") pod \"78398b98-5396-4f98-bce4-682198b80f4b\" (UID: \"78398b98-5396-4f98-bce4-682198b80f4b\") " Nov 03 13:40:56.278582 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:40:56.278575 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/78398b98-5396-4f98-bce4-682198b80f4b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/78398b98-5396-4f98-bce4-682198b80f4b-istio-podinfo\") pod \"78398b98-5396-4f98-bce4-682198b80f4b\" (UID: \"78398b98-5396-4f98-bce4-682198b80f4b\") " Nov 03 13:40:56.278689 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:40:56.278636 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/78398b98-5396-4f98-bce4-682198b80f4b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/78398b98-5396-4f98-bce4-682198b80f4b-istio-envoy\") pod \"78398b98-5396-4f98-bce4-682198b80f4b\" (UID: \"78398b98-5396-4f98-bce4-682198b80f4b\") " Nov 03 13:40:56.278748 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:40:56.278729 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/78398b98-5396-4f98-bce4-682198b80f4b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/78398b98-5396-4f98-bce4-682198b80f4b-workload-socket\") pod \"78398b98-5396-4f98-bce4-682198b80f4b\" (UID: \"78398b98-5396-4f98-bce4-682198b80f4b\") " Nov 03 13:40:56.281521 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:40:56.281447 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78398b98-5396-4f98-bce4-682198b80f4b-kube-api-access-4pc72" pod "78398b98-5396-4f98-bce4-682198b80f4b" (UID: "78398b98-5396-4f98-bce4-682198b80f4b"). InnerVolumeSpecName "kube-api-access-4pc72". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:40:56.291639 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:40:56.291579 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78398b98-5396-4f98-bce4-682198b80f4b-istio-envoy" pod "78398b98-5396-4f98-bce4-682198b80f4b" (UID: "78398b98-5396-4f98-bce4-682198b80f4b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:40:56.299437 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:40:56.299401 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78398b98-5396-4f98-bce4-682198b80f4b-workload-socket" pod "78398b98-5396-4f98-bce4-682198b80f4b" (UID: "78398b98-5396-4f98-bce4-682198b80f4b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:40:56.300207 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:40:56.300175 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78398b98-5396-4f98-bce4-682198b80f4b-istio-token" pod "78398b98-5396-4f98-bce4-682198b80f4b" (UID: "78398b98-5396-4f98-bce4-682198b80f4b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:40:56.301430 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:40:56.301405 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/78398b98-5396-4f98-bce4-682198b80f4b-istio-podinfo" pod "78398b98-5396-4f98-bce4-682198b80f4b" (UID: "78398b98-5396-4f98-bce4-682198b80f4b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 13:40:56.302122 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:40:56.302095 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78398b98-5396-4f98-bce4-682198b80f4b-istiod-ca-cert" pod "78398b98-5396-4f98-bce4-682198b80f4b" (UID: "78398b98-5396-4f98-bce4-682198b80f4b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:40:56.381611 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:40:56.381568 5024 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/78398b98-5396-4f98-bce4-682198b80f4b-credential-socket\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:40:56.381611 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:40:56.381607 5024 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/78398b98-5396-4f98-bce4-682198b80f4b-istio-data\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:40:56.381611 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:40:56.381617 5024 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/78398b98-5396-4f98-bce4-682198b80f4b-istio-token\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:40:56.381935 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:40:56.381626 5024 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/78398b98-5396-4f98-bce4-682198b80f4b-istiod-ca-cert\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:40:56.381935 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:40:56.381637 5024 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/78398b98-5396-4f98-bce4-682198b80f4b-istio-podinfo\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:40:56.381935 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:40:56.381646 5024 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/78398b98-5396-4f98-bce4-682198b80f4b-istio-envoy\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:40:56.381935 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:40:56.381654 5024 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/78398b98-5396-4f98-bce4-682198b80f4b-workload-socket\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:40:56.381935 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:40:56.381662 5024 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/78398b98-5396-4f98-bce4-682198b80f4b-workload-certs\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:40:56.381935 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:40:56.381671 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4pc72\" (UniqueName: \"kubernetes.io/projected/78398b98-5396-4f98-bce4-682198b80f4b-kube-api-access-4pc72\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:40:56.656500 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:40:56.656461 5024 generic.go:356] "Generic (PLEG): container finished" podID="78398b98-5396-4f98-bce4-682198b80f4b" containerID="fd6a5b9b94d997fd6f479495e6afbe1a3fe56e9781b6deb4d2edb54647748d71" exitCode=0 Nov 03 13:40:56.656706 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:40:56.656535 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-a62d22-openshift-default-d9c57bf65-pvtj7" Nov 03 13:40:56.656706 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:40:56.656543 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-a62d22-openshift-default-d9c57bf65-pvtj7" event={"ID":"78398b98-5396-4f98-bce4-682198b80f4b","Type":"ContainerDied","Data":"fd6a5b9b94d997fd6f479495e6afbe1a3fe56e9781b6deb4d2edb54647748d71"} Nov 03 13:40:56.656706 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:40:56.656630 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-a62d22-openshift-default-d9c57bf65-pvtj7" event={"ID":"78398b98-5396-4f98-bce4-682198b80f4b","Type":"ContainerDied","Data":"4a4383630542f5ab5f3c6bde60d167e5527cf98a6cb75107f7acab11d9e0695d"} Nov 03 13:40:56.656706 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:40:56.656655 5024 scope.go:122] "RemoveContainer" containerID="fd6a5b9b94d997fd6f479495e6afbe1a3fe56e9781b6deb4d2edb54647748d71" Nov 03 13:40:56.668815 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:40:56.668784 5024 scope.go:122] "RemoveContainer" containerID="8be48a1471b5009f52dfa6f88215e1419deebb1457e0e7b59bf17f1cd67a6869" Nov 03 13:40:56.682261 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:40:56.682228 5024 scope.go:122] "RemoveContainer" containerID="fd6a5b9b94d997fd6f479495e6afbe1a3fe56e9781b6deb4d2edb54647748d71" Nov 03 13:40:56.682736 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:40:56.682710 5024 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd6a5b9b94d997fd6f479495e6afbe1a3fe56e9781b6deb4d2edb54647748d71\": container with ID starting with fd6a5b9b94d997fd6f479495e6afbe1a3fe56e9781b6deb4d2edb54647748d71 not found: ID does not exist" containerID="fd6a5b9b94d997fd6f479495e6afbe1a3fe56e9781b6deb4d2edb54647748d71" Nov 03 13:40:56.682856 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:40:56.682747 5024 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd6a5b9b94d997fd6f479495e6afbe1a3fe56e9781b6deb4d2edb54647748d71"} err="failed to get container status \"fd6a5b9b94d997fd6f479495e6afbe1a3fe56e9781b6deb4d2edb54647748d71\": rpc error: code = NotFound desc = could not find container \"fd6a5b9b94d997fd6f479495e6afbe1a3fe56e9781b6deb4d2edb54647748d71\": container with ID starting with fd6a5b9b94d997fd6f479495e6afbe1a3fe56e9781b6deb4d2edb54647748d71 not found: ID does not exist" Nov 03 13:40:56.682856 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:40:56.682768 5024 scope.go:122] "RemoveContainer" containerID="8be48a1471b5009f52dfa6f88215e1419deebb1457e0e7b59bf17f1cd67a6869" Nov 03 13:40:56.683125 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:40:56.683103 5024 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8be48a1471b5009f52dfa6f88215e1419deebb1457e0e7b59bf17f1cd67a6869\": container with ID starting with 8be48a1471b5009f52dfa6f88215e1419deebb1457e0e7b59bf17f1cd67a6869 not found: ID does not exist" containerID="8be48a1471b5009f52dfa6f88215e1419deebb1457e0e7b59bf17f1cd67a6869" Nov 03 13:40:56.683204 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:40:56.683132 5024 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8be48a1471b5009f52dfa6f88215e1419deebb1457e0e7b59bf17f1cd67a6869"} err="failed to get container status \"8be48a1471b5009f52dfa6f88215e1419deebb1457e0e7b59bf17f1cd67a6869\": rpc error: code = NotFound desc = could not find container \"8be48a1471b5009f52dfa6f88215e1419deebb1457e0e7b59bf17f1cd67a6869\": container with ID starting with 8be48a1471b5009f52dfa6f88215e1419deebb1457e0e7b59bf17f1cd67a6869 not found: ID does not exist" Nov 03 13:40:56.689558 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:40:56.689528 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-a62d22-openshift-default-d9c57bf65-pvtj7"] Nov 03 13:40:56.695277 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:40:56.695246 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-a62d22-openshift-default-d9c57bf65-pvtj7"] Nov 03 13:40:58.079677 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:40:58.079628 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-cf53a4-6689d89fbf-9mxzh"] Nov 03 13:40:58.093058 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:40:58.093023 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-cf53a4-7db4cb8dd7-ttdzp"] Nov 03 13:40:58.126328 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:40:58.126266 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="78398b98-5396-4f98-bce4-682198b80f4b" path="/var/lib/kubelet/pods/78398b98-5396-4f98-bce4-682198b80f4b/volumes" Nov 03 13:40:58.513031 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:40:58.512975 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a62d22-69fbf9644c-2svhb" podUID="8a432763-9702-42b0-97e4-c2294b4fe866" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 13:40:58.517662 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:40:58.517574 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-a62d22-6746df564f-wv77r" podUID="327a26e7-752e-4271-a3eb-1100182124a0" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 13:41:03.080697 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:03.080624 5024 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-cf53a4-6689d89fbf-9mxzh" podUID="6b7e669a-c136-4731-bbdc-f877696d4c31" containerName="payload-processing" containerID="cri-o://7110dcebe620eba41e23d2995f204d44fbdff5aa04b771a7d5544946480130c3" gracePeriod=25 Nov 03 13:41:03.094385 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:03.094315 5024 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-cf53a4-7db4cb8dd7-ttdzp" podUID="16ff3f43-3bfd-4dd9-9c9c-32ca8b9fc35e" containerName="payload-pre-processing" containerID="cri-o://e38f2354637a9e8afd8b4aad7308073dc18ae4e0ff7f33be660458f0926519dd" gracePeriod=25 Nov 03 13:41:03.494779 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:03.493121 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-cf53a4-6689d89fbf-9mxzh" Nov 03 13:41:03.617965 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:03.617938 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-cf53a4-7db4cb8dd7-ttdzp" Nov 03 13:41:03.631075 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:03.631039 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16ff3f43-3bfd-4dd9-9c9c-32ca8b9fc35e-kube-api-access-qkq6d\" (UniqueName: \"kubernetes.io/projected/16ff3f43-3bfd-4dd9-9c9c-32ca8b9fc35e-kube-api-access-qkq6d\") pod \"16ff3f43-3bfd-4dd9-9c9c-32ca8b9fc35e\" (UID: \"16ff3f43-3bfd-4dd9-9c9c-32ca8b9fc35e\") " Nov 03 13:41:03.631232 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:03.631086 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b7e669a-c136-4731-bbdc-f877696d4c31-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b7e669a-c136-4731-bbdc-f877696d4c31-plugins-config-volume\") pod \"6b7e669a-c136-4731-bbdc-f877696d4c31\" (UID: \"6b7e669a-c136-4731-bbdc-f877696d4c31\") " Nov 03 13:41:03.631232 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:03.631117 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b7e669a-c136-4731-bbdc-f877696d4c31-kube-api-access-hm4ms\" (UniqueName: \"kubernetes.io/projected/6b7e669a-c136-4731-bbdc-f877696d4c31-kube-api-access-hm4ms\") pod \"6b7e669a-c136-4731-bbdc-f877696d4c31\" (UID: \"6b7e669a-c136-4731-bbdc-f877696d4c31\") " Nov 03 13:41:03.631375 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:03.631249 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/16ff3f43-3bfd-4dd9-9c9c-32ca8b9fc35e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16ff3f43-3bfd-4dd9-9c9c-32ca8b9fc35e-plugins-config-volume\") pod \"16ff3f43-3bfd-4dd9-9c9c-32ca8b9fc35e\" (UID: \"16ff3f43-3bfd-4dd9-9c9c-32ca8b9fc35e\") " Nov 03 13:41:03.631505 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:03.631482 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b7e669a-c136-4731-bbdc-f877696d4c31-plugins-config-volume" pod "6b7e669a-c136-4731-bbdc-f877696d4c31" (UID: "6b7e669a-c136-4731-bbdc-f877696d4c31"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:41:03.631692 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:03.631668 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16ff3f43-3bfd-4dd9-9c9c-32ca8b9fc35e-plugins-config-volume" pod "16ff3f43-3bfd-4dd9-9c9c-32ca8b9fc35e" (UID: "16ff3f43-3bfd-4dd9-9c9c-32ca8b9fc35e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:41:03.631832 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:03.631816 5024 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16ff3f43-3bfd-4dd9-9c9c-32ca8b9fc35e-plugins-config-volume\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:41:03.631907 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:03.631833 5024 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b7e669a-c136-4731-bbdc-f877696d4c31-plugins-config-volume\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:41:03.633242 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:03.633216 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16ff3f43-3bfd-4dd9-9c9c-32ca8b9fc35e-kube-api-access-qkq6d" pod "16ff3f43-3bfd-4dd9-9c9c-32ca8b9fc35e" (UID: "16ff3f43-3bfd-4dd9-9c9c-32ca8b9fc35e"). InnerVolumeSpecName "kube-api-access-qkq6d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:41:03.633548 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:03.633530 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b7e669a-c136-4731-bbdc-f877696d4c31-kube-api-access-hm4ms" pod "6b7e669a-c136-4731-bbdc-f877696d4c31" (UID: "6b7e669a-c136-4731-bbdc-f877696d4c31"). InnerVolumeSpecName "kube-api-access-hm4ms". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:41:03.719490 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:03.719400 5024 generic.go:356] "Generic (PLEG): container finished" podID="6b7e669a-c136-4731-bbdc-f877696d4c31" containerID="7110dcebe620eba41e23d2995f204d44fbdff5aa04b771a7d5544946480130c3" exitCode=0 Nov 03 13:41:03.719490 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:03.719450 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-cf53a4-6689d89fbf-9mxzh" Nov 03 13:41:03.719709 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:03.719490 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-cf53a4-6689d89fbf-9mxzh" event={"ID":"6b7e669a-c136-4731-bbdc-f877696d4c31","Type":"ContainerDied","Data":"7110dcebe620eba41e23d2995f204d44fbdff5aa04b771a7d5544946480130c3"} Nov 03 13:41:03.719709 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:03.719550 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-cf53a4-6689d89fbf-9mxzh" event={"ID":"6b7e669a-c136-4731-bbdc-f877696d4c31","Type":"ContainerDied","Data":"b1b641fa9bc04435f128203a9ca6a111b7dbe7f190964ca3142f898fc9928c04"} Nov 03 13:41:03.719709 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:03.719582 5024 scope.go:122] "RemoveContainer" containerID="7110dcebe620eba41e23d2995f204d44fbdff5aa04b771a7d5544946480130c3" Nov 03 13:41:03.721057 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:03.721034 5024 generic.go:356] "Generic (PLEG): container finished" podID="16ff3f43-3bfd-4dd9-9c9c-32ca8b9fc35e" containerID="e38f2354637a9e8afd8b4aad7308073dc18ae4e0ff7f33be660458f0926519dd" exitCode=0 Nov 03 13:41:03.721182 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:03.721062 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-cf53a4-7db4cb8dd7-ttdzp" event={"ID":"16ff3f43-3bfd-4dd9-9c9c-32ca8b9fc35e","Type":"ContainerDied","Data":"e38f2354637a9e8afd8b4aad7308073dc18ae4e0ff7f33be660458f0926519dd"} Nov 03 13:41:03.721182 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:03.721082 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-cf53a4-7db4cb8dd7-ttdzp" Nov 03 13:41:03.721182 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:03.721109 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-cf53a4-7db4cb8dd7-ttdzp" event={"ID":"16ff3f43-3bfd-4dd9-9c9c-32ca8b9fc35e","Type":"ContainerDied","Data":"977baf1355b338bf06540abcfdfd84bcf69dce320cab268913457718059b3c21"} Nov 03 13:41:03.733338 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:03.733255 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qkq6d\" (UniqueName: \"kubernetes.io/projected/16ff3f43-3bfd-4dd9-9c9c-32ca8b9fc35e-kube-api-access-qkq6d\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:41:03.733338 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:03.733296 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hm4ms\" (UniqueName: \"kubernetes.io/projected/6b7e669a-c136-4731-bbdc-f877696d4c31-kube-api-access-hm4ms\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:41:03.733480 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:03.733421 5024 scope.go:122] "RemoveContainer" containerID="7110dcebe620eba41e23d2995f204d44fbdff5aa04b771a7d5544946480130c3" Nov 03 13:41:03.733787 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:41:03.733753 5024 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7110dcebe620eba41e23d2995f204d44fbdff5aa04b771a7d5544946480130c3\": container with ID starting with 7110dcebe620eba41e23d2995f204d44fbdff5aa04b771a7d5544946480130c3 not found: ID does not exist" containerID="7110dcebe620eba41e23d2995f204d44fbdff5aa04b771a7d5544946480130c3" Nov 03 13:41:03.733870 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:03.733796 5024 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7110dcebe620eba41e23d2995f204d44fbdff5aa04b771a7d5544946480130c3"} err="failed to get container status \"7110dcebe620eba41e23d2995f204d44fbdff5aa04b771a7d5544946480130c3\": rpc error: code = NotFound desc = could not find container \"7110dcebe620eba41e23d2995f204d44fbdff5aa04b771a7d5544946480130c3\": container with ID starting with 7110dcebe620eba41e23d2995f204d44fbdff5aa04b771a7d5544946480130c3 not found: ID does not exist" Nov 03 13:41:03.733870 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:03.733822 5024 scope.go:122] "RemoveContainer" containerID="e38f2354637a9e8afd8b4aad7308073dc18ae4e0ff7f33be660458f0926519dd" Nov 03 13:41:03.743495 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:03.743466 5024 scope.go:122] "RemoveContainer" containerID="e38f2354637a9e8afd8b4aad7308073dc18ae4e0ff7f33be660458f0926519dd" Nov 03 13:41:03.743787 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:41:03.743763 5024 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e38f2354637a9e8afd8b4aad7308073dc18ae4e0ff7f33be660458f0926519dd\": container with ID starting with e38f2354637a9e8afd8b4aad7308073dc18ae4e0ff7f33be660458f0926519dd not found: ID does not exist" containerID="e38f2354637a9e8afd8b4aad7308073dc18ae4e0ff7f33be660458f0926519dd" Nov 03 13:41:03.743871 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:03.743797 5024 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e38f2354637a9e8afd8b4aad7308073dc18ae4e0ff7f33be660458f0926519dd"} err="failed to get container status \"e38f2354637a9e8afd8b4aad7308073dc18ae4e0ff7f33be660458f0926519dd\": rpc error: code = NotFound desc = could not find container \"e38f2354637a9e8afd8b4aad7308073dc18ae4e0ff7f33be660458f0926519dd\": container with ID starting with e38f2354637a9e8afd8b4aad7308073dc18ae4e0ff7f33be660458f0926519dd not found: ID does not exist" Nov 03 13:41:03.753497 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:03.753465 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-cf53a4-7db4cb8dd7-ttdzp"] Nov 03 13:41:03.757722 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:03.757694 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-cf53a4-7db4cb8dd7-ttdzp"] Nov 03 13:41:03.771403 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:03.771362 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-cf53a4-6689d89fbf-9mxzh"] Nov 03 13:41:03.780161 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:03.780126 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-cf53a4-6689d89fbf-9mxzh"] Nov 03 13:41:04.112719 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:04.112678 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="16ff3f43-3bfd-4dd9-9c9c-32ca8b9fc35e" path="/var/lib/kubelet/pods/16ff3f43-3bfd-4dd9-9c9c-32ca8b9fc35e/volumes" Nov 03 13:41:04.113202 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:04.113158 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b7e669a-c136-4731-bbdc-f877696d4c31" path="/var/lib/kubelet/pods/6b7e669a-c136-4731-bbdc-f877696d4c31/volumes" Nov 03 13:41:08.454158 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:08.454118 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-a62d22-6746df564f-wv77r" podUID="327a26e7-752e-4271-a3eb-1100182124a0" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 13:41:08.465037 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:08.464998 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a62d22-69fbf9644c-2svhb" podUID="8a432763-9702-42b0-97e4-c2294b4fe866" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 13:41:09.169182 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:09.168769 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-a62d22-6746df564f-wv77r" Nov 03 13:41:09.263808 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:09.263771 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a62d22-69fbf9644c-2svhb" Nov 03 13:41:09.291174 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:09.289042 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/327a26e7-752e-4271-a3eb-1100182124a0-kube-api-access-mbdgb\" (UniqueName: \"kubernetes.io/projected/327a26e7-752e-4271-a3eb-1100182124a0-kube-api-access-mbdgb\") pod \"327a26e7-752e-4271-a3eb-1100182124a0\" (UID: \"327a26e7-752e-4271-a3eb-1100182124a0\") " Nov 03 13:41:09.291174 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:09.290337 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/327a26e7-752e-4271-a3eb-1100182124a0-plugins-config-volume" pod "327a26e7-752e-4271-a3eb-1100182124a0" (UID: "327a26e7-752e-4271-a3eb-1100182124a0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:41:09.291174 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:09.290392 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/327a26e7-752e-4271-a3eb-1100182124a0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/327a26e7-752e-4271-a3eb-1100182124a0-plugins-config-volume\") pod \"327a26e7-752e-4271-a3eb-1100182124a0\" (UID: \"327a26e7-752e-4271-a3eb-1100182124a0\") " Nov 03 13:41:09.292382 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:09.292334 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/327a26e7-752e-4271-a3eb-1100182124a0-kube-api-access-mbdgb" pod "327a26e7-752e-4271-a3eb-1100182124a0" (UID: "327a26e7-752e-4271-a3eb-1100182124a0"). InnerVolumeSpecName "kube-api-access-mbdgb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:41:09.294329 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:09.294300 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mbdgb\" (UniqueName: \"kubernetes.io/projected/327a26e7-752e-4271-a3eb-1100182124a0-kube-api-access-mbdgb\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:41:09.294445 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:09.294332 5024 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/327a26e7-752e-4271-a3eb-1100182124a0-plugins-config-volume\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:41:09.395479 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:09.395384 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8a432763-9702-42b0-97e4-c2294b4fe866-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a432763-9702-42b0-97e4-c2294b4fe866-plugins-config-volume\") pod \"8a432763-9702-42b0-97e4-c2294b4fe866\" (UID: \"8a432763-9702-42b0-97e4-c2294b4fe866\") " Nov 03 13:41:09.395645 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:09.395501 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a432763-9702-42b0-97e4-c2294b4fe866-kube-api-access-2gkn9\" (UniqueName: \"kubernetes.io/projected/8a432763-9702-42b0-97e4-c2294b4fe866-kube-api-access-2gkn9\") pod \"8a432763-9702-42b0-97e4-c2294b4fe866\" (UID: \"8a432763-9702-42b0-97e4-c2294b4fe866\") " Nov 03 13:41:09.399199 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:09.399157 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a432763-9702-42b0-97e4-c2294b4fe866-kube-api-access-2gkn9" pod "8a432763-9702-42b0-97e4-c2294b4fe866" (UID: "8a432763-9702-42b0-97e4-c2294b4fe866"). InnerVolumeSpecName "kube-api-access-2gkn9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:41:09.399596 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:09.399570 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a432763-9702-42b0-97e4-c2294b4fe866-plugins-config-volume" pod "8a432763-9702-42b0-97e4-c2294b4fe866" (UID: "8a432763-9702-42b0-97e4-c2294b4fe866"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:41:09.517314 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:09.502043 5024 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a432763-9702-42b0-97e4-c2294b4fe866-plugins-config-volume\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:41:09.517314 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:09.502100 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2gkn9\" (UniqueName: \"kubernetes.io/projected/8a432763-9702-42b0-97e4-c2294b4fe866-kube-api-access-2gkn9\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:41:09.595604 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:41:09.595551 5024 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d7mwz" containerName="extractor" Nov 03 13:41:09.934374 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:09.934319 5024 generic.go:356] "Generic (PLEG): container finished" podID="8a432763-9702-42b0-97e4-c2294b4fe866" containerID="39e97a01dd324a8ca31f278514a2a0cac2b1a5985ac9a20d076b011a62c45afe" exitCode=0 Nov 03 13:41:09.934714 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:09.934680 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a62d22-69fbf9644c-2svhb" event={"ID":"8a432763-9702-42b0-97e4-c2294b4fe866","Type":"ContainerDied","Data":"39e97a01dd324a8ca31f278514a2a0cac2b1a5985ac9a20d076b011a62c45afe"} Nov 03 13:41:09.934933 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:09.934914 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a62d22-69fbf9644c-2svhb" event={"ID":"8a432763-9702-42b0-97e4-c2294b4fe866","Type":"ContainerDied","Data":"dd5e7a871b0d6b6501ca4ae7dc581eecb8a0fd5dbea8e967c5da4770f7d197b1"} Nov 03 13:41:09.935063 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:09.935050 5024 scope.go:122] "RemoveContainer" containerID="39e97a01dd324a8ca31f278514a2a0cac2b1a5985ac9a20d076b011a62c45afe" Nov 03 13:41:09.935876 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:09.935854 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a62d22-69fbf9644c-2svhb" Nov 03 13:41:09.938103 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:09.938073 5024 generic.go:356] "Generic (PLEG): container finished" podID="327a26e7-752e-4271-a3eb-1100182124a0" containerID="6b3106d0694a65dc70dcf23058c45ceff9ec6af61a7d112e09571410cfc2e036" exitCode=0 Nov 03 13:41:09.938276 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:09.938258 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-a62d22-6746df564f-wv77r" event={"ID":"327a26e7-752e-4271-a3eb-1100182124a0","Type":"ContainerDied","Data":"6b3106d0694a65dc70dcf23058c45ceff9ec6af61a7d112e09571410cfc2e036"} Nov 03 13:41:09.938393 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:09.938371 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-a62d22-6746df564f-wv77r" event={"ID":"327a26e7-752e-4271-a3eb-1100182124a0","Type":"ContainerDied","Data":"1f06529a64f7ea2f1850c3e5c9c4fdd24f17abfe3985bd263711c8f8ccd71ab2"} Nov 03 13:41:09.938513 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:09.938453 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-a62d22-6746df564f-wv77r" Nov 03 13:41:09.950953 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:09.950929 5024 scope.go:122] "RemoveContainer" containerID="39e97a01dd324a8ca31f278514a2a0cac2b1a5985ac9a20d076b011a62c45afe" Nov 03 13:41:09.951439 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:41:09.951406 5024 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"39e97a01dd324a8ca31f278514a2a0cac2b1a5985ac9a20d076b011a62c45afe\": container with ID starting with 39e97a01dd324a8ca31f278514a2a0cac2b1a5985ac9a20d076b011a62c45afe not found: ID does not exist" containerID="39e97a01dd324a8ca31f278514a2a0cac2b1a5985ac9a20d076b011a62c45afe" Nov 03 13:41:09.951546 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:09.951447 5024 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39e97a01dd324a8ca31f278514a2a0cac2b1a5985ac9a20d076b011a62c45afe"} err="failed to get container status \"39e97a01dd324a8ca31f278514a2a0cac2b1a5985ac9a20d076b011a62c45afe\": rpc error: code = NotFound desc = could not find container \"39e97a01dd324a8ca31f278514a2a0cac2b1a5985ac9a20d076b011a62c45afe\": container with ID starting with 39e97a01dd324a8ca31f278514a2a0cac2b1a5985ac9a20d076b011a62c45afe not found: ID does not exist" Nov 03 13:41:09.951546 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:09.951472 5024 scope.go:122] "RemoveContainer" containerID="6b3106d0694a65dc70dcf23058c45ceff9ec6af61a7d112e09571410cfc2e036" Nov 03 13:41:09.962083 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:09.962054 5024 scope.go:122] "RemoveContainer" containerID="6b3106d0694a65dc70dcf23058c45ceff9ec6af61a7d112e09571410cfc2e036" Nov 03 13:41:09.962407 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:41:09.962383 5024 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b3106d0694a65dc70dcf23058c45ceff9ec6af61a7d112e09571410cfc2e036\": container with ID starting with 6b3106d0694a65dc70dcf23058c45ceff9ec6af61a7d112e09571410cfc2e036 not found: ID does not exist" containerID="6b3106d0694a65dc70dcf23058c45ceff9ec6af61a7d112e09571410cfc2e036" Nov 03 13:41:09.962493 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:09.962418 5024 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b3106d0694a65dc70dcf23058c45ceff9ec6af61a7d112e09571410cfc2e036"} err="failed to get container status \"6b3106d0694a65dc70dcf23058c45ceff9ec6af61a7d112e09571410cfc2e036\": rpc error: code = NotFound desc = could not find container \"6b3106d0694a65dc70dcf23058c45ceff9ec6af61a7d112e09571410cfc2e036\": container with ID starting with 6b3106d0694a65dc70dcf23058c45ceff9ec6af61a7d112e09571410cfc2e036 not found: ID does not exist" Nov 03 13:41:09.969133 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:09.969106 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-a62d22-69fbf9644c-2svhb"] Nov 03 13:41:09.976216 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:09.976186 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-a62d22-69fbf9644c-2svhb"] Nov 03 13:41:09.991131 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:09.991098 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-a62d22-6746df564f-wv77r"] Nov 03 13:41:09.996255 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:09.996226 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-a62d22-6746df564f-wv77r"] Nov 03 13:41:10.110808 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:10.110719 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="327a26e7-752e-4271-a3eb-1100182124a0" path="/var/lib/kubelet/pods/327a26e7-752e-4271-a3eb-1100182124a0/volumes" Nov 03 13:41:10.111357 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:10.111336 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a432763-9702-42b0-97e4-c2294b4fe866" path="/var/lib/kubelet/pods/8a432763-9702-42b0-97e4-c2294b4fe866/volumes" Nov 03 13:41:12.180201 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:12.180163 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-d3ca48/premium-w3-d3ca48-kserve-75ffc959b-zvjbc"] Nov 03 13:41:19.400397 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:19.400353 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-7h97f"] Nov 03 13:41:19.402093 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:19.402063 5024 cpu_manager.go:414] "containerMap: removing container" podUID="b61db32a-7a3b-4aff-b331-f7888a1a4e0e" containerName="istio-proxy" Nov 03 13:41:19.402271 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:19.402254 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b61db32a-7a3b-4aff-b331-f7888a1a4e0e" containerName="istio-proxy" Nov 03 13:41:19.402439 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:19.402423 5024 cpu_manager.go:414] "containerMap: removing container" podUID="e6d115b8-3b33-43c6-890b-a87bb608444b" containerName="istio-proxy" Nov 03 13:41:19.402541 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:19.402528 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e6d115b8-3b33-43c6-890b-a87bb608444b" containerName="istio-proxy" Nov 03 13:41:19.402639 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:19.402628 5024 cpu_manager.go:414] "containerMap: removing container" podUID="8a432763-9702-42b0-97e4-c2294b4fe866" containerName="payload-pre-processing" Nov 03 13:41:19.402723 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:19.402711 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a432763-9702-42b0-97e4-c2294b4fe866" containerName="payload-pre-processing" Nov 03 13:41:19.402812 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:19.402801 5024 cpu_manager.go:414] "containerMap: removing container" podUID="c27e059d-8901-486f-9e7f-60fc2c2e7416" containerName="main" Nov 03 13:41:19.402886 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:19.402875 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c27e059d-8901-486f-9e7f-60fc2c2e7416" containerName="main" Nov 03 13:41:19.414309 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:19.410712 5024 cpu_manager.go:414] "containerMap: removing container" podUID="16ff3f43-3bfd-4dd9-9c9c-32ca8b9fc35e" containerName="payload-pre-processing" Nov 03 13:41:19.414309 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:19.410752 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16ff3f43-3bfd-4dd9-9c9c-32ca8b9fc35e" containerName="payload-pre-processing" Nov 03 13:41:19.414309 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:19.410803 5024 cpu_manager.go:414] "containerMap: removing container" podUID="327a26e7-752e-4271-a3eb-1100182124a0" containerName="payload-processing" Nov 03 13:41:19.414309 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:19.410811 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="327a26e7-752e-4271-a3eb-1100182124a0" containerName="payload-processing" Nov 03 13:41:19.414309 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:19.410823 5024 cpu_manager.go:414] "containerMap: removing container" podUID="6b7e669a-c136-4731-bbdc-f877696d4c31" containerName="payload-processing" Nov 03 13:41:19.414309 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:19.410831 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b7e669a-c136-4731-bbdc-f877696d4c31" containerName="payload-processing" Nov 03 13:41:19.414309 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:19.410852 5024 cpu_manager.go:414] "containerMap: removing container" podUID="17976293-ef64-4977-a6a2-d872ac7b9b4a" containerName="payload-processing" Nov 03 13:41:19.414309 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:19.410859 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17976293-ef64-4977-a6a2-d872ac7b9b4a" containerName="payload-processing" Nov 03 13:41:19.414309 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:19.410869 5024 cpu_manager.go:414] "containerMap: removing container" podUID="a5bccdb6-fde7-475e-bb9e-a256e83a83d1" containerName="maas-api" Nov 03 13:41:19.414309 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:19.410878 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5bccdb6-fde7-475e-bb9e-a256e83a83d1" containerName="maas-api" Nov 03 13:41:19.414309 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:19.410899 5024 cpu_manager.go:414] "containerMap: removing container" podUID="78398b98-5396-4f98-bce4-682198b80f4b" containerName="istio-proxy" Nov 03 13:41:19.414309 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:19.410906 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="78398b98-5396-4f98-bce4-682198b80f4b" containerName="istio-proxy" Nov 03 13:41:19.414309 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:19.410918 5024 cpu_manager.go:414] "containerMap: removing container" podUID="78398b98-5396-4f98-bce4-682198b80f4b" containerName="istio-proxy" Nov 03 13:41:19.414309 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:19.410925 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="78398b98-5396-4f98-bce4-682198b80f4b" containerName="istio-proxy" Nov 03 13:41:19.414309 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:19.411144 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="17976293-ef64-4977-a6a2-d872ac7b9b4a" containerName="payload-processing" Nov 03 13:41:19.414309 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:19.411158 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5bccdb6-fde7-475e-bb9e-a256e83a83d1" containerName="maas-api" Nov 03 13:41:19.414309 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:19.411172 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="78398b98-5396-4f98-bce4-682198b80f4b" containerName="istio-proxy" Nov 03 13:41:19.414309 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:19.411184 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="78398b98-5396-4f98-bce4-682198b80f4b" containerName="istio-proxy" Nov 03 13:41:19.414309 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:19.411200 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="b61db32a-7a3b-4aff-b331-f7888a1a4e0e" containerName="istio-proxy" Nov 03 13:41:19.414309 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:19.411212 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="e6d115b8-3b33-43c6-890b-a87bb608444b" containerName="istio-proxy" Nov 03 13:41:19.414309 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:19.411225 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a432763-9702-42b0-97e4-c2294b4fe866" containerName="payload-pre-processing" Nov 03 13:41:19.414309 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:19.411238 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="c27e059d-8901-486f-9e7f-60fc2c2e7416" containerName="main" Nov 03 13:41:19.414309 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:19.411249 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="16ff3f43-3bfd-4dd9-9c9c-32ca8b9fc35e" containerName="payload-pre-processing" Nov 03 13:41:19.414309 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:19.411261 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="327a26e7-752e-4271-a3eb-1100182124a0" containerName="payload-processing" Nov 03 13:41:19.414309 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:19.411270 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b7e669a-c136-4731-bbdc-f877696d4c31" containerName="payload-processing" Nov 03 13:41:19.414309 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:19.411304 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0bde2d4-7fae-41d0-960a-20a0e664d597" containerName="istio-proxy" Nov 03 13:41:19.421156 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:19.417372 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-7h97f" Nov 03 13:41:19.452318 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:19.435342 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-7h97f"] Nov 03 13:41:19.577013 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:19.514867 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/823a705e-26f9-422c-83b0-75dc73349072-istio-podinfo\") pod \"e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-7h97f\" (UID: \"823a705e-26f9-422c-83b0-75dc73349072\") " pod="openshift-ingress/e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-7h97f" Nov 03 13:41:19.577013 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:19.555085 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/823a705e-26f9-422c-83b0-75dc73349072-istio-token\") pod \"e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-7h97f\" (UID: \"823a705e-26f9-422c-83b0-75dc73349072\") " pod="openshift-ingress/e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-7h97f" Nov 03 13:41:19.577013 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:19.555142 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/823a705e-26f9-422c-83b0-75dc73349072-credential-socket\") pod \"e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-7h97f\" (UID: \"823a705e-26f9-422c-83b0-75dc73349072\") " pod="openshift-ingress/e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-7h97f" Nov 03 13:41:19.577013 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:19.555173 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/823a705e-26f9-422c-83b0-75dc73349072-istio-data\") pod \"e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-7h97f\" (UID: \"823a705e-26f9-422c-83b0-75dc73349072\") " pod="openshift-ingress/e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-7h97f" Nov 03 13:41:19.577013 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:19.555227 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/823a705e-26f9-422c-83b0-75dc73349072-workload-certs\") pod \"e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-7h97f\" (UID: \"823a705e-26f9-422c-83b0-75dc73349072\") " pod="openshift-ingress/e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-7h97f" Nov 03 13:41:19.577013 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:19.555255 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7s97c\" (UniqueName: \"kubernetes.io/projected/823a705e-26f9-422c-83b0-75dc73349072-kube-api-access-7s97c\") pod \"e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-7h97f\" (UID: \"823a705e-26f9-422c-83b0-75dc73349072\") " pod="openshift-ingress/e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-7h97f" Nov 03 13:41:19.577013 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:19.555298 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/823a705e-26f9-422c-83b0-75dc73349072-istiod-ca-cert\") pod \"e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-7h97f\" (UID: \"823a705e-26f9-422c-83b0-75dc73349072\") " pod="openshift-ingress/e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-7h97f" Nov 03 13:41:19.577013 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:19.559827 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/823a705e-26f9-422c-83b0-75dc73349072-workload-socket\") pod \"e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-7h97f\" (UID: \"823a705e-26f9-422c-83b0-75dc73349072\") " pod="openshift-ingress/e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-7h97f" Nov 03 13:41:19.577013 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:19.560027 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/823a705e-26f9-422c-83b0-75dc73349072-istio-envoy\") pod \"e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-7h97f\" (UID: \"823a705e-26f9-422c-83b0-75dc73349072\") " pod="openshift-ingress/e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-7h97f" Nov 03 13:41:19.661732 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:19.661634 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/823a705e-26f9-422c-83b0-75dc73349072-istio-envoy\") pod \"e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-7h97f\" (UID: \"823a705e-26f9-422c-83b0-75dc73349072\") " pod="openshift-ingress/e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-7h97f" Nov 03 13:41:19.661912 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:19.661793 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/823a705e-26f9-422c-83b0-75dc73349072-istio-podinfo\") pod \"e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-7h97f\" (UID: \"823a705e-26f9-422c-83b0-75dc73349072\") " pod="openshift-ingress/e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-7h97f" Nov 03 13:41:19.661985 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:19.661920 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/823a705e-26f9-422c-83b0-75dc73349072-istio-token\") pod \"e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-7h97f\" (UID: \"823a705e-26f9-422c-83b0-75dc73349072\") " pod="openshift-ingress/e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-7h97f" Nov 03 13:41:19.661985 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:19.661964 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/823a705e-26f9-422c-83b0-75dc73349072-credential-socket\") pod \"e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-7h97f\" (UID: \"823a705e-26f9-422c-83b0-75dc73349072\") " pod="openshift-ingress/e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-7h97f" Nov 03 13:41:19.662100 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:19.661993 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/823a705e-26f9-422c-83b0-75dc73349072-istio-data\") pod \"e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-7h97f\" (UID: \"823a705e-26f9-422c-83b0-75dc73349072\") " pod="openshift-ingress/e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-7h97f" Nov 03 13:41:19.662100 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:19.662045 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/823a705e-26f9-422c-83b0-75dc73349072-workload-certs\") pod \"e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-7h97f\" (UID: \"823a705e-26f9-422c-83b0-75dc73349072\") " pod="openshift-ingress/e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-7h97f" Nov 03 13:41:19.662100 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:19.662074 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7s97c\" (UniqueName: \"kubernetes.io/projected/823a705e-26f9-422c-83b0-75dc73349072-kube-api-access-7s97c\") pod \"e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-7h97f\" (UID: \"823a705e-26f9-422c-83b0-75dc73349072\") " pod="openshift-ingress/e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-7h97f" Nov 03 13:41:19.662249 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:19.662107 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/823a705e-26f9-422c-83b0-75dc73349072-istiod-ca-cert\") pod \"e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-7h97f\" (UID: \"823a705e-26f9-422c-83b0-75dc73349072\") " pod="openshift-ingress/e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-7h97f" Nov 03 13:41:19.662249 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:19.662165 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/823a705e-26f9-422c-83b0-75dc73349072-workload-socket\") pod \"e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-7h97f\" (UID: \"823a705e-26f9-422c-83b0-75dc73349072\") " pod="openshift-ingress/e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-7h97f" Nov 03 13:41:19.662424 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:19.662388 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/823a705e-26f9-422c-83b0-75dc73349072-credential-socket\") pod \"e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-7h97f\" (UID: \"823a705e-26f9-422c-83b0-75dc73349072\") " pod="openshift-ingress/e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-7h97f" Nov 03 13:41:19.662502 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:19.662426 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/823a705e-26f9-422c-83b0-75dc73349072-workload-certs\") pod \"e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-7h97f\" (UID: \"823a705e-26f9-422c-83b0-75dc73349072\") " pod="openshift-ingress/e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-7h97f" Nov 03 13:41:19.662565 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:19.662539 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/823a705e-26f9-422c-83b0-75dc73349072-workload-socket\") pod \"e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-7h97f\" (UID: \"823a705e-26f9-422c-83b0-75dc73349072\") " pod="openshift-ingress/e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-7h97f" Nov 03 13:41:19.662750 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:19.662724 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/823a705e-26f9-422c-83b0-75dc73349072-istio-data\") pod \"e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-7h97f\" (UID: \"823a705e-26f9-422c-83b0-75dc73349072\") " pod="openshift-ingress/e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-7h97f" Nov 03 13:41:19.663133 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:19.663105 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/823a705e-26f9-422c-83b0-75dc73349072-istiod-ca-cert\") pod \"e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-7h97f\" (UID: \"823a705e-26f9-422c-83b0-75dc73349072\") " pod="openshift-ingress/e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-7h97f" Nov 03 13:41:19.664923 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:19.664897 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/823a705e-26f9-422c-83b0-75dc73349072-istio-envoy\") pod \"e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-7h97f\" (UID: \"823a705e-26f9-422c-83b0-75dc73349072\") " pod="openshift-ingress/e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-7h97f" Nov 03 13:41:19.664923 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:19.664906 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/823a705e-26f9-422c-83b0-75dc73349072-istio-podinfo\") pod \"e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-7h97f\" (UID: \"823a705e-26f9-422c-83b0-75dc73349072\") " pod="openshift-ingress/e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-7h97f" Nov 03 13:41:19.671871 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:19.671833 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7s97c\" (UniqueName: \"kubernetes.io/projected/823a705e-26f9-422c-83b0-75dc73349072-kube-api-access-7s97c\") pod \"e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-7h97f\" (UID: \"823a705e-26f9-422c-83b0-75dc73349072\") " pod="openshift-ingress/e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-7h97f" Nov 03 13:41:19.674647 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:19.674611 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/823a705e-26f9-422c-83b0-75dc73349072-istio-token\") pod \"e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-7h97f\" (UID: \"823a705e-26f9-422c-83b0-75dc73349072\") " pod="openshift-ingress/e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-7h97f" Nov 03 13:41:19.930847 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:19.930736 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-7h97f" Nov 03 13:41:20.234155 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:41:20.234121 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod823a705e_26f9_422c_83b0_75dc73349072.slice/crio-c98955c7bc5e38bb4596a68e0610846b2ddd8a94a7eefc4415d5c700886e9a1f WatchSource:0}: Error finding container c98955c7bc5e38bb4596a68e0610846b2ddd8a94a7eefc4415d5c700886e9a1f: Status 404 returned error can't find the container with id c98955c7bc5e38bb4596a68e0610846b2ddd8a94a7eefc4415d5c700886e9a1f Nov 03 13:41:20.234278 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:20.234240 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-7h97f"] Nov 03 13:41:20.236910 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:20.236871 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:41:20.237030 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:20.236959 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:41:20.237030 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:20.237008 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:41:21.062971 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:21.062936 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-7h97f" event={"ID":"823a705e-26f9-422c-83b0-75dc73349072","Type":"ContainerStarted","Data":"58dc34962e683ed7b74140e17f6067eba9b1720fb11f3d51cfc0d746e336670d"} Nov 03 13:41:21.063440 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:21.062986 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-7h97f" event={"ID":"823a705e-26f9-422c-83b0-75dc73349072","Type":"ContainerStarted","Data":"c98955c7bc5e38bb4596a68e0610846b2ddd8a94a7eefc4415d5c700886e9a1f"} Nov 03 13:41:21.112850 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:21.112786 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-7h97f" podStartSLOduration=2.112770609 podStartE2EDuration="2.112770609s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:41:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:41:21.109015198 +0000 UTC m=+1961.117206476" watchObservedRunningTime="2026-11-03 13:41:21.112770609 +0000 UTC m=+1961.120961885" Nov 03 13:41:21.933536 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:21.933495 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-7h97f" Nov 03 13:41:21.936038 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:21.935998 5024 patch_prober.go:29] interesting pod/e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-7h97f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" start-of-body= Nov 03 13:41:21.936206 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:21.936060 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-7h97f" podUID="823a705e-26f9-422c-83b0-75dc73349072" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" Nov 03 13:41:22.933465 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:22.933422 5024 patch_prober.go:29] interesting pod/e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-7h97f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" start-of-body= Nov 03 13:41:22.933961 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:22.933483 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-7h97f" podUID="823a705e-26f9-422c-83b0-75dc73349072" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" Nov 03 13:41:23.938309 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:23.935421 5024 patch_prober.go:29] interesting pod/e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-7h97f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" start-of-body= Nov 03 13:41:23.938309 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:23.935486 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-7h97f" podUID="823a705e-26f9-422c-83b0-75dc73349072" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" Nov 03 13:41:24.936318 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:24.935705 5024 patch_prober.go:29] interesting pod/e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-7h97f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" start-of-body= Nov 03 13:41:24.936318 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:24.935774 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-7h97f" podUID="823a705e-26f9-422c-83b0-75dc73349072" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" Nov 03 13:41:25.934301 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:25.933257 5024 patch_prober.go:29] interesting pod/e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-7h97f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" start-of-body= Nov 03 13:41:25.934301 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:25.933345 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-7h97f" podUID="823a705e-26f9-422c-83b0-75dc73349072" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" Nov 03 13:41:26.932861 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:26.932789 5024 patch_prober.go:29] interesting pod/e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-7h97f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" start-of-body= Nov 03 13:41:26.933030 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:26.932890 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-7h97f" podUID="823a705e-26f9-422c-83b0-75dc73349072" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" Nov 03 13:41:27.205060 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:27.204956 5024 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-d3ca48/premium-w3-d3ca48-kserve-75ffc959b-zvjbc" podUID="add1e8b9-e7fd-4e4d-8d10-f97d0e2612ba" containerName="main" containerID="cri-o://b4a70cd448c0750951bc94ce807bda6662f9f5085e553ef4d4b687941d8f8fb7" gracePeriod=45 Nov 03 13:41:27.276535 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:27.276485 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d3ca48/premium-w3-d3ca48-kserve-75ffc959b-zvjbc" podUID="add1e8b9-e7fd-4e4d-8d10-f97d0e2612ba" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": dial tcp 10.132.0.82:8000: connect: connection refused" Nov 03 13:41:27.622352 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:27.622313 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-d3ca48/premium-w3-d3ca48-kserve-75ffc959b-zvjbc" Nov 03 13:41:27.642305 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:27.640447 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/add1e8b9-e7fd-4e4d-8d10-f97d0e2612ba-home\" (UniqueName: \"kubernetes.io/empty-dir/add1e8b9-e7fd-4e4d-8d10-f97d0e2612ba-home\") pod \"add1e8b9-e7fd-4e4d-8d10-f97d0e2612ba\" (UID: \"add1e8b9-e7fd-4e4d-8d10-f97d0e2612ba\") " Nov 03 13:41:27.642305 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:27.640536 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/add1e8b9-e7fd-4e4d-8d10-f97d0e2612ba-tls-certs\" (UniqueName: \"kubernetes.io/secret/add1e8b9-e7fd-4e4d-8d10-f97d0e2612ba-tls-certs\") pod \"add1e8b9-e7fd-4e4d-8d10-f97d0e2612ba\" (UID: \"add1e8b9-e7fd-4e4d-8d10-f97d0e2612ba\") " Nov 03 13:41:27.642305 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:27.640620 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/add1e8b9-e7fd-4e4d-8d10-f97d0e2612ba-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/add1e8b9-e7fd-4e4d-8d10-f97d0e2612ba-model-cache\") pod \"add1e8b9-e7fd-4e4d-8d10-f97d0e2612ba\" (UID: \"add1e8b9-e7fd-4e4d-8d10-f97d0e2612ba\") " Nov 03 13:41:27.642305 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:27.640702 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/add1e8b9-e7fd-4e4d-8d10-f97d0e2612ba-kube-api-access-4v9bn\" (UniqueName: \"kubernetes.io/projected/add1e8b9-e7fd-4e4d-8d10-f97d0e2612ba-kube-api-access-4v9bn\") pod \"add1e8b9-e7fd-4e4d-8d10-f97d0e2612ba\" (UID: \"add1e8b9-e7fd-4e4d-8d10-f97d0e2612ba\") " Nov 03 13:41:27.642305 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:27.640753 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/add1e8b9-e7fd-4e4d-8d10-f97d0e2612ba-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/add1e8b9-e7fd-4e4d-8d10-f97d0e2612ba-tmp-dir\") pod \"add1e8b9-e7fd-4e4d-8d10-f97d0e2612ba\" (UID: \"add1e8b9-e7fd-4e4d-8d10-f97d0e2612ba\") " Nov 03 13:41:27.642305 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:27.640829 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/add1e8b9-e7fd-4e4d-8d10-f97d0e2612ba-dshm\" (UniqueName: \"kubernetes.io/empty-dir/add1e8b9-e7fd-4e4d-8d10-f97d0e2612ba-dshm\") pod \"add1e8b9-e7fd-4e4d-8d10-f97d0e2612ba\" (UID: \"add1e8b9-e7fd-4e4d-8d10-f97d0e2612ba\") " Nov 03 13:41:27.645448 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:27.644628 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/add1e8b9-e7fd-4e4d-8d10-f97d0e2612ba-model-cache" pod "add1e8b9-e7fd-4e4d-8d10-f97d0e2612ba" (UID: "add1e8b9-e7fd-4e4d-8d10-f97d0e2612ba"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:41:27.645448 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:27.644720 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/add1e8b9-e7fd-4e4d-8d10-f97d0e2612ba-home" pod "add1e8b9-e7fd-4e4d-8d10-f97d0e2612ba" (UID: "add1e8b9-e7fd-4e4d-8d10-f97d0e2612ba"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:41:27.645448 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:27.644790 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/add1e8b9-e7fd-4e4d-8d10-f97d0e2612ba-tmp-dir" pod "add1e8b9-e7fd-4e4d-8d10-f97d0e2612ba" (UID: "add1e8b9-e7fd-4e4d-8d10-f97d0e2612ba"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:41:27.647140 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:27.647107 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/add1e8b9-e7fd-4e4d-8d10-f97d0e2612ba-tls-certs" pod "add1e8b9-e7fd-4e4d-8d10-f97d0e2612ba" (UID: "add1e8b9-e7fd-4e4d-8d10-f97d0e2612ba"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 13:41:27.667994 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:27.667946 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/add1e8b9-e7fd-4e4d-8d10-f97d0e2612ba-kube-api-access-4v9bn" pod "add1e8b9-e7fd-4e4d-8d10-f97d0e2612ba" (UID: "add1e8b9-e7fd-4e4d-8d10-f97d0e2612ba"). InnerVolumeSpecName "kube-api-access-4v9bn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:41:27.687501 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:27.687436 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/add1e8b9-e7fd-4e4d-8d10-f97d0e2612ba-dshm" pod "add1e8b9-e7fd-4e4d-8d10-f97d0e2612ba" (UID: "add1e8b9-e7fd-4e4d-8d10-f97d0e2612ba"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:41:27.743575 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:27.743530 5024 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/add1e8b9-e7fd-4e4d-8d10-f97d0e2612ba-home\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:41:27.743725 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:27.743583 5024 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/add1e8b9-e7fd-4e4d-8d10-f97d0e2612ba-tls-certs\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:41:27.743725 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:27.743600 5024 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/add1e8b9-e7fd-4e4d-8d10-f97d0e2612ba-model-cache\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:41:27.743725 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:27.743619 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4v9bn\" (UniqueName: \"kubernetes.io/projected/add1e8b9-e7fd-4e4d-8d10-f97d0e2612ba-kube-api-access-4v9bn\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:41:27.743725 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:27.743636 5024 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/add1e8b9-e7fd-4e4d-8d10-f97d0e2612ba-tmp-dir\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:41:27.743725 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:27.743651 5024 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/add1e8b9-e7fd-4e4d-8d10-f97d0e2612ba-dshm\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:41:27.938065 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:27.938028 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-7h97f" Nov 03 13:41:27.938570 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:27.938539 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-7h97f" Nov 03 13:41:27.939258 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:27.939238 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-7h97f" Nov 03 13:41:28.302641 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:28.302548 5024 generic.go:356] "Generic (PLEG): container finished" podID="add1e8b9-e7fd-4e4d-8d10-f97d0e2612ba" containerID="b4a70cd448c0750951bc94ce807bda6662f9f5085e553ef4d4b687941d8f8fb7" exitCode=0 Nov 03 13:41:28.302641 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:28.302595 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-d3ca48/premium-w3-d3ca48-kserve-75ffc959b-zvjbc" Nov 03 13:41:28.303174 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:28.302793 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-d3ca48/premium-w3-d3ca48-kserve-75ffc959b-zvjbc" event={"ID":"add1e8b9-e7fd-4e4d-8d10-f97d0e2612ba","Type":"ContainerDied","Data":"b4a70cd448c0750951bc94ce807bda6662f9f5085e553ef4d4b687941d8f8fb7"} Nov 03 13:41:28.303174 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:28.302840 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-d3ca48/premium-w3-d3ca48-kserve-75ffc959b-zvjbc" event={"ID":"add1e8b9-e7fd-4e4d-8d10-f97d0e2612ba","Type":"ContainerDied","Data":"13487f6b814c734a9bb764f83259ef7f6816e8cfa6e7aa3b2a940e3922c94450"} Nov 03 13:41:28.303174 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:28.302865 5024 scope.go:122] "RemoveContainer" containerID="b4a70cd448c0750951bc94ce807bda6662f9f5085e553ef4d4b687941d8f8fb7" Nov 03 13:41:28.314103 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:28.314078 5024 scope.go:122] "RemoveContainer" containerID="b4a70cd448c0750951bc94ce807bda6662f9f5085e553ef4d4b687941d8f8fb7" Nov 03 13:41:28.314496 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:41:28.314459 5024 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b4a70cd448c0750951bc94ce807bda6662f9f5085e553ef4d4b687941d8f8fb7\": container with ID starting with b4a70cd448c0750951bc94ce807bda6662f9f5085e553ef4d4b687941d8f8fb7 not found: ID does not exist" containerID="b4a70cd448c0750951bc94ce807bda6662f9f5085e553ef4d4b687941d8f8fb7" Nov 03 13:41:28.314612 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:28.314507 5024 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4a70cd448c0750951bc94ce807bda6662f9f5085e553ef4d4b687941d8f8fb7"} err="failed to get container status \"b4a70cd448c0750951bc94ce807bda6662f9f5085e553ef4d4b687941d8f8fb7\": rpc error: code = NotFound desc = could not find container \"b4a70cd448c0750951bc94ce807bda6662f9f5085e553ef4d4b687941d8f8fb7\": container with ID starting with b4a70cd448c0750951bc94ce807bda6662f9f5085e553ef4d4b687941d8f8fb7 not found: ID does not exist" Nov 03 13:41:28.324646 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:28.324616 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-d3ca48/premium-w3-d3ca48-kserve-75ffc959b-zvjbc"] Nov 03 13:41:28.327916 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:28.327891 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-d3ca48/premium-w3-d3ca48-kserve-75ffc959b-zvjbc"] Nov 03 13:41:30.116502 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:30.116461 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="add1e8b9-e7fd-4e4d-8d10-f97d0e2612ba" path="/var/lib/kubelet/pods/add1e8b9-e7fd-4e4d-8d10-f97d0e2612ba/volumes" Nov 03 13:41:37.204475 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:37.204295 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-a724a6/sub-test-model-a724a6-kserve-57d7c88d9f-5nk9m"] Nov 03 13:41:38.459910 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:38.459870 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-9csnb"] Nov 03 13:41:38.460800 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:38.460776 5024 cpu_manager.go:414] "containerMap: removing container" podUID="add1e8b9-e7fd-4e4d-8d10-f97d0e2612ba" containerName="main" Nov 03 13:41:38.460918 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:38.460800 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="add1e8b9-e7fd-4e4d-8d10-f97d0e2612ba" containerName="main" Nov 03 13:41:38.461024 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:38.461008 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="add1e8b9-e7fd-4e4d-8d10-f97d0e2612ba" containerName="main" Nov 03 13:41:38.466692 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:38.466666 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-9csnb" Nov 03 13:41:38.491220 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:38.488544 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-9csnb"] Nov 03 13:41:38.548904 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:38.548859 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94ab7b96-9dfc-4065-b1eb-802f97e92abe-credential-socket\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-9csnb\" (UID: \"94ab7b96-9dfc-4065-b1eb-802f97e92abe\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-9csnb" Nov 03 13:41:38.549087 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:38.548979 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94ab7b96-9dfc-4065-b1eb-802f97e92abe-workload-certs\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-9csnb\" (UID: \"94ab7b96-9dfc-4065-b1eb-802f97e92abe\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-9csnb" Nov 03 13:41:38.549134 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:38.549086 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94ab7b96-9dfc-4065-b1eb-802f97e92abe-workload-socket\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-9csnb\" (UID: \"94ab7b96-9dfc-4065-b1eb-802f97e92abe\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-9csnb" Nov 03 13:41:38.549134 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:38.549108 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94ab7b96-9dfc-4065-b1eb-802f97e92abe-istiod-ca-cert\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-9csnb\" (UID: \"94ab7b96-9dfc-4065-b1eb-802f97e92abe\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-9csnb" Nov 03 13:41:38.549223 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:38.549192 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94ab7b96-9dfc-4065-b1eb-802f97e92abe-istio-podinfo\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-9csnb\" (UID: \"94ab7b96-9dfc-4065-b1eb-802f97e92abe\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-9csnb" Nov 03 13:41:38.549275 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:38.549225 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fqlrk\" (UniqueName: \"kubernetes.io/projected/94ab7b96-9dfc-4065-b1eb-802f97e92abe-kube-api-access-fqlrk\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-9csnb\" (UID: \"94ab7b96-9dfc-4065-b1eb-802f97e92abe\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-9csnb" Nov 03 13:41:38.549378 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:38.549359 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94ab7b96-9dfc-4065-b1eb-802f97e92abe-istio-token\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-9csnb\" (UID: \"94ab7b96-9dfc-4065-b1eb-802f97e92abe\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-9csnb" Nov 03 13:41:38.549441 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:38.549386 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94ab7b96-9dfc-4065-b1eb-802f97e92abe-istio-data\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-9csnb\" (UID: \"94ab7b96-9dfc-4065-b1eb-802f97e92abe\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-9csnb" Nov 03 13:41:38.549732 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:38.549466 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94ab7b96-9dfc-4065-b1eb-802f97e92abe-istio-envoy\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-9csnb\" (UID: \"94ab7b96-9dfc-4065-b1eb-802f97e92abe\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-9csnb" Nov 03 13:41:38.650760 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:38.650715 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94ab7b96-9dfc-4065-b1eb-802f97e92abe-workload-socket\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-9csnb\" (UID: \"94ab7b96-9dfc-4065-b1eb-802f97e92abe\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-9csnb" Nov 03 13:41:38.650760 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:38.650763 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94ab7b96-9dfc-4065-b1eb-802f97e92abe-istiod-ca-cert\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-9csnb\" (UID: \"94ab7b96-9dfc-4065-b1eb-802f97e92abe\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-9csnb" Nov 03 13:41:38.651017 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:38.650809 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94ab7b96-9dfc-4065-b1eb-802f97e92abe-istio-podinfo\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-9csnb\" (UID: \"94ab7b96-9dfc-4065-b1eb-802f97e92abe\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-9csnb" Nov 03 13:41:38.651017 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:38.650832 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fqlrk\" (UniqueName: \"kubernetes.io/projected/94ab7b96-9dfc-4065-b1eb-802f97e92abe-kube-api-access-fqlrk\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-9csnb\" (UID: \"94ab7b96-9dfc-4065-b1eb-802f97e92abe\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-9csnb" Nov 03 13:41:38.651017 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:38.650988 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94ab7b96-9dfc-4065-b1eb-802f97e92abe-istio-token\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-9csnb\" (UID: \"94ab7b96-9dfc-4065-b1eb-802f97e92abe\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-9csnb" Nov 03 13:41:38.651191 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:38.651034 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94ab7b96-9dfc-4065-b1eb-802f97e92abe-istio-data\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-9csnb\" (UID: \"94ab7b96-9dfc-4065-b1eb-802f97e92abe\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-9csnb" Nov 03 13:41:38.651191 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:38.651148 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94ab7b96-9dfc-4065-b1eb-802f97e92abe-istio-envoy\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-9csnb\" (UID: \"94ab7b96-9dfc-4065-b1eb-802f97e92abe\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-9csnb" Nov 03 13:41:38.651191 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:38.651173 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94ab7b96-9dfc-4065-b1eb-802f97e92abe-workload-socket\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-9csnb\" (UID: \"94ab7b96-9dfc-4065-b1eb-802f97e92abe\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-9csnb" Nov 03 13:41:38.651396 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:38.651208 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94ab7b96-9dfc-4065-b1eb-802f97e92abe-credential-socket\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-9csnb\" (UID: \"94ab7b96-9dfc-4065-b1eb-802f97e92abe\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-9csnb" Nov 03 13:41:38.651396 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:38.651341 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94ab7b96-9dfc-4065-b1eb-802f97e92abe-workload-certs\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-9csnb\" (UID: \"94ab7b96-9dfc-4065-b1eb-802f97e92abe\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-9csnb" Nov 03 13:41:38.651501 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:38.651479 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94ab7b96-9dfc-4065-b1eb-802f97e92abe-istio-data\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-9csnb\" (UID: \"94ab7b96-9dfc-4065-b1eb-802f97e92abe\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-9csnb" Nov 03 13:41:38.651653 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:38.651625 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94ab7b96-9dfc-4065-b1eb-802f97e92abe-credential-socket\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-9csnb\" (UID: \"94ab7b96-9dfc-4065-b1eb-802f97e92abe\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-9csnb" Nov 03 13:41:38.651727 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:38.651672 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94ab7b96-9dfc-4065-b1eb-802f97e92abe-workload-certs\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-9csnb\" (UID: \"94ab7b96-9dfc-4065-b1eb-802f97e92abe\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-9csnb" Nov 03 13:41:38.651950 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:38.651932 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94ab7b96-9dfc-4065-b1eb-802f97e92abe-istiod-ca-cert\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-9csnb\" (UID: \"94ab7b96-9dfc-4065-b1eb-802f97e92abe\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-9csnb" Nov 03 13:41:38.653572 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:38.653550 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94ab7b96-9dfc-4065-b1eb-802f97e92abe-istio-envoy\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-9csnb\" (UID: \"94ab7b96-9dfc-4065-b1eb-802f97e92abe\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-9csnb" Nov 03 13:41:38.653896 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:38.653874 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94ab7b96-9dfc-4065-b1eb-802f97e92abe-istio-podinfo\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-9csnb\" (UID: \"94ab7b96-9dfc-4065-b1eb-802f97e92abe\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-9csnb" Nov 03 13:41:38.660320 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:38.660265 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94ab7b96-9dfc-4065-b1eb-802f97e92abe-istio-token\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-9csnb\" (UID: \"94ab7b96-9dfc-4065-b1eb-802f97e92abe\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-9csnb" Nov 03 13:41:38.661012 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:38.660988 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fqlrk\" (UniqueName: \"kubernetes.io/projected/94ab7b96-9dfc-4065-b1eb-802f97e92abe-kube-api-access-fqlrk\") pod \"e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-9csnb\" (UID: \"94ab7b96-9dfc-4065-b1eb-802f97e92abe\") " pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-9csnb" Nov 03 13:41:38.796020 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:38.795923 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-9csnb" Nov 03 13:41:39.114267 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:39.114233 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-9csnb"] Nov 03 13:41:39.115820 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:41:39.115787 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94ab7b96_9dfc_4065_b1eb_802f97e92abe.slice/crio-2341b0b8c6991cdae4c9fa31411ac6424c2686d4e44404c05b2fa0b3cac89b0b WatchSource:0}: Error finding container 2341b0b8c6991cdae4c9fa31411ac6424c2686d4e44404c05b2fa0b3cac89b0b: Status 404 returned error can't find the container with id 2341b0b8c6991cdae4c9fa31411ac6424c2686d4e44404c05b2fa0b3cac89b0b Nov 03 13:41:39.119984 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:39.119941 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:41:39.120198 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:39.120166 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:41:39.120334 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:39.120228 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:41:39.483834 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:39.483720 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-9csnb" event={"ID":"94ab7b96-9dfc-4065-b1eb-802f97e92abe","Type":"ContainerStarted","Data":"1eb966b318f277b1890dfa50b9b2750b6c1eb74401eaad55fcc249e0beb44ed3"} Nov 03 13:41:39.483834 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:39.483794 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-9csnb" event={"ID":"94ab7b96-9dfc-4065-b1eb-802f97e92abe","Type":"ContainerStarted","Data":"2341b0b8c6991cdae4c9fa31411ac6424c2686d4e44404c05b2fa0b3cac89b0b"} Nov 03 13:41:39.514399 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:39.514328 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-9csnb" podStartSLOduration=1.5143094910000001 podStartE2EDuration="1.514309491s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:41:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:41:39.50947723 +0000 UTC m=+1979.517668505" watchObservedRunningTime="2026-11-03 13:41:39.514309491 +0000 UTC m=+1979.522500769" Nov 03 13:41:39.521251 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:41:39.521212 5024 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d7mwz" containerName="extractor" Nov 03 13:41:39.817364 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:39.807402 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-9csnb" Nov 03 13:41:40.798458 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:40.798401 5024 patch_prober.go:29] interesting pod/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-9csnb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" start-of-body= Nov 03 13:41:40.799080 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:40.799042 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-9csnb" podUID="94ab7b96-9dfc-4065-b1eb-802f97e92abe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" Nov 03 13:41:41.800306 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:41.799743 5024 patch_prober.go:29] interesting pod/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-9csnb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" start-of-body= Nov 03 13:41:41.800306 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:41.799810 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-9csnb" podUID="94ab7b96-9dfc-4065-b1eb-802f97e92abe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" Nov 03 13:41:42.798436 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:42.798390 5024 patch_prober.go:29] interesting pod/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-9csnb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" start-of-body= Nov 03 13:41:42.798764 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:42.798731 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-9csnb" podUID="94ab7b96-9dfc-4065-b1eb-802f97e92abe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" Nov 03 13:41:43.802311 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:43.796774 5024 patch_prober.go:29] interesting pod/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-9csnb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" start-of-body= Nov 03 13:41:43.802311 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:43.796844 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-9csnb" podUID="94ab7b96-9dfc-4065-b1eb-802f97e92abe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" Nov 03 13:41:44.801740 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:44.801697 5024 patch_prober.go:29] interesting pod/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-9csnb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" start-of-body= Nov 03 13:41:44.802030 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:44.801998 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-9csnb" podUID="94ab7b96-9dfc-4065-b1eb-802f97e92abe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" Nov 03 13:41:45.804312 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:45.802780 5024 patch_prober.go:29] interesting pod/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-9csnb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" start-of-body= Nov 03 13:41:45.804312 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:45.802849 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-9csnb" podUID="94ab7b96-9dfc-4065-b1eb-802f97e92abe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" Nov 03 13:41:46.809309 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:46.808769 5024 patch_prober.go:29] interesting pod/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-9csnb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" start-of-body= Nov 03 13:41:46.809309 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:46.808835 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-9csnb" podUID="94ab7b96-9dfc-4065-b1eb-802f97e92abe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" Nov 03 13:41:46.992896 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:46.986436 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w2-a724a6/sub-test-model-a724a6-kserve-57d7c88d9f-5nk9m" podUID="3e6246a7-8c0e-4712-8f67-d834948a5673" containerName="main" probeResult="failure" output="Get \"https://10.132.0.153:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 13:41:48.724307 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:48.711688 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-9csnb" Nov 03 13:41:48.724307 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:48.712804 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-9csnb" Nov 03 13:41:48.724307 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:48.721504 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-9csnb" Nov 03 13:41:52.227528 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:52.227480 5024 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w2-a724a6/sub-test-model-a724a6-kserve-57d7c88d9f-5nk9m" podUID="3e6246a7-8c0e-4712-8f67-d834948a5673" containerName="main" containerID="cri-o://7cc8141653a4d544c0d8386c404e53b50a74307690e6e671fbf43dec102ecba3" gracePeriod=45 Nov 03 13:41:52.621795 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:52.621766 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-a724a6/sub-test-model-a724a6-kserve-57d7c88d9f-5nk9m" Nov 03 13:41:52.668528 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:52.668484 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e6246a7-8c0e-4712-8f67-d834948a5673-home\" (UniqueName: \"kubernetes.io/empty-dir/3e6246a7-8c0e-4712-8f67-d834948a5673-home\") pod \"3e6246a7-8c0e-4712-8f67-d834948a5673\" (UID: \"3e6246a7-8c0e-4712-8f67-d834948a5673\") " Nov 03 13:41:52.668773 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:52.668631 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e6246a7-8c0e-4712-8f67-d834948a5673-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3e6246a7-8c0e-4712-8f67-d834948a5673-tmp-dir\") pod \"3e6246a7-8c0e-4712-8f67-d834948a5673\" (UID: \"3e6246a7-8c0e-4712-8f67-d834948a5673\") " Nov 03 13:41:52.668773 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:52.668682 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e6246a7-8c0e-4712-8f67-d834948a5673-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3e6246a7-8c0e-4712-8f67-d834948a5673-model-cache\") pod \"3e6246a7-8c0e-4712-8f67-d834948a5673\" (UID: \"3e6246a7-8c0e-4712-8f67-d834948a5673\") " Nov 03 13:41:52.668773 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:52.668760 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3e6246a7-8c0e-4712-8f67-d834948a5673-tls-certs\" (UniqueName: \"kubernetes.io/secret/3e6246a7-8c0e-4712-8f67-d834948a5673-tls-certs\") pod \"3e6246a7-8c0e-4712-8f67-d834948a5673\" (UID: \"3e6246a7-8c0e-4712-8f67-d834948a5673\") " Nov 03 13:41:52.668773 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:52.668758 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e6246a7-8c0e-4712-8f67-d834948a5673-home" pod "3e6246a7-8c0e-4712-8f67-d834948a5673" (UID: "3e6246a7-8c0e-4712-8f67-d834948a5673"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:41:52.669027 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:52.668880 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e6246a7-8c0e-4712-8f67-d834948a5673-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3e6246a7-8c0e-4712-8f67-d834948a5673-dshm\") pod \"3e6246a7-8c0e-4712-8f67-d834948a5673\" (UID: \"3e6246a7-8c0e-4712-8f67-d834948a5673\") " Nov 03 13:41:52.669027 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:52.669000 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e6246a7-8c0e-4712-8f67-d834948a5673-kube-api-access-zs84z\" (UniqueName: \"kubernetes.io/projected/3e6246a7-8c0e-4712-8f67-d834948a5673-kube-api-access-zs84z\") pod \"3e6246a7-8c0e-4712-8f67-d834948a5673\" (UID: \"3e6246a7-8c0e-4712-8f67-d834948a5673\") " Nov 03 13:41:52.669027 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:52.669015 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e6246a7-8c0e-4712-8f67-d834948a5673-model-cache" pod "3e6246a7-8c0e-4712-8f67-d834948a5673" (UID: "3e6246a7-8c0e-4712-8f67-d834948a5673"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:41:52.669186 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:52.669028 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e6246a7-8c0e-4712-8f67-d834948a5673-tmp-dir" pod "3e6246a7-8c0e-4712-8f67-d834948a5673" (UID: "3e6246a7-8c0e-4712-8f67-d834948a5673"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:41:52.669709 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:52.669682 5024 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3e6246a7-8c0e-4712-8f67-d834948a5673-home\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:41:52.669828 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:52.669713 5024 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3e6246a7-8c0e-4712-8f67-d834948a5673-tmp-dir\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:41:52.669828 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:52.669727 5024 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3e6246a7-8c0e-4712-8f67-d834948a5673-model-cache\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:41:52.672047 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:52.672015 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e6246a7-8c0e-4712-8f67-d834948a5673-dshm" pod "3e6246a7-8c0e-4712-8f67-d834948a5673" (UID: "3e6246a7-8c0e-4712-8f67-d834948a5673"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:41:52.672161 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:52.672110 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3e6246a7-8c0e-4712-8f67-d834948a5673-tls-certs" pod "3e6246a7-8c0e-4712-8f67-d834948a5673" (UID: "3e6246a7-8c0e-4712-8f67-d834948a5673"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 13:41:52.672399 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:52.672374 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e6246a7-8c0e-4712-8f67-d834948a5673-kube-api-access-zs84z" pod "3e6246a7-8c0e-4712-8f67-d834948a5673" (UID: "3e6246a7-8c0e-4712-8f67-d834948a5673"). InnerVolumeSpecName "kube-api-access-zs84z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:41:52.771002 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:52.770955 5024 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3e6246a7-8c0e-4712-8f67-d834948a5673-tls-certs\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:41:52.771002 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:52.770993 5024 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3e6246a7-8c0e-4712-8f67-d834948a5673-dshm\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:41:52.771002 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:52.771004 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zs84z\" (UniqueName: \"kubernetes.io/projected/3e6246a7-8c0e-4712-8f67-d834948a5673-kube-api-access-zs84z\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:41:52.873010 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:52.872967 5024 generic.go:356] "Generic (PLEG): container finished" podID="3e6246a7-8c0e-4712-8f67-d834948a5673" containerID="7cc8141653a4d544c0d8386c404e53b50a74307690e6e671fbf43dec102ecba3" exitCode=0 Nov 03 13:41:52.873197 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:52.873021 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-a724a6/sub-test-model-a724a6-kserve-57d7c88d9f-5nk9m" Nov 03 13:41:52.873197 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:52.873052 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-a724a6/sub-test-model-a724a6-kserve-57d7c88d9f-5nk9m" event={"ID":"3e6246a7-8c0e-4712-8f67-d834948a5673","Type":"ContainerDied","Data":"7cc8141653a4d544c0d8386c404e53b50a74307690e6e671fbf43dec102ecba3"} Nov 03 13:41:52.873197 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:52.873112 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-a724a6/sub-test-model-a724a6-kserve-57d7c88d9f-5nk9m" event={"ID":"3e6246a7-8c0e-4712-8f67-d834948a5673","Type":"ContainerDied","Data":"ccc6fc1870df4c8aff637321973ff615fe3d32238a16ae27748fc857316666a3"} Nov 03 13:41:52.873197 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:52.873132 5024 scope.go:122] "RemoveContainer" containerID="7cc8141653a4d544c0d8386c404e53b50a74307690e6e671fbf43dec102ecba3" Nov 03 13:41:52.885944 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:52.885913 5024 scope.go:122] "RemoveContainer" containerID="7cc8141653a4d544c0d8386c404e53b50a74307690e6e671fbf43dec102ecba3" Nov 03 13:41:52.886346 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:41:52.886267 5024 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7cc8141653a4d544c0d8386c404e53b50a74307690e6e671fbf43dec102ecba3\": container with ID starting with 7cc8141653a4d544c0d8386c404e53b50a74307690e6e671fbf43dec102ecba3 not found: ID does not exist" containerID="7cc8141653a4d544c0d8386c404e53b50a74307690e6e671fbf43dec102ecba3" Nov 03 13:41:52.886445 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:52.886357 5024 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7cc8141653a4d544c0d8386c404e53b50a74307690e6e671fbf43dec102ecba3"} err="failed to get container status \"7cc8141653a4d544c0d8386c404e53b50a74307690e6e671fbf43dec102ecba3\": rpc error: code = NotFound desc = could not find container \"7cc8141653a4d544c0d8386c404e53b50a74307690e6e671fbf43dec102ecba3\": container with ID starting with 7cc8141653a4d544c0d8386c404e53b50a74307690e6e671fbf43dec102ecba3 not found: ID does not exist" Nov 03 13:41:52.901232 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:52.901197 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-a724a6/sub-test-model-a724a6-kserve-57d7c88d9f-5nk9m"] Nov 03 13:41:52.904409 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:52.904382 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w2-a724a6/sub-test-model-a724a6-kserve-57d7c88d9f-5nk9m"] Nov 03 13:41:53.730681 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:53.730315 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-xrw9k"] Nov 03 13:41:53.730681 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:53.730602 5024 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-xrw9k" podUID="e218218f-5605-485f-a27c-eb68f01c01a6" containerName="istio-proxy" containerID="cri-o://cfc03a1da0578b9dd7718f73f17c1aed80b383d1cdfbe54a74c7fe51347d3896" gracePeriod=30 Nov 03 13:41:53.748394 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:53.748360 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-9csnb"] Nov 03 13:41:53.748664 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:53.748637 5024 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-9csnb" podUID="94ab7b96-9dfc-4065-b1eb-802f97e92abe" containerName="istio-proxy" containerID="cri-o://1eb966b318f277b1890dfa50b9b2750b6c1eb74401eaad55fcc249e0beb44ed3" gracePeriod=30 Nov 03 13:41:53.758984 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:53.758950 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-mq7ng"] Nov 03 13:41:53.759188 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:53.759166 5024 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-mq7ng" podUID="ca19960c-df6f-44d1-b6b3-c20e06564315" containerName="istio-proxy" containerID="cri-o://b92d1a3a091f65c21db96f359a7f560892bbc3869f93eba34c250c2e957b6b8c" gracePeriod=30 Nov 03 13:41:54.112965 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:54.112922 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3e6246a7-8c0e-4712-8f67-d834948a5673" path="/var/lib/kubelet/pods/3e6246a7-8c0e-4712-8f67-d834948a5673/volumes" Nov 03 13:41:58.961044 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:58.959799 5024 generic.go:356] "Generic (PLEG): container finished" podID="e218218f-5605-485f-a27c-eb68f01c01a6" containerID="cfc03a1da0578b9dd7718f73f17c1aed80b383d1cdfbe54a74c7fe51347d3896" exitCode=0 Nov 03 13:41:58.961044 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:58.959897 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-xrw9k" event={"ID":"e218218f-5605-485f-a27c-eb68f01c01a6","Type":"ContainerDied","Data":"cfc03a1da0578b9dd7718f73f17c1aed80b383d1cdfbe54a74c7fe51347d3896"} Nov 03 13:41:58.961044 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:58.959990 5024 scope.go:122] "RemoveContainer" containerID="16e134c028375600bf74362974e1500fece99126914878b251a8136f7788669c" Nov 03 13:41:58.965104 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:58.964926 5024 generic.go:356] "Generic (PLEG): container finished" podID="ca19960c-df6f-44d1-b6b3-c20e06564315" containerID="b92d1a3a091f65c21db96f359a7f560892bbc3869f93eba34c250c2e957b6b8c" exitCode=0 Nov 03 13:41:58.965104 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:58.964989 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-mq7ng" event={"ID":"ca19960c-df6f-44d1-b6b3-c20e06564315","Type":"ContainerDied","Data":"b92d1a3a091f65c21db96f359a7f560892bbc3869f93eba34c250c2e957b6b8c"} Nov 03 13:41:58.967744 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:58.967714 5024 generic.go:356] "Generic (PLEG): container finished" podID="94ab7b96-9dfc-4065-b1eb-802f97e92abe" containerID="1eb966b318f277b1890dfa50b9b2750b6c1eb74401eaad55fcc249e0beb44ed3" exitCode=0 Nov 03 13:41:58.968009 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:58.967914 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-9csnb" event={"ID":"94ab7b96-9dfc-4065-b1eb-802f97e92abe","Type":"ContainerDied","Data":"1eb966b318f277b1890dfa50b9b2750b6c1eb74401eaad55fcc249e0beb44ed3"} Nov 03 13:41:59.083138 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:59.083098 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-9csnb" Nov 03 13:41:59.186120 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:59.186088 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-xrw9k" Nov 03 13:41:59.195013 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:59.192782 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/94ab7b96-9dfc-4065-b1eb-802f97e92abe-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94ab7b96-9dfc-4065-b1eb-802f97e92abe-istio-podinfo\") pod \"94ab7b96-9dfc-4065-b1eb-802f97e92abe\" (UID: \"94ab7b96-9dfc-4065-b1eb-802f97e92abe\") " Nov 03 13:41:59.195013 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:59.192852 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94ab7b96-9dfc-4065-b1eb-802f97e92abe-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94ab7b96-9dfc-4065-b1eb-802f97e92abe-istio-data\") pod \"94ab7b96-9dfc-4065-b1eb-802f97e92abe\" (UID: \"94ab7b96-9dfc-4065-b1eb-802f97e92abe\") " Nov 03 13:41:59.195013 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:59.192931 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94ab7b96-9dfc-4065-b1eb-802f97e92abe-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94ab7b96-9dfc-4065-b1eb-802f97e92abe-istio-envoy\") pod \"94ab7b96-9dfc-4065-b1eb-802f97e92abe\" (UID: \"94ab7b96-9dfc-4065-b1eb-802f97e92abe\") " Nov 03 13:41:59.195013 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:59.193056 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/94ab7b96-9dfc-4065-b1eb-802f97e92abe-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94ab7b96-9dfc-4065-b1eb-802f97e92abe-istiod-ca-cert\") pod \"94ab7b96-9dfc-4065-b1eb-802f97e92abe\" (UID: \"94ab7b96-9dfc-4065-b1eb-802f97e92abe\") " Nov 03 13:41:59.195013 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:59.193109 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94ab7b96-9dfc-4065-b1eb-802f97e92abe-istio-token\" (UniqueName: \"kubernetes.io/projected/94ab7b96-9dfc-4065-b1eb-802f97e92abe-istio-token\") pod \"94ab7b96-9dfc-4065-b1eb-802f97e92abe\" (UID: \"94ab7b96-9dfc-4065-b1eb-802f97e92abe\") " Nov 03 13:41:59.195013 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:59.193148 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94ab7b96-9dfc-4065-b1eb-802f97e92abe-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94ab7b96-9dfc-4065-b1eb-802f97e92abe-workload-certs\") pod \"94ab7b96-9dfc-4065-b1eb-802f97e92abe\" (UID: \"94ab7b96-9dfc-4065-b1eb-802f97e92abe\") " Nov 03 13:41:59.195013 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:59.193228 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94ab7b96-9dfc-4065-b1eb-802f97e92abe-kube-api-access-fqlrk\" (UniqueName: \"kubernetes.io/projected/94ab7b96-9dfc-4065-b1eb-802f97e92abe-kube-api-access-fqlrk\") pod \"94ab7b96-9dfc-4065-b1eb-802f97e92abe\" (UID: \"94ab7b96-9dfc-4065-b1eb-802f97e92abe\") " Nov 03 13:41:59.195013 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:59.193363 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94ab7b96-9dfc-4065-b1eb-802f97e92abe-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94ab7b96-9dfc-4065-b1eb-802f97e92abe-workload-socket\") pod \"94ab7b96-9dfc-4065-b1eb-802f97e92abe\" (UID: \"94ab7b96-9dfc-4065-b1eb-802f97e92abe\") " Nov 03 13:41:59.195013 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:59.193503 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94ab7b96-9dfc-4065-b1eb-802f97e92abe-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94ab7b96-9dfc-4065-b1eb-802f97e92abe-credential-socket\") pod \"94ab7b96-9dfc-4065-b1eb-802f97e92abe\" (UID: \"94ab7b96-9dfc-4065-b1eb-802f97e92abe\") " Nov 03 13:41:59.197609 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:59.197404 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94ab7b96-9dfc-4065-b1eb-802f97e92abe-istio-data" pod "94ab7b96-9dfc-4065-b1eb-802f97e92abe" (UID: "94ab7b96-9dfc-4065-b1eb-802f97e92abe"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:41:59.197609 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:59.197514 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94ab7b96-9dfc-4065-b1eb-802f97e92abe-istiod-ca-cert" pod "94ab7b96-9dfc-4065-b1eb-802f97e92abe" (UID: "94ab7b96-9dfc-4065-b1eb-802f97e92abe"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:41:59.197609 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:59.197555 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94ab7b96-9dfc-4065-b1eb-802f97e92abe-workload-socket" pod "94ab7b96-9dfc-4065-b1eb-802f97e92abe" (UID: "94ab7b96-9dfc-4065-b1eb-802f97e92abe"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:41:59.197891 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:59.197853 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94ab7b96-9dfc-4065-b1eb-802f97e92abe-workload-certs" pod "94ab7b96-9dfc-4065-b1eb-802f97e92abe" (UID: "94ab7b96-9dfc-4065-b1eb-802f97e92abe"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:41:59.198018 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:59.197902 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94ab7b96-9dfc-4065-b1eb-802f97e92abe-credential-socket" pod "94ab7b96-9dfc-4065-b1eb-802f97e92abe" (UID: "94ab7b96-9dfc-4065-b1eb-802f97e92abe"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:41:59.198939 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:59.198909 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/94ab7b96-9dfc-4065-b1eb-802f97e92abe-istio-podinfo" pod "94ab7b96-9dfc-4065-b1eb-802f97e92abe" (UID: "94ab7b96-9dfc-4065-b1eb-802f97e92abe"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 13:41:59.199519 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:59.199491 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94ab7b96-9dfc-4065-b1eb-802f97e92abe-istio-envoy" pod "94ab7b96-9dfc-4065-b1eb-802f97e92abe" (UID: "94ab7b96-9dfc-4065-b1eb-802f97e92abe"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:41:59.200167 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:59.200145 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94ab7b96-9dfc-4065-b1eb-802f97e92abe-istio-token" pod "94ab7b96-9dfc-4065-b1eb-802f97e92abe" (UID: "94ab7b96-9dfc-4065-b1eb-802f97e92abe"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:41:59.207437 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:59.207384 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94ab7b96-9dfc-4065-b1eb-802f97e92abe-kube-api-access-fqlrk" pod "94ab7b96-9dfc-4065-b1eb-802f97e92abe" (UID: "94ab7b96-9dfc-4065-b1eb-802f97e92abe"). InnerVolumeSpecName "kube-api-access-fqlrk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:41:59.247687 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:59.247646 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-mq7ng" Nov 03 13:41:59.295797 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:59.295695 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e218218f-5605-485f-a27c-eb68f01c01a6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e218218f-5605-485f-a27c-eb68f01c01a6-workload-certs\") pod \"e218218f-5605-485f-a27c-eb68f01c01a6\" (UID: \"e218218f-5605-485f-a27c-eb68f01c01a6\") " Nov 03 13:41:59.295797 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:59.295770 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e218218f-5605-485f-a27c-eb68f01c01a6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e218218f-5605-485f-a27c-eb68f01c01a6-credential-socket\") pod \"e218218f-5605-485f-a27c-eb68f01c01a6\" (UID: \"e218218f-5605-485f-a27c-eb68f01c01a6\") " Nov 03 13:41:59.296039 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:59.295851 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e218218f-5605-485f-a27c-eb68f01c01a6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e218218f-5605-485f-a27c-eb68f01c01a6-istio-envoy\") pod \"e218218f-5605-485f-a27c-eb68f01c01a6\" (UID: \"e218218f-5605-485f-a27c-eb68f01c01a6\") " Nov 03 13:41:59.296039 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:59.295962 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e218218f-5605-485f-a27c-eb68f01c01a6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e218218f-5605-485f-a27c-eb68f01c01a6-workload-socket\") pod \"e218218f-5605-485f-a27c-eb68f01c01a6\" (UID: \"e218218f-5605-485f-a27c-eb68f01c01a6\") " Nov 03 13:41:59.296039 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:59.296000 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e218218f-5605-485f-a27c-eb68f01c01a6-kube-api-access-w9xsp\" (UniqueName: \"kubernetes.io/projected/e218218f-5605-485f-a27c-eb68f01c01a6-kube-api-access-w9xsp\") pod \"e218218f-5605-485f-a27c-eb68f01c01a6\" (UID: \"e218218f-5605-485f-a27c-eb68f01c01a6\") " Nov 03 13:41:59.296221 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:59.296044 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e218218f-5605-485f-a27c-eb68f01c01a6-istio-token\" (UniqueName: \"kubernetes.io/projected/e218218f-5605-485f-a27c-eb68f01c01a6-istio-token\") pod \"e218218f-5605-485f-a27c-eb68f01c01a6\" (UID: \"e218218f-5605-485f-a27c-eb68f01c01a6\") " Nov 03 13:41:59.296221 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:59.296074 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e218218f-5605-485f-a27c-eb68f01c01a6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e218218f-5605-485f-a27c-eb68f01c01a6-istio-podinfo\") pod \"e218218f-5605-485f-a27c-eb68f01c01a6\" (UID: \"e218218f-5605-485f-a27c-eb68f01c01a6\") " Nov 03 13:41:59.296221 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:59.296120 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e218218f-5605-485f-a27c-eb68f01c01a6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e218218f-5605-485f-a27c-eb68f01c01a6-istiod-ca-cert\") pod \"e218218f-5605-485f-a27c-eb68f01c01a6\" (UID: \"e218218f-5605-485f-a27c-eb68f01c01a6\") " Nov 03 13:41:59.296221 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:59.296158 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e218218f-5605-485f-a27c-eb68f01c01a6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e218218f-5605-485f-a27c-eb68f01c01a6-istio-data\") pod \"e218218f-5605-485f-a27c-eb68f01c01a6\" (UID: \"e218218f-5605-485f-a27c-eb68f01c01a6\") " Nov 03 13:41:59.296645 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:59.296622 5024 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94ab7b96-9dfc-4065-b1eb-802f97e92abe-istio-envoy\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:41:59.296715 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:59.296647 5024 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94ab7b96-9dfc-4065-b1eb-802f97e92abe-istiod-ca-cert\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:41:59.296715 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:59.296657 5024 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94ab7b96-9dfc-4065-b1eb-802f97e92abe-istio-token\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:41:59.296715 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:59.296666 5024 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94ab7b96-9dfc-4065-b1eb-802f97e92abe-workload-certs\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:41:59.296715 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:59.296676 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fqlrk\" (UniqueName: \"kubernetes.io/projected/94ab7b96-9dfc-4065-b1eb-802f97e92abe-kube-api-access-fqlrk\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:41:59.296715 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:59.296684 5024 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94ab7b96-9dfc-4065-b1eb-802f97e92abe-workload-socket\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:41:59.296715 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:59.296693 5024 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94ab7b96-9dfc-4065-b1eb-802f97e92abe-credential-socket\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:41:59.296715 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:59.296702 5024 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94ab7b96-9dfc-4065-b1eb-802f97e92abe-istio-podinfo\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:41:59.296715 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:59.296715 5024 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94ab7b96-9dfc-4065-b1eb-802f97e92abe-istio-data\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:41:59.297119 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:59.296941 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e218218f-5605-485f-a27c-eb68f01c01a6-istio-data" pod "e218218f-5605-485f-a27c-eb68f01c01a6" (UID: "e218218f-5605-485f-a27c-eb68f01c01a6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:41:59.297119 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:59.297092 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e218218f-5605-485f-a27c-eb68f01c01a6-workload-certs" pod "e218218f-5605-485f-a27c-eb68f01c01a6" (UID: "e218218f-5605-485f-a27c-eb68f01c01a6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:41:59.297337 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:59.297315 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e218218f-5605-485f-a27c-eb68f01c01a6-credential-socket" pod "e218218f-5605-485f-a27c-eb68f01c01a6" (UID: "e218218f-5605-485f-a27c-eb68f01c01a6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:41:59.297977 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:59.297950 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e218218f-5605-485f-a27c-eb68f01c01a6-workload-socket" pod "e218218f-5605-485f-a27c-eb68f01c01a6" (UID: "e218218f-5605-485f-a27c-eb68f01c01a6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:41:59.298176 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:59.298144 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e218218f-5605-485f-a27c-eb68f01c01a6-istiod-ca-cert" pod "e218218f-5605-485f-a27c-eb68f01c01a6" (UID: "e218218f-5605-485f-a27c-eb68f01c01a6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:41:59.300961 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:59.300923 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e218218f-5605-485f-a27c-eb68f01c01a6-istio-envoy" pod "e218218f-5605-485f-a27c-eb68f01c01a6" (UID: "e218218f-5605-485f-a27c-eb68f01c01a6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:41:59.301747 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:59.301721 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e218218f-5605-485f-a27c-eb68f01c01a6-kube-api-access-w9xsp" pod "e218218f-5605-485f-a27c-eb68f01c01a6" (UID: "e218218f-5605-485f-a27c-eb68f01c01a6"). InnerVolumeSpecName "kube-api-access-w9xsp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:41:59.301980 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:59.301958 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e218218f-5605-485f-a27c-eb68f01c01a6-istio-podinfo" pod "e218218f-5605-485f-a27c-eb68f01c01a6" (UID: "e218218f-5605-485f-a27c-eb68f01c01a6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 13:41:59.303480 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:59.303456 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e218218f-5605-485f-a27c-eb68f01c01a6-istio-token" pod "e218218f-5605-485f-a27c-eb68f01c01a6" (UID: "e218218f-5605-485f-a27c-eb68f01c01a6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:41:59.397636 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:59.397600 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca19960c-df6f-44d1-b6b3-c20e06564315-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca19960c-df6f-44d1-b6b3-c20e06564315-istio-data\") pod \"ca19960c-df6f-44d1-b6b3-c20e06564315\" (UID: \"ca19960c-df6f-44d1-b6b3-c20e06564315\") " Nov 03 13:41:59.397636 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:59.397641 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca19960c-df6f-44d1-b6b3-c20e06564315-istio-token\" (UniqueName: \"kubernetes.io/projected/ca19960c-df6f-44d1-b6b3-c20e06564315-istio-token\") pod \"ca19960c-df6f-44d1-b6b3-c20e06564315\" (UID: \"ca19960c-df6f-44d1-b6b3-c20e06564315\") " Nov 03 13:41:59.397904 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:59.397818 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca19960c-df6f-44d1-b6b3-c20e06564315-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca19960c-df6f-44d1-b6b3-c20e06564315-istio-envoy\") pod \"ca19960c-df6f-44d1-b6b3-c20e06564315\" (UID: \"ca19960c-df6f-44d1-b6b3-c20e06564315\") " Nov 03 13:41:59.397904 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:59.397855 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ca19960c-df6f-44d1-b6b3-c20e06564315-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca19960c-df6f-44d1-b6b3-c20e06564315-istio-podinfo\") pod \"ca19960c-df6f-44d1-b6b3-c20e06564315\" (UID: \"ca19960c-df6f-44d1-b6b3-c20e06564315\") " Nov 03 13:41:59.397904 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:59.397884 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca19960c-df6f-44d1-b6b3-c20e06564315-istio-data" pod "ca19960c-df6f-44d1-b6b3-c20e06564315" (UID: "ca19960c-df6f-44d1-b6b3-c20e06564315"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:41:59.398165 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:59.397973 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ca19960c-df6f-44d1-b6b3-c20e06564315-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca19960c-df6f-44d1-b6b3-c20e06564315-istiod-ca-cert\") pod \"ca19960c-df6f-44d1-b6b3-c20e06564315\" (UID: \"ca19960c-df6f-44d1-b6b3-c20e06564315\") " Nov 03 13:41:59.398165 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:59.398037 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca19960c-df6f-44d1-b6b3-c20e06564315-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca19960c-df6f-44d1-b6b3-c20e06564315-workload-certs\") pod \"ca19960c-df6f-44d1-b6b3-c20e06564315\" (UID: \"ca19960c-df6f-44d1-b6b3-c20e06564315\") " Nov 03 13:41:59.398165 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:59.398077 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca19960c-df6f-44d1-b6b3-c20e06564315-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca19960c-df6f-44d1-b6b3-c20e06564315-workload-socket\") pod \"ca19960c-df6f-44d1-b6b3-c20e06564315\" (UID: \"ca19960c-df6f-44d1-b6b3-c20e06564315\") " Nov 03 13:41:59.398165 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:59.398116 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca19960c-df6f-44d1-b6b3-c20e06564315-kube-api-access-vmthd\" (UniqueName: \"kubernetes.io/projected/ca19960c-df6f-44d1-b6b3-c20e06564315-kube-api-access-vmthd\") pod \"ca19960c-df6f-44d1-b6b3-c20e06564315\" (UID: \"ca19960c-df6f-44d1-b6b3-c20e06564315\") " Nov 03 13:41:59.398422 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:59.398242 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca19960c-df6f-44d1-b6b3-c20e06564315-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca19960c-df6f-44d1-b6b3-c20e06564315-credential-socket\") pod \"ca19960c-df6f-44d1-b6b3-c20e06564315\" (UID: \"ca19960c-df6f-44d1-b6b3-c20e06564315\") " Nov 03 13:41:59.398639 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:59.398613 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca19960c-df6f-44d1-b6b3-c20e06564315-istiod-ca-cert" pod "ca19960c-df6f-44d1-b6b3-c20e06564315" (UID: "ca19960c-df6f-44d1-b6b3-c20e06564315"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:41:59.398722 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:59.398703 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca19960c-df6f-44d1-b6b3-c20e06564315-workload-socket" pod "ca19960c-df6f-44d1-b6b3-c20e06564315" (UID: "ca19960c-df6f-44d1-b6b3-c20e06564315"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:41:59.398835 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:59.398811 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca19960c-df6f-44d1-b6b3-c20e06564315-credential-socket" pod "ca19960c-df6f-44d1-b6b3-c20e06564315" (UID: "ca19960c-df6f-44d1-b6b3-c20e06564315"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:41:59.398896 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:59.398875 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca19960c-df6f-44d1-b6b3-c20e06564315-workload-certs" pod "ca19960c-df6f-44d1-b6b3-c20e06564315" (UID: "ca19960c-df6f-44d1-b6b3-c20e06564315"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:41:59.399034 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:59.399019 5024 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e218218f-5605-485f-a27c-eb68f01c01a6-istio-token\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:41:59.399094 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:59.399041 5024 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e218218f-5605-485f-a27c-eb68f01c01a6-istio-podinfo\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:41:59.399094 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:59.399054 5024 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e218218f-5605-485f-a27c-eb68f01c01a6-istiod-ca-cert\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:41:59.399094 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:59.399064 5024 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca19960c-df6f-44d1-b6b3-c20e06564315-credential-socket\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:41:59.399094 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:59.399074 5024 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e218218f-5605-485f-a27c-eb68f01c01a6-istio-data\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:41:59.399094 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:59.399082 5024 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca19960c-df6f-44d1-b6b3-c20e06564315-istio-data\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:41:59.399094 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:59.399089 5024 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e218218f-5605-485f-a27c-eb68f01c01a6-workload-certs\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:41:59.399449 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:59.399098 5024 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e218218f-5605-485f-a27c-eb68f01c01a6-credential-socket\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:41:59.399449 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:59.399107 5024 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e218218f-5605-485f-a27c-eb68f01c01a6-istio-envoy\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:41:59.399449 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:59.399120 5024 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca19960c-df6f-44d1-b6b3-c20e06564315-istiod-ca-cert\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:41:59.399449 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:59.399131 5024 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e218218f-5605-485f-a27c-eb68f01c01a6-workload-socket\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:41:59.399449 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:59.399139 5024 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca19960c-df6f-44d1-b6b3-c20e06564315-workload-certs\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:41:59.399449 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:59.399147 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w9xsp\" (UniqueName: \"kubernetes.io/projected/e218218f-5605-485f-a27c-eb68f01c01a6-kube-api-access-w9xsp\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:41:59.399449 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:59.399157 5024 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca19960c-df6f-44d1-b6b3-c20e06564315-workload-socket\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:41:59.400639 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:59.400615 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca19960c-df6f-44d1-b6b3-c20e06564315-istio-envoy" pod "ca19960c-df6f-44d1-b6b3-c20e06564315" (UID: "ca19960c-df6f-44d1-b6b3-c20e06564315"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:41:59.400758 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:59.400741 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca19960c-df6f-44d1-b6b3-c20e06564315-istio-token" pod "ca19960c-df6f-44d1-b6b3-c20e06564315" (UID: "ca19960c-df6f-44d1-b6b3-c20e06564315"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:41:59.400832 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:59.400813 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca19960c-df6f-44d1-b6b3-c20e06564315-kube-api-access-vmthd" pod "ca19960c-df6f-44d1-b6b3-c20e06564315" (UID: "ca19960c-df6f-44d1-b6b3-c20e06564315"). InnerVolumeSpecName "kube-api-access-vmthd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:41:59.400880 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:59.400847 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ca19960c-df6f-44d1-b6b3-c20e06564315-istio-podinfo" pod "ca19960c-df6f-44d1-b6b3-c20e06564315" (UID: "ca19960c-df6f-44d1-b6b3-c20e06564315"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 13:41:59.500813 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:59.500762 5024 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca19960c-df6f-44d1-b6b3-c20e06564315-istio-envoy\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:41:59.500813 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:59.500814 5024 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca19960c-df6f-44d1-b6b3-c20e06564315-istio-podinfo\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:41:59.501030 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:59.500830 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vmthd\" (UniqueName: \"kubernetes.io/projected/ca19960c-df6f-44d1-b6b3-c20e06564315-kube-api-access-vmthd\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:41:59.501030 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:59.500847 5024 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca19960c-df6f-44d1-b6b3-c20e06564315-istio-token\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:41:59.976557 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:59.976522 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-xrw9k" Nov 03 13:41:59.976982 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:59.976523 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-xrw9k" event={"ID":"e218218f-5605-485f-a27c-eb68f01c01a6","Type":"ContainerDied","Data":"f8dca2598bf2af0366353e4c2091628ced1d134fc5f36cd02dd5554ca7819b25"} Nov 03 13:41:59.976982 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:59.976684 5024 scope.go:122] "RemoveContainer" containerID="cfc03a1da0578b9dd7718f73f17c1aed80b383d1cdfbe54a74c7fe51347d3896" Nov 03 13:41:59.978054 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:59.978036 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-mq7ng" Nov 03 13:41:59.978148 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:59.978072 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-mq7ng" event={"ID":"ca19960c-df6f-44d1-b6b3-c20e06564315","Type":"ContainerDied","Data":"8512f180aac54a29ea18cc9f23cc0d6224997188580c7efa28df4886824fff5c"} Nov 03 13:41:59.979837 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:59.979810 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-9csnb" event={"ID":"94ab7b96-9dfc-4065-b1eb-802f97e92abe","Type":"ContainerDied","Data":"2341b0b8c6991cdae4c9fa31411ac6424c2686d4e44404c05b2fa0b3cac89b0b"} Nov 03 13:41:59.979837 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:59.979831 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-9csnb" Nov 03 13:41:59.989486 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:41:59.989458 5024 scope.go:122] "RemoveContainer" containerID="b92d1a3a091f65c21db96f359a7f560892bbc3869f93eba34c250c2e957b6b8c" Nov 03 13:42:00.001322 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:00.001263 5024 scope.go:122] "RemoveContainer" containerID="1eb966b318f277b1890dfa50b9b2750b6c1eb74401eaad55fcc249e0beb44ed3" Nov 03 13:42:00.022643 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:00.022567 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-xrw9k"] Nov 03 13:42:00.028357 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:00.028324 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-xrw9k"] Nov 03 13:42:00.042898 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:00.042864 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-9csnb"] Nov 03 13:42:00.048656 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:00.048616 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-9csnb"] Nov 03 13:42:00.065194 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:00.065158 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-mq7ng"] Nov 03 13:42:00.068155 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:00.068133 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-d3ca48-openshift-default-67b76b76f7-mq7ng"] Nov 03 13:42:00.109869 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:00.109827 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="94ab7b96-9dfc-4065-b1eb-802f97e92abe" path="/var/lib/kubelet/pods/94ab7b96-9dfc-4065-b1eb-802f97e92abe/volumes" Nov 03 13:42:00.110328 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:00.110306 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ca19960c-df6f-44d1-b6b3-c20e06564315" path="/var/lib/kubelet/pods/ca19960c-df6f-44d1-b6b3-c20e06564315/volumes" Nov 03 13:42:00.110759 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:00.110744 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e218218f-5605-485f-a27c-eb68f01c01a6" path="/var/lib/kubelet/pods/e218218f-5605-485f-a27c-eb68f01c01a6/volumes" Nov 03 13:42:03.529156 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:03.529111 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-e440bc/e2e-conflict-b-9d13b16c-kserve-5d69bbd584-l47wz"] Nov 03 13:42:03.529793 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:03.529783 5024 cpu_manager.go:414] "containerMap: removing container" podUID="ca19960c-df6f-44d1-b6b3-c20e06564315" containerName="istio-proxy" Nov 03 13:42:03.529873 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:03.529798 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca19960c-df6f-44d1-b6b3-c20e06564315" containerName="istio-proxy" Nov 03 13:42:03.529873 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:03.529810 5024 cpu_manager.go:414] "containerMap: removing container" podUID="94ab7b96-9dfc-4065-b1eb-802f97e92abe" containerName="istio-proxy" Nov 03 13:42:03.529873 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:03.529816 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94ab7b96-9dfc-4065-b1eb-802f97e92abe" containerName="istio-proxy" Nov 03 13:42:03.529873 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:03.529837 5024 cpu_manager.go:414] "containerMap: removing container" podUID="3e6246a7-8c0e-4712-8f67-d834948a5673" containerName="main" Nov 03 13:42:03.529873 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:03.529843 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e6246a7-8c0e-4712-8f67-d834948a5673" containerName="main" Nov 03 13:42:03.529873 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:03.529870 5024 cpu_manager.go:414] "containerMap: removing container" podUID="e218218f-5605-485f-a27c-eb68f01c01a6" containerName="istio-proxy" Nov 03 13:42:03.529873 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:03.529874 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e218218f-5605-485f-a27c-eb68f01c01a6" containerName="istio-proxy" Nov 03 13:42:03.530242 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:03.529893 5024 cpu_manager.go:414] "containerMap: removing container" podUID="e218218f-5605-485f-a27c-eb68f01c01a6" containerName="istio-proxy" Nov 03 13:42:03.530242 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:03.529899 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e218218f-5605-485f-a27c-eb68f01c01a6" containerName="istio-proxy" Nov 03 13:42:03.530242 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:03.529982 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="e218218f-5605-485f-a27c-eb68f01c01a6" containerName="istio-proxy" Nov 03 13:42:03.530242 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:03.529993 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="e218218f-5605-485f-a27c-eb68f01c01a6" containerName="istio-proxy" Nov 03 13:42:03.530242 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:03.530002 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca19960c-df6f-44d1-b6b3-c20e06564315" containerName="istio-proxy" Nov 03 13:42:03.530242 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:03.530009 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="94ab7b96-9dfc-4065-b1eb-802f97e92abe" containerName="istio-proxy" Nov 03 13:42:03.530242 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:03.530015 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e6246a7-8c0e-4712-8f67-d834948a5673" containerName="main" Nov 03 13:42:03.535761 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:03.535732 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-e440bc/e2e-conflict-b-9d13b16c-kserve-5d69bbd584-l47wz" Nov 03 13:42:03.551110 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:03.551085 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-e440bc\"/\"default-dockercfg-5nftw\"" Nov 03 13:42:03.557997 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:03.557967 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-e440bc\"/\"openshift-service-ca.crt\"" Nov 03 13:42:03.558736 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:03.558714 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-e440bc\"/\"kube-root-ca.crt\"" Nov 03 13:42:03.560404 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:03.560379 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-e440bc\"/\"e2e-conflict-b-9d13b16c-kserve-self-signed-certs\"" Nov 03 13:42:03.566644 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:03.566615 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-e440bc/e2e-conflict-b-9d13b16c-kserve-5d69bbd584-l47wz"] Nov 03 13:42:03.574899 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:03.574856 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4f929810-b049-4820-a076-b8f9b78dffb0-model-cache\") pod \"e2e-conflict-b-9d13b16c-kserve-5d69bbd584-l47wz\" (UID: \"4f929810-b049-4820-a076-b8f9b78dffb0\") " pod="e2e-models-e2e-worker-w2-e440bc/e2e-conflict-b-9d13b16c-kserve-5d69bbd584-l47wz" Nov 03 13:42:03.575094 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:03.574916 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4f929810-b049-4820-a076-b8f9b78dffb0-dshm\") pod \"e2e-conflict-b-9d13b16c-kserve-5d69bbd584-l47wz\" (UID: \"4f929810-b049-4820-a076-b8f9b78dffb0\") " pod="e2e-models-e2e-worker-w2-e440bc/e2e-conflict-b-9d13b16c-kserve-5d69bbd584-l47wz" Nov 03 13:42:03.575094 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:03.575063 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4f929810-b049-4820-a076-b8f9b78dffb0-home\") pod \"e2e-conflict-b-9d13b16c-kserve-5d69bbd584-l47wz\" (UID: \"4f929810-b049-4820-a076-b8f9b78dffb0\") " pod="e2e-models-e2e-worker-w2-e440bc/e2e-conflict-b-9d13b16c-kserve-5d69bbd584-l47wz" Nov 03 13:42:03.575223 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:03.575122 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f929810-b049-4820-a076-b8f9b78dffb0-tls-certs\") pod \"e2e-conflict-b-9d13b16c-kserve-5d69bbd584-l47wz\" (UID: \"4f929810-b049-4820-a076-b8f9b78dffb0\") " pod="e2e-models-e2e-worker-w2-e440bc/e2e-conflict-b-9d13b16c-kserve-5d69bbd584-l47wz" Nov 03 13:42:03.575462 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:03.575432 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5t7sr\" (UniqueName: \"kubernetes.io/projected/4f929810-b049-4820-a076-b8f9b78dffb0-kube-api-access-5t7sr\") pod \"e2e-conflict-b-9d13b16c-kserve-5d69bbd584-l47wz\" (UID: \"4f929810-b049-4820-a076-b8f9b78dffb0\") " pod="e2e-models-e2e-worker-w2-e440bc/e2e-conflict-b-9d13b16c-kserve-5d69bbd584-l47wz" Nov 03 13:42:03.575585 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:03.575507 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f929810-b049-4820-a076-b8f9b78dffb0-tmp-dir\") pod \"e2e-conflict-b-9d13b16c-kserve-5d69bbd584-l47wz\" (UID: \"4f929810-b049-4820-a076-b8f9b78dffb0\") " pod="e2e-models-e2e-worker-w2-e440bc/e2e-conflict-b-9d13b16c-kserve-5d69bbd584-l47wz" Nov 03 13:42:03.677406 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:03.677356 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4f929810-b049-4820-a076-b8f9b78dffb0-model-cache\") pod \"e2e-conflict-b-9d13b16c-kserve-5d69bbd584-l47wz\" (UID: \"4f929810-b049-4820-a076-b8f9b78dffb0\") " pod="e2e-models-e2e-worker-w2-e440bc/e2e-conflict-b-9d13b16c-kserve-5d69bbd584-l47wz" Nov 03 13:42:03.677617 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:03.677416 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4f929810-b049-4820-a076-b8f9b78dffb0-dshm\") pod \"e2e-conflict-b-9d13b16c-kserve-5d69bbd584-l47wz\" (UID: \"4f929810-b049-4820-a076-b8f9b78dffb0\") " pod="e2e-models-e2e-worker-w2-e440bc/e2e-conflict-b-9d13b16c-kserve-5d69bbd584-l47wz" Nov 03 13:42:03.677617 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:03.677494 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4f929810-b049-4820-a076-b8f9b78dffb0-home\") pod \"e2e-conflict-b-9d13b16c-kserve-5d69bbd584-l47wz\" (UID: \"4f929810-b049-4820-a076-b8f9b78dffb0\") " pod="e2e-models-e2e-worker-w2-e440bc/e2e-conflict-b-9d13b16c-kserve-5d69bbd584-l47wz" Nov 03 13:42:03.677617 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:03.677534 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f929810-b049-4820-a076-b8f9b78dffb0-tls-certs\") pod \"e2e-conflict-b-9d13b16c-kserve-5d69bbd584-l47wz\" (UID: \"4f929810-b049-4820-a076-b8f9b78dffb0\") " pod="e2e-models-e2e-worker-w2-e440bc/e2e-conflict-b-9d13b16c-kserve-5d69bbd584-l47wz" Nov 03 13:42:03.677800 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:03.677654 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5t7sr\" (UniqueName: \"kubernetes.io/projected/4f929810-b049-4820-a076-b8f9b78dffb0-kube-api-access-5t7sr\") pod \"e2e-conflict-b-9d13b16c-kserve-5d69bbd584-l47wz\" (UID: \"4f929810-b049-4820-a076-b8f9b78dffb0\") " pod="e2e-models-e2e-worker-w2-e440bc/e2e-conflict-b-9d13b16c-kserve-5d69bbd584-l47wz" Nov 03 13:42:03.677800 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:03.677716 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f929810-b049-4820-a076-b8f9b78dffb0-tmp-dir\") pod \"e2e-conflict-b-9d13b16c-kserve-5d69bbd584-l47wz\" (UID: \"4f929810-b049-4820-a076-b8f9b78dffb0\") " pod="e2e-models-e2e-worker-w2-e440bc/e2e-conflict-b-9d13b16c-kserve-5d69bbd584-l47wz" Nov 03 13:42:03.677933 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:03.677897 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4f929810-b049-4820-a076-b8f9b78dffb0-model-cache\") pod \"e2e-conflict-b-9d13b16c-kserve-5d69bbd584-l47wz\" (UID: \"4f929810-b049-4820-a076-b8f9b78dffb0\") " pod="e2e-models-e2e-worker-w2-e440bc/e2e-conflict-b-9d13b16c-kserve-5d69bbd584-l47wz" Nov 03 13:42:03.678255 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:03.678228 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4f929810-b049-4820-a076-b8f9b78dffb0-home\") pod \"e2e-conflict-b-9d13b16c-kserve-5d69bbd584-l47wz\" (UID: \"4f929810-b049-4820-a076-b8f9b78dffb0\") " pod="e2e-models-e2e-worker-w2-e440bc/e2e-conflict-b-9d13b16c-kserve-5d69bbd584-l47wz" Nov 03 13:42:03.678397 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:03.678296 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f929810-b049-4820-a076-b8f9b78dffb0-tmp-dir\") pod \"e2e-conflict-b-9d13b16c-kserve-5d69bbd584-l47wz\" (UID: \"4f929810-b049-4820-a076-b8f9b78dffb0\") " pod="e2e-models-e2e-worker-w2-e440bc/e2e-conflict-b-9d13b16c-kserve-5d69bbd584-l47wz" Nov 03 13:42:03.679939 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:03.679908 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4f929810-b049-4820-a076-b8f9b78dffb0-dshm\") pod \"e2e-conflict-b-9d13b16c-kserve-5d69bbd584-l47wz\" (UID: \"4f929810-b049-4820-a076-b8f9b78dffb0\") " pod="e2e-models-e2e-worker-w2-e440bc/e2e-conflict-b-9d13b16c-kserve-5d69bbd584-l47wz" Nov 03 13:42:03.680076 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:03.680056 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f929810-b049-4820-a076-b8f9b78dffb0-tls-certs\") pod \"e2e-conflict-b-9d13b16c-kserve-5d69bbd584-l47wz\" (UID: \"4f929810-b049-4820-a076-b8f9b78dffb0\") " pod="e2e-models-e2e-worker-w2-e440bc/e2e-conflict-b-9d13b16c-kserve-5d69bbd584-l47wz" Nov 03 13:42:03.690635 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:03.690602 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5t7sr\" (UniqueName: \"kubernetes.io/projected/4f929810-b049-4820-a076-b8f9b78dffb0-kube-api-access-5t7sr\") pod \"e2e-conflict-b-9d13b16c-kserve-5d69bbd584-l47wz\" (UID: \"4f929810-b049-4820-a076-b8f9b78dffb0\") " pod="e2e-models-e2e-worker-w2-e440bc/e2e-conflict-b-9d13b16c-kserve-5d69bbd584-l47wz" Nov 03 13:42:03.855356 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:03.854669 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-e440bc/e2e-conflict-b-9d13b16c-kserve-5d69bbd584-l47wz" Nov 03 13:42:04.191997 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:04.191954 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-e440bc/e2e-conflict-b-9d13b16c-kserve-5d69bbd584-l47wz"] Nov 03 13:42:04.195126 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:42:04.195080 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4f929810_b049_4820_a076_b8f9b78dffb0.slice/crio-cc3cb8c1b32e87097e23d2a06475d040ad4fc6531b80b6c86c8336693f780e1c WatchSource:0}: Error finding container cc3cb8c1b32e87097e23d2a06475d040ad4fc6531b80b6c86c8336693f780e1c: Status 404 returned error can't find the container with id cc3cb8c1b32e87097e23d2a06475d040ad4fc6531b80b6c86c8336693f780e1c Nov 03 13:42:05.081920 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:05.081882 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-e440bc/e2e-conflict-b-9d13b16c-kserve-5d69bbd584-l47wz" event={"ID":"4f929810-b049-4820-a076-b8f9b78dffb0","Type":"ContainerStarted","Data":"20fb91a9c183c00b0721478b8f202fc75bd5dd014c0e1ddc6919abc54f8189ae"} Nov 03 13:42:05.082368 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:05.081933 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-e440bc/e2e-conflict-b-9d13b16c-kserve-5d69bbd584-l47wz" event={"ID":"4f929810-b049-4820-a076-b8f9b78dffb0","Type":"ContainerStarted","Data":"cc3cb8c1b32e87097e23d2a06475d040ad4fc6531b80b6c86c8336693f780e1c"} Nov 03 13:42:05.103493 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:05.103434 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-e440bc/e2e-conflict-b-9d13b16c-kserve-5d69bbd584-l47wz" podStartSLOduration=2.103421277 podStartE2EDuration="2.103421277s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:42:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:42:05.102459976 +0000 UTC m=+2005.110651253" watchObservedRunningTime="2026-11-03 13:42:05.103421277 +0000 UTC m=+2005.111612596" Nov 03 13:42:05.855356 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:05.855278 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-e440bc/e2e-conflict-b-9d13b16c-kserve-5d69bbd584-l47wz" Nov 03 13:42:09.511168 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:42:09.511069 5024 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d7mwz" containerName="extractor" Nov 03 13:42:13.855616 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:13.855576 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-e440bc/e2e-conflict-b-9d13b16c-kserve-5d69bbd584-l47wz" Nov 03 13:42:13.868617 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:13.868584 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-e440bc/e2e-conflict-b-9d13b16c-kserve-5d69bbd584-l47wz" Nov 03 13:42:14.180371 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:14.180260 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-e440bc/e2e-conflict-b-9d13b16c-kserve-5d69bbd584-l47wz" Nov 03 13:42:19.507998 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:19.507954 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-e440bc/e2e-conflict-b-9d13b16c-kserve-5d69bbd584-l47wz"] Nov 03 13:42:34.396585 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:34.396466 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-lftt4"] Nov 03 13:42:34.403512 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:34.403480 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-lftt4" Nov 03 13:42:34.428199 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:34.428162 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-lftt4"] Nov 03 13:42:34.541733 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:34.541659 5024 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-e440bc/e2e-conflict-b-9d13b16c-kserve-5d69bbd584-l47wz" podUID="4f929810-b049-4820-a076-b8f9b78dffb0" containerName="main" containerID="cri-o://20fb91a9c183c00b0721478b8f202fc75bd5dd014c0e1ddc6919abc54f8189ae" gracePeriod=45 Nov 03 13:42:34.544996 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:34.544929 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99af7f3f-6179-495a-a526-4cce51075588-credential-socket\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-lftt4\" (UID: \"99af7f3f-6179-495a-a526-4cce51075588\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-lftt4" Nov 03 13:42:34.545351 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:34.545321 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99af7f3f-6179-495a-a526-4cce51075588-istio-data\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-lftt4\" (UID: \"99af7f3f-6179-495a-a526-4cce51075588\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-lftt4" Nov 03 13:42:34.545737 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:34.545647 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-724pq\" (UniqueName: \"kubernetes.io/projected/99af7f3f-6179-495a-a526-4cce51075588-kube-api-access-724pq\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-lftt4\" (UID: \"99af7f3f-6179-495a-a526-4cce51075588\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-lftt4" Nov 03 13:42:34.545946 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:34.545889 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99af7f3f-6179-495a-a526-4cce51075588-workload-certs\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-lftt4\" (UID: \"99af7f3f-6179-495a-a526-4cce51075588\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-lftt4" Nov 03 13:42:34.546071 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:34.546055 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99af7f3f-6179-495a-a526-4cce51075588-istio-envoy\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-lftt4\" (UID: \"99af7f3f-6179-495a-a526-4cce51075588\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-lftt4" Nov 03 13:42:34.546193 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:34.546175 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99af7f3f-6179-495a-a526-4cce51075588-workload-socket\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-lftt4\" (UID: \"99af7f3f-6179-495a-a526-4cce51075588\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-lftt4" Nov 03 13:42:34.546350 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:34.546329 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99af7f3f-6179-495a-a526-4cce51075588-istiod-ca-cert\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-lftt4\" (UID: \"99af7f3f-6179-495a-a526-4cce51075588\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-lftt4" Nov 03 13:42:34.547910 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:34.547883 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99af7f3f-6179-495a-a526-4cce51075588-istio-podinfo\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-lftt4\" (UID: \"99af7f3f-6179-495a-a526-4cce51075588\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-lftt4" Nov 03 13:42:34.548132 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:34.548064 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99af7f3f-6179-495a-a526-4cce51075588-istio-token\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-lftt4\" (UID: \"99af7f3f-6179-495a-a526-4cce51075588\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-lftt4" Nov 03 13:42:34.650075 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:34.649970 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99af7f3f-6179-495a-a526-4cce51075588-istiod-ca-cert\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-lftt4\" (UID: \"99af7f3f-6179-495a-a526-4cce51075588\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-lftt4" Nov 03 13:42:34.650075 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:34.650059 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99af7f3f-6179-495a-a526-4cce51075588-istio-podinfo\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-lftt4\" (UID: \"99af7f3f-6179-495a-a526-4cce51075588\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-lftt4" Nov 03 13:42:34.650452 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:34.650094 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99af7f3f-6179-495a-a526-4cce51075588-istio-token\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-lftt4\" (UID: \"99af7f3f-6179-495a-a526-4cce51075588\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-lftt4" Nov 03 13:42:34.650452 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:34.650304 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99af7f3f-6179-495a-a526-4cce51075588-credential-socket\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-lftt4\" (UID: \"99af7f3f-6179-495a-a526-4cce51075588\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-lftt4" Nov 03 13:42:34.650452 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:34.650383 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99af7f3f-6179-495a-a526-4cce51075588-istio-data\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-lftt4\" (UID: \"99af7f3f-6179-495a-a526-4cce51075588\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-lftt4" Nov 03 13:42:34.650452 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:34.650415 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-724pq\" (UniqueName: \"kubernetes.io/projected/99af7f3f-6179-495a-a526-4cce51075588-kube-api-access-724pq\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-lftt4\" (UID: \"99af7f3f-6179-495a-a526-4cce51075588\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-lftt4" Nov 03 13:42:34.650809 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:34.650465 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99af7f3f-6179-495a-a526-4cce51075588-workload-certs\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-lftt4\" (UID: \"99af7f3f-6179-495a-a526-4cce51075588\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-lftt4" Nov 03 13:42:34.650809 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:34.650504 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99af7f3f-6179-495a-a526-4cce51075588-istio-envoy\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-lftt4\" (UID: \"99af7f3f-6179-495a-a526-4cce51075588\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-lftt4" Nov 03 13:42:34.650809 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:34.650533 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99af7f3f-6179-495a-a526-4cce51075588-workload-socket\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-lftt4\" (UID: \"99af7f3f-6179-495a-a526-4cce51075588\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-lftt4" Nov 03 13:42:34.650809 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:34.650712 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99af7f3f-6179-495a-a526-4cce51075588-credential-socket\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-lftt4\" (UID: \"99af7f3f-6179-495a-a526-4cce51075588\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-lftt4" Nov 03 13:42:34.650809 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:34.650757 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99af7f3f-6179-495a-a526-4cce51075588-istio-data\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-lftt4\" (UID: \"99af7f3f-6179-495a-a526-4cce51075588\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-lftt4" Nov 03 13:42:34.651047 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:34.650910 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99af7f3f-6179-495a-a526-4cce51075588-workload-certs\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-lftt4\" (UID: \"99af7f3f-6179-495a-a526-4cce51075588\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-lftt4" Nov 03 13:42:34.651047 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:34.650928 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99af7f3f-6179-495a-a526-4cce51075588-workload-socket\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-lftt4\" (UID: \"99af7f3f-6179-495a-a526-4cce51075588\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-lftt4" Nov 03 13:42:34.651335 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:34.651263 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99af7f3f-6179-495a-a526-4cce51075588-istiod-ca-cert\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-lftt4\" (UID: \"99af7f3f-6179-495a-a526-4cce51075588\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-lftt4" Nov 03 13:42:34.652952 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:34.652925 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99af7f3f-6179-495a-a526-4cce51075588-istio-envoy\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-lftt4\" (UID: \"99af7f3f-6179-495a-a526-4cce51075588\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-lftt4" Nov 03 13:42:34.652952 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:34.652948 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99af7f3f-6179-495a-a526-4cce51075588-istio-podinfo\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-lftt4\" (UID: \"99af7f3f-6179-495a-a526-4cce51075588\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-lftt4" Nov 03 13:42:34.659974 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:34.659948 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-724pq\" (UniqueName: \"kubernetes.io/projected/99af7f3f-6179-495a-a526-4cce51075588-kube-api-access-724pq\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-lftt4\" (UID: \"99af7f3f-6179-495a-a526-4cce51075588\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-lftt4" Nov 03 13:42:34.660085 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:34.659975 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99af7f3f-6179-495a-a526-4cce51075588-istio-token\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-lftt4\" (UID: \"99af7f3f-6179-495a-a526-4cce51075588\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-lftt4" Nov 03 13:42:34.727873 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:34.727836 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-lftt4" Nov 03 13:42:34.829352 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:34.829137 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-e440bc/e2e-conflict-b-9d13b16c-kserve-5d69bbd584-l47wz" Nov 03 13:42:34.959405 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:34.959270 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f929810-b049-4820-a076-b8f9b78dffb0-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4f929810-b049-4820-a076-b8f9b78dffb0-model-cache\") pod \"4f929810-b049-4820-a076-b8f9b78dffb0\" (UID: \"4f929810-b049-4820-a076-b8f9b78dffb0\") " Nov 03 13:42:34.959405 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:34.959385 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4f929810-b049-4820-a076-b8f9b78dffb0-tls-certs\" (UniqueName: \"kubernetes.io/secret/4f929810-b049-4820-a076-b8f9b78dffb0-tls-certs\") pod \"4f929810-b049-4820-a076-b8f9b78dffb0\" (UID: \"4f929810-b049-4820-a076-b8f9b78dffb0\") " Nov 03 13:42:34.959641 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:34.959415 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f929810-b049-4820-a076-b8f9b78dffb0-home\" (UniqueName: \"kubernetes.io/empty-dir/4f929810-b049-4820-a076-b8f9b78dffb0-home\") pod \"4f929810-b049-4820-a076-b8f9b78dffb0\" (UID: \"4f929810-b049-4820-a076-b8f9b78dffb0\") " Nov 03 13:42:34.959641 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:34.959469 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f929810-b049-4820-a076-b8f9b78dffb0-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f929810-b049-4820-a076-b8f9b78dffb0-tmp-dir\") pod \"4f929810-b049-4820-a076-b8f9b78dffb0\" (UID: \"4f929810-b049-4820-a076-b8f9b78dffb0\") " Nov 03 13:42:34.959641 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:34.959579 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f929810-b049-4820-a076-b8f9b78dffb0-model-cache" pod "4f929810-b049-4820-a076-b8f9b78dffb0" (UID: "4f929810-b049-4820-a076-b8f9b78dffb0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:42:34.959848 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:34.959636 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f929810-b049-4820-a076-b8f9b78dffb0-kube-api-access-5t7sr\" (UniqueName: \"kubernetes.io/projected/4f929810-b049-4820-a076-b8f9b78dffb0-kube-api-access-5t7sr\") pod \"4f929810-b049-4820-a076-b8f9b78dffb0\" (UID: \"4f929810-b049-4820-a076-b8f9b78dffb0\") " Nov 03 13:42:34.959848 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:34.959689 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f929810-b049-4820-a076-b8f9b78dffb0-dshm\" (UniqueName: \"kubernetes.io/empty-dir/4f929810-b049-4820-a076-b8f9b78dffb0-dshm\") pod \"4f929810-b049-4820-a076-b8f9b78dffb0\" (UID: \"4f929810-b049-4820-a076-b8f9b78dffb0\") " Nov 03 13:42:34.959848 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:34.959710 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f929810-b049-4820-a076-b8f9b78dffb0-home" pod "4f929810-b049-4820-a076-b8f9b78dffb0" (UID: "4f929810-b049-4820-a076-b8f9b78dffb0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:42:34.959848 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:34.959789 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f929810-b049-4820-a076-b8f9b78dffb0-tmp-dir" pod "4f929810-b049-4820-a076-b8f9b78dffb0" (UID: "4f929810-b049-4820-a076-b8f9b78dffb0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:42:34.960381 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:34.960354 5024 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4f929810-b049-4820-a076-b8f9b78dffb0-model-cache\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:42:34.960511 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:34.960384 5024 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4f929810-b049-4820-a076-b8f9b78dffb0-home\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:42:34.960511 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:34.960399 5024 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f929810-b049-4820-a076-b8f9b78dffb0-tmp-dir\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:42:34.961948 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:34.961921 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f929810-b049-4820-a076-b8f9b78dffb0-dshm" pod "4f929810-b049-4820-a076-b8f9b78dffb0" (UID: "4f929810-b049-4820-a076-b8f9b78dffb0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:42:34.962046 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:34.961970 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f929810-b049-4820-a076-b8f9b78dffb0-tls-certs" pod "4f929810-b049-4820-a076-b8f9b78dffb0" (UID: "4f929810-b049-4820-a076-b8f9b78dffb0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 13:42:34.962046 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:34.961980 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f929810-b049-4820-a076-b8f9b78dffb0-kube-api-access-5t7sr" pod "4f929810-b049-4820-a076-b8f9b78dffb0" (UID: "4f929810-b049-4820-a076-b8f9b78dffb0"). InnerVolumeSpecName "kube-api-access-5t7sr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:42:35.053425 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:35.053386 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-lftt4"] Nov 03 13:42:35.057562 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:42:35.057528 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99af7f3f_6179_495a_a526_4cce51075588.slice/crio-a2aafd7ddec973236d4d96dfc439d40c88144b6db061b1f298b48300a37fd6d2 WatchSource:0}: Error finding container a2aafd7ddec973236d4d96dfc439d40c88144b6db061b1f298b48300a37fd6d2: Status 404 returned error can't find the container with id a2aafd7ddec973236d4d96dfc439d40c88144b6db061b1f298b48300a37fd6d2 Nov 03 13:42:35.060900 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:35.060860 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:42:35.060987 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:35.060948 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:42:35.061024 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:35.060992 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:42:35.061974 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:35.061953 5024 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f929810-b049-4820-a076-b8f9b78dffb0-tls-certs\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:42:35.062047 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:35.061979 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5t7sr\" (UniqueName: \"kubernetes.io/projected/4f929810-b049-4820-a076-b8f9b78dffb0-kube-api-access-5t7sr\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:42:35.062047 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:35.061990 5024 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4f929810-b049-4820-a076-b8f9b78dffb0-dshm\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:42:35.438332 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:35.438270 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-lftt4" event={"ID":"99af7f3f-6179-495a-a526-4cce51075588","Type":"ContainerStarted","Data":"b2d7fc25419ec27517c330dbbbea8e457c53a3c0f1f4ed49c153f9b9bcfeb43d"} Nov 03 13:42:35.438805 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:35.438353 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-lftt4" event={"ID":"99af7f3f-6179-495a-a526-4cce51075588","Type":"ContainerStarted","Data":"a2aafd7ddec973236d4d96dfc439d40c88144b6db061b1f298b48300a37fd6d2"} Nov 03 13:42:35.440206 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:35.440180 5024 generic.go:356] "Generic (PLEG): container finished" podID="4f929810-b049-4820-a076-b8f9b78dffb0" containerID="20fb91a9c183c00b0721478b8f202fc75bd5dd014c0e1ddc6919abc54f8189ae" exitCode=0 Nov 03 13:42:35.440405 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:35.440230 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-e440bc/e2e-conflict-b-9d13b16c-kserve-5d69bbd584-l47wz" Nov 03 13:42:35.440405 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:35.440324 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-e440bc/e2e-conflict-b-9d13b16c-kserve-5d69bbd584-l47wz" event={"ID":"4f929810-b049-4820-a076-b8f9b78dffb0","Type":"ContainerDied","Data":"20fb91a9c183c00b0721478b8f202fc75bd5dd014c0e1ddc6919abc54f8189ae"} Nov 03 13:42:35.440405 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:35.440376 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-e440bc/e2e-conflict-b-9d13b16c-kserve-5d69bbd584-l47wz" event={"ID":"4f929810-b049-4820-a076-b8f9b78dffb0","Type":"ContainerDied","Data":"cc3cb8c1b32e87097e23d2a06475d040ad4fc6531b80b6c86c8336693f780e1c"} Nov 03 13:42:35.440405 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:35.440403 5024 scope.go:122] "RemoveContainer" containerID="20fb91a9c183c00b0721478b8f202fc75bd5dd014c0e1ddc6919abc54f8189ae" Nov 03 13:42:35.451888 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:35.451868 5024 scope.go:122] "RemoveContainer" containerID="20fb91a9c183c00b0721478b8f202fc75bd5dd014c0e1ddc6919abc54f8189ae" Nov 03 13:42:35.452496 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:42:35.452461 5024 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"20fb91a9c183c00b0721478b8f202fc75bd5dd014c0e1ddc6919abc54f8189ae\": container with ID starting with 20fb91a9c183c00b0721478b8f202fc75bd5dd014c0e1ddc6919abc54f8189ae not found: ID does not exist" containerID="20fb91a9c183c00b0721478b8f202fc75bd5dd014c0e1ddc6919abc54f8189ae" Nov 03 13:42:35.452606 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:35.452506 5024 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20fb91a9c183c00b0721478b8f202fc75bd5dd014c0e1ddc6919abc54f8189ae"} err="failed to get container status \"20fb91a9c183c00b0721478b8f202fc75bd5dd014c0e1ddc6919abc54f8189ae\": rpc error: code = NotFound desc = could not find container \"20fb91a9c183c00b0721478b8f202fc75bd5dd014c0e1ddc6919abc54f8189ae\": container with ID starting with 20fb91a9c183c00b0721478b8f202fc75bd5dd014c0e1ddc6919abc54f8189ae not found: ID does not exist" Nov 03 13:42:35.469782 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:35.469710 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-lftt4" podStartSLOduration=1.469691304 podStartE2EDuration="1.469691304s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:42:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:42:35.46582268 +0000 UTC m=+2035.474013957" watchObservedRunningTime="2026-11-03 13:42:35.469691304 +0000 UTC m=+2035.477882583" Nov 03 13:42:35.487480 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:35.487445 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-e440bc/e2e-conflict-b-9d13b16c-kserve-5d69bbd584-l47wz"] Nov 03 13:42:35.490598 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:35.490560 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-e440bc/e2e-conflict-b-9d13b16c-kserve-5d69bbd584-l47wz"] Nov 03 13:42:35.728603 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:35.728516 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-lftt4" Nov 03 13:42:36.110206 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:36.110158 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4f929810-b049-4820-a076-b8f9b78dffb0" path="/var/lib/kubelet/pods/4f929810-b049-4820-a076-b8f9b78dffb0/volumes" Nov 03 13:42:36.632052 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:36.630270 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-2cwvx"] Nov 03 13:42:36.632052 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:36.631211 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-cf696"] Nov 03 13:42:36.632052 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:36.631479 5024 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-cf696" podUID="2a69c6f3-da97-4a27-9809-951395987c7c" containerName="istio-proxy" containerID="cri-o://a3d9337fc97b523e5eb2e7cbb8c9477d8cb12da0c5c4f282784bc9d62b52283c" gracePeriod=30 Nov 03 13:42:36.632656 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:36.632264 5024 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-2cwvx" podUID="7d6b62e0-65bf-4766-93f4-b065c82304fb" containerName="istio-proxy" containerID="cri-o://061c2f55a0635f745abff9de665d06e2ca33a3f186426b5378c3654d6b59fe1a" gracePeriod=30 Nov 03 13:42:36.758501 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:36.758452 5024 patch_prober.go:29] interesting pod/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-lftt4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" start-of-body= Nov 03 13:42:36.758797 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:36.758762 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-lftt4" podUID="99af7f3f-6179-495a-a526-4cce51075588" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" Nov 03 13:42:37.728645 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:37.728603 5024 patch_prober.go:29] interesting pod/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-lftt4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" start-of-body= Nov 03 13:42:37.729052 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:37.728671 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-lftt4" podUID="99af7f3f-6179-495a-a526-4cce51075588" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" Nov 03 13:42:38.728527 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:38.728479 5024 patch_prober.go:29] interesting pod/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-lftt4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" start-of-body= Nov 03 13:42:38.728716 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:38.728541 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-lftt4" podUID="99af7f3f-6179-495a-a526-4cce51075588" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" Nov 03 13:42:39.529460 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:42:39.529408 5024 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d7mwz" containerName="extractor" Nov 03 13:42:39.729210 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:39.729169 5024 patch_prober.go:29] interesting pod/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-lftt4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" start-of-body= Nov 03 13:42:39.729719 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:39.729233 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-lftt4" podUID="99af7f3f-6179-495a-a526-4cce51075588" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" Nov 03 13:42:40.733807 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:40.733766 5024 patch_prober.go:29] interesting pod/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-lftt4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" start-of-body= Nov 03 13:42:40.734436 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:40.734398 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-lftt4" podUID="99af7f3f-6179-495a-a526-4cce51075588" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" Nov 03 13:42:41.729172 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:41.729129 5024 patch_prober.go:29] interesting pod/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-lftt4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" start-of-body= Nov 03 13:42:41.729528 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:41.729483 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-lftt4" podUID="99af7f3f-6179-495a-a526-4cce51075588" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" Nov 03 13:42:42.033711 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:42.033659 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-2cwvx" Nov 03 13:42:42.064709 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:42.064671 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-cf696" Nov 03 13:42:42.131308 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:42.130360 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a69c6f3-da97-4a27-9809-951395987c7c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a69c6f3-da97-4a27-9809-951395987c7c-workload-socket\") pod \"2a69c6f3-da97-4a27-9809-951395987c7c\" (UID: \"2a69c6f3-da97-4a27-9809-951395987c7c\") " Nov 03 13:42:42.131308 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:42.130424 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d6b62e0-65bf-4766-93f4-b065c82304fb-istio-token\" (UniqueName: \"kubernetes.io/projected/7d6b62e0-65bf-4766-93f4-b065c82304fb-istio-token\") pod \"7d6b62e0-65bf-4766-93f4-b065c82304fb\" (UID: \"7d6b62e0-65bf-4766-93f4-b065c82304fb\") " Nov 03 13:42:42.131308 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:42.130472 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a69c6f3-da97-4a27-9809-951395987c7c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a69c6f3-da97-4a27-9809-951395987c7c-istiod-ca-cert\") pod \"2a69c6f3-da97-4a27-9809-951395987c7c\" (UID: \"2a69c6f3-da97-4a27-9809-951395987c7c\") " Nov 03 13:42:42.131308 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:42.130591 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a69c6f3-da97-4a27-9809-951395987c7c-kube-api-access-9m9zw\" (UniqueName: \"kubernetes.io/projected/2a69c6f3-da97-4a27-9809-951395987c7c-kube-api-access-9m9zw\") pod \"2a69c6f3-da97-4a27-9809-951395987c7c\" (UID: \"2a69c6f3-da97-4a27-9809-951395987c7c\") " Nov 03 13:42:42.131308 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:42.130646 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d6b62e0-65bf-4766-93f4-b065c82304fb-kube-api-access-xsqv9\" (UniqueName: \"kubernetes.io/projected/7d6b62e0-65bf-4766-93f4-b065c82304fb-kube-api-access-xsqv9\") pod \"7d6b62e0-65bf-4766-93f4-b065c82304fb\" (UID: \"7d6b62e0-65bf-4766-93f4-b065c82304fb\") " Nov 03 13:42:42.131308 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:42.130683 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a69c6f3-da97-4a27-9809-951395987c7c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a69c6f3-da97-4a27-9809-951395987c7c-istio-envoy\") pod \"2a69c6f3-da97-4a27-9809-951395987c7c\" (UID: \"2a69c6f3-da97-4a27-9809-951395987c7c\") " Nov 03 13:42:42.131308 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:42.130757 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a69c6f3-da97-4a27-9809-951395987c7c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a69c6f3-da97-4a27-9809-951395987c7c-workload-certs\") pod \"2a69c6f3-da97-4a27-9809-951395987c7c\" (UID: \"2a69c6f3-da97-4a27-9809-951395987c7c\") " Nov 03 13:42:42.131308 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:42.130783 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d6b62e0-65bf-4766-93f4-b065c82304fb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d6b62e0-65bf-4766-93f4-b065c82304fb-workload-certs\") pod \"7d6b62e0-65bf-4766-93f4-b065c82304fb\" (UID: \"7d6b62e0-65bf-4766-93f4-b065c82304fb\") " Nov 03 13:42:42.131308 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:42.130816 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a69c6f3-da97-4a27-9809-951395987c7c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a69c6f3-da97-4a27-9809-951395987c7c-credential-socket\") pod \"2a69c6f3-da97-4a27-9809-951395987c7c\" (UID: \"2a69c6f3-da97-4a27-9809-951395987c7c\") " Nov 03 13:42:42.131308 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:42.130896 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d6b62e0-65bf-4766-93f4-b065c82304fb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d6b62e0-65bf-4766-93f4-b065c82304fb-workload-socket\") pod \"7d6b62e0-65bf-4766-93f4-b065c82304fb\" (UID: \"7d6b62e0-65bf-4766-93f4-b065c82304fb\") " Nov 03 13:42:42.131308 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:42.130940 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d6b62e0-65bf-4766-93f4-b065c82304fb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d6b62e0-65bf-4766-93f4-b065c82304fb-credential-socket\") pod \"7d6b62e0-65bf-4766-93f4-b065c82304fb\" (UID: \"7d6b62e0-65bf-4766-93f4-b065c82304fb\") " Nov 03 13:42:42.131308 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:42.131006 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a69c6f3-da97-4a27-9809-951395987c7c-istio-token\" (UniqueName: \"kubernetes.io/projected/2a69c6f3-da97-4a27-9809-951395987c7c-istio-token\") pod \"2a69c6f3-da97-4a27-9809-951395987c7c\" (UID: \"2a69c6f3-da97-4a27-9809-951395987c7c\") " Nov 03 13:42:42.131308 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:42.131071 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2a69c6f3-da97-4a27-9809-951395987c7c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a69c6f3-da97-4a27-9809-951395987c7c-istio-podinfo\") pod \"2a69c6f3-da97-4a27-9809-951395987c7c\" (UID: \"2a69c6f3-da97-4a27-9809-951395987c7c\") " Nov 03 13:42:42.131308 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:42.131103 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a69c6f3-da97-4a27-9809-951395987c7c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a69c6f3-da97-4a27-9809-951395987c7c-istio-data\") pod \"2a69c6f3-da97-4a27-9809-951395987c7c\" (UID: \"2a69c6f3-da97-4a27-9809-951395987c7c\") " Nov 03 13:42:42.131308 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:42.131147 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7d6b62e0-65bf-4766-93f4-b065c82304fb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d6b62e0-65bf-4766-93f4-b065c82304fb-istio-podinfo\") pod \"7d6b62e0-65bf-4766-93f4-b065c82304fb\" (UID: \"7d6b62e0-65bf-4766-93f4-b065c82304fb\") " Nov 03 13:42:42.131308 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:42.131212 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7d6b62e0-65bf-4766-93f4-b065c82304fb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d6b62e0-65bf-4766-93f4-b065c82304fb-istiod-ca-cert\") pod \"7d6b62e0-65bf-4766-93f4-b065c82304fb\" (UID: \"7d6b62e0-65bf-4766-93f4-b065c82304fb\") " Nov 03 13:42:42.131308 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:42.131246 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d6b62e0-65bf-4766-93f4-b065c82304fb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d6b62e0-65bf-4766-93f4-b065c82304fb-istio-data\") pod \"7d6b62e0-65bf-4766-93f4-b065c82304fb\" (UID: \"7d6b62e0-65bf-4766-93f4-b065c82304fb\") " Nov 03 13:42:42.131308 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:42.131322 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d6b62e0-65bf-4766-93f4-b065c82304fb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d6b62e0-65bf-4766-93f4-b065c82304fb-istio-envoy\") pod \"7d6b62e0-65bf-4766-93f4-b065c82304fb\" (UID: \"7d6b62e0-65bf-4766-93f4-b065c82304fb\") " Nov 03 13:42:42.137344 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:42.132607 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a69c6f3-da97-4a27-9809-951395987c7c-credential-socket" pod "2a69c6f3-da97-4a27-9809-951395987c7c" (UID: "2a69c6f3-da97-4a27-9809-951395987c7c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:42:42.137344 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:42.133154 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d6b62e0-65bf-4766-93f4-b065c82304fb-credential-socket" pod "7d6b62e0-65bf-4766-93f4-b065c82304fb" (UID: "7d6b62e0-65bf-4766-93f4-b065c82304fb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:42:42.137344 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:42.133437 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a69c6f3-da97-4a27-9809-951395987c7c-istiod-ca-cert" pod "2a69c6f3-da97-4a27-9809-951395987c7c" (UID: "2a69c6f3-da97-4a27-9809-951395987c7c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:42:42.137344 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:42.133935 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d6b62e0-65bf-4766-93f4-b065c82304fb-workload-socket" pod "7d6b62e0-65bf-4766-93f4-b065c82304fb" (UID: "7d6b62e0-65bf-4766-93f4-b065c82304fb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:42:42.137344 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:42.134129 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a69c6f3-da97-4a27-9809-951395987c7c-workload-certs" pod "2a69c6f3-da97-4a27-9809-951395987c7c" (UID: "2a69c6f3-da97-4a27-9809-951395987c7c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:42:42.137344 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:42.135716 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d6b62e0-65bf-4766-93f4-b065c82304fb-istio-envoy" pod "7d6b62e0-65bf-4766-93f4-b065c82304fb" (UID: "7d6b62e0-65bf-4766-93f4-b065c82304fb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:42:42.137344 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:42.136063 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d6b62e0-65bf-4766-93f4-b065c82304fb-istiod-ca-cert" pod "7d6b62e0-65bf-4766-93f4-b065c82304fb" (UID: "7d6b62e0-65bf-4766-93f4-b065c82304fb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:42:42.137344 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:42.136339 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d6b62e0-65bf-4766-93f4-b065c82304fb-istio-data" pod "7d6b62e0-65bf-4766-93f4-b065c82304fb" (UID: "7d6b62e0-65bf-4766-93f4-b065c82304fb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:42:42.137909 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:42.137376 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a69c6f3-da97-4a27-9809-951395987c7c-istio-data" pod "2a69c6f3-da97-4a27-9809-951395987c7c" (UID: "2a69c6f3-da97-4a27-9809-951395987c7c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:42:42.137909 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:42.137568 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a69c6f3-da97-4a27-9809-951395987c7c-workload-socket" pod "2a69c6f3-da97-4a27-9809-951395987c7c" (UID: "2a69c6f3-da97-4a27-9809-951395987c7c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:42:42.137909 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:42.137733 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d6b62e0-65bf-4766-93f4-b065c82304fb-workload-certs" pod "7d6b62e0-65bf-4766-93f4-b065c82304fb" (UID: "7d6b62e0-65bf-4766-93f4-b065c82304fb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:42:42.138483 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:42.138455 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a69c6f3-da97-4a27-9809-951395987c7c-istio-envoy" pod "2a69c6f3-da97-4a27-9809-951395987c7c" (UID: "2a69c6f3-da97-4a27-9809-951395987c7c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:42:42.138609 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:42.138590 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a69c6f3-da97-4a27-9809-951395987c7c-kube-api-access-9m9zw" pod "2a69c6f3-da97-4a27-9809-951395987c7c" (UID: "2a69c6f3-da97-4a27-9809-951395987c7c"). InnerVolumeSpecName "kube-api-access-9m9zw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:42:42.138670 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:42.138623 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a69c6f3-da97-4a27-9809-951395987c7c-istio-token" pod "2a69c6f3-da97-4a27-9809-951395987c7c" (UID: "2a69c6f3-da97-4a27-9809-951395987c7c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:42:42.138670 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:42.138635 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7d6b62e0-65bf-4766-93f4-b065c82304fb-istio-podinfo" pod "7d6b62e0-65bf-4766-93f4-b065c82304fb" (UID: "7d6b62e0-65bf-4766-93f4-b065c82304fb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 13:42:42.139474 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:42.139446 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d6b62e0-65bf-4766-93f4-b065c82304fb-istio-token" pod "7d6b62e0-65bf-4766-93f4-b065c82304fb" (UID: "7d6b62e0-65bf-4766-93f4-b065c82304fb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:42:42.139547 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:42.139472 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d6b62e0-65bf-4766-93f4-b065c82304fb-kube-api-access-xsqv9" pod "7d6b62e0-65bf-4766-93f4-b065c82304fb" (UID: "7d6b62e0-65bf-4766-93f4-b065c82304fb"). InnerVolumeSpecName "kube-api-access-xsqv9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:42:42.139643 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:42.139625 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2a69c6f3-da97-4a27-9809-951395987c7c-istio-podinfo" pod "2a69c6f3-da97-4a27-9809-951395987c7c" (UID: "2a69c6f3-da97-4a27-9809-951395987c7c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 13:42:42.245364 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:42.235586 5024 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a69c6f3-da97-4a27-9809-951395987c7c-istio-podinfo\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:42:42.245364 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:42.235635 5024 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a69c6f3-da97-4a27-9809-951395987c7c-istio-data\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:42:42.245364 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:42.235651 5024 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d6b62e0-65bf-4766-93f4-b065c82304fb-istio-podinfo\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:42:42.245364 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:42.235665 5024 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d6b62e0-65bf-4766-93f4-b065c82304fb-istiod-ca-cert\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:42:42.245364 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:42.235680 5024 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d6b62e0-65bf-4766-93f4-b065c82304fb-istio-data\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:42:42.245364 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:42.235758 5024 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d6b62e0-65bf-4766-93f4-b065c82304fb-istio-envoy\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:42:42.245364 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:42.235773 5024 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a69c6f3-da97-4a27-9809-951395987c7c-workload-socket\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:42:42.245364 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:42.235790 5024 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d6b62e0-65bf-4766-93f4-b065c82304fb-istio-token\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:42:42.245364 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:42.235804 5024 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a69c6f3-da97-4a27-9809-951395987c7c-istiod-ca-cert\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:42:42.245364 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:42.235819 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9m9zw\" (UniqueName: \"kubernetes.io/projected/2a69c6f3-da97-4a27-9809-951395987c7c-kube-api-access-9m9zw\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:42:42.245364 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:42.235836 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xsqv9\" (UniqueName: \"kubernetes.io/projected/7d6b62e0-65bf-4766-93f4-b065c82304fb-kube-api-access-xsqv9\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:42:42.245364 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:42.235852 5024 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a69c6f3-da97-4a27-9809-951395987c7c-istio-envoy\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:42:42.245364 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:42.235867 5024 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a69c6f3-da97-4a27-9809-951395987c7c-workload-certs\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:42:42.245364 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:42.235882 5024 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d6b62e0-65bf-4766-93f4-b065c82304fb-workload-certs\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:42:42.245364 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:42.235896 5024 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a69c6f3-da97-4a27-9809-951395987c7c-credential-socket\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:42:42.245364 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:42.235911 5024 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d6b62e0-65bf-4766-93f4-b065c82304fb-workload-socket\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:42:42.245364 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:42.235926 5024 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d6b62e0-65bf-4766-93f4-b065c82304fb-credential-socket\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:42:42.245364 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:42.235944 5024 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a69c6f3-da97-4a27-9809-951395987c7c-istio-token\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:42:42.529604 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:42.529505 5024 generic.go:356] "Generic (PLEG): container finished" podID="7d6b62e0-65bf-4766-93f4-b065c82304fb" containerID="061c2f55a0635f745abff9de665d06e2ca33a3f186426b5378c3654d6b59fe1a" exitCode=0 Nov 03 13:42:42.529604 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:42.529556 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-2cwvx" Nov 03 13:42:42.529604 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:42.529578 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-2cwvx" event={"ID":"7d6b62e0-65bf-4766-93f4-b065c82304fb","Type":"ContainerDied","Data":"061c2f55a0635f745abff9de665d06e2ca33a3f186426b5378c3654d6b59fe1a"} Nov 03 13:42:42.529905 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:42.529637 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-2cwvx" event={"ID":"7d6b62e0-65bf-4766-93f4-b065c82304fb","Type":"ContainerDied","Data":"bd4e9e461757e70a821cf35531e7dc146453cf960245647af328754bfbe705e7"} Nov 03 13:42:42.529905 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:42.529655 5024 scope.go:122] "RemoveContainer" containerID="061c2f55a0635f745abff9de665d06e2ca33a3f186426b5378c3654d6b59fe1a" Nov 03 13:42:42.538869 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:42.538638 5024 generic.go:356] "Generic (PLEG): container finished" podID="2a69c6f3-da97-4a27-9809-951395987c7c" containerID="a3d9337fc97b523e5eb2e7cbb8c9477d8cb12da0c5c4f282784bc9d62b52283c" exitCode=0 Nov 03 13:42:42.538869 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:42.538840 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-cf696" event={"ID":"2a69c6f3-da97-4a27-9809-951395987c7c","Type":"ContainerDied","Data":"a3d9337fc97b523e5eb2e7cbb8c9477d8cb12da0c5c4f282784bc9d62b52283c"} Nov 03 13:42:42.539099 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:42.538882 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-cf696" event={"ID":"2a69c6f3-da97-4a27-9809-951395987c7c","Type":"ContainerDied","Data":"eac38ca0c29438663a7b42c7c715a6b74b76043efff869d90f12db8aa03210df"} Nov 03 13:42:42.539099 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:42.538921 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-cf696" Nov 03 13:42:42.544482 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:42.544403 5024 scope.go:122] "RemoveContainer" containerID="3eed62c34bb403ff209a40eeeb098257d766a8681440034ee7d547e8b364ce98" Nov 03 13:42:42.558018 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:42.557972 5024 scope.go:122] "RemoveContainer" containerID="061c2f55a0635f745abff9de665d06e2ca33a3f186426b5378c3654d6b59fe1a" Nov 03 13:42:42.558399 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:42:42.558358 5024 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"061c2f55a0635f745abff9de665d06e2ca33a3f186426b5378c3654d6b59fe1a\": container with ID starting with 061c2f55a0635f745abff9de665d06e2ca33a3f186426b5378c3654d6b59fe1a not found: ID does not exist" containerID="061c2f55a0635f745abff9de665d06e2ca33a3f186426b5378c3654d6b59fe1a" Nov 03 13:42:42.558512 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:42.558414 5024 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"061c2f55a0635f745abff9de665d06e2ca33a3f186426b5378c3654d6b59fe1a"} err="failed to get container status \"061c2f55a0635f745abff9de665d06e2ca33a3f186426b5378c3654d6b59fe1a\": rpc error: code = NotFound desc = could not find container \"061c2f55a0635f745abff9de665d06e2ca33a3f186426b5378c3654d6b59fe1a\": container with ID starting with 061c2f55a0635f745abff9de665d06e2ca33a3f186426b5378c3654d6b59fe1a not found: ID does not exist" Nov 03 13:42:42.558512 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:42.558442 5024 scope.go:122] "RemoveContainer" containerID="3eed62c34bb403ff209a40eeeb098257d766a8681440034ee7d547e8b364ce98" Nov 03 13:42:42.558792 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:42:42.558770 5024 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3eed62c34bb403ff209a40eeeb098257d766a8681440034ee7d547e8b364ce98\": container with ID starting with 3eed62c34bb403ff209a40eeeb098257d766a8681440034ee7d547e8b364ce98 not found: ID does not exist" containerID="3eed62c34bb403ff209a40eeeb098257d766a8681440034ee7d547e8b364ce98" Nov 03 13:42:42.558852 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:42.558799 5024 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3eed62c34bb403ff209a40eeeb098257d766a8681440034ee7d547e8b364ce98"} err="failed to get container status \"3eed62c34bb403ff209a40eeeb098257d766a8681440034ee7d547e8b364ce98\": rpc error: code = NotFound desc = could not find container \"3eed62c34bb403ff209a40eeeb098257d766a8681440034ee7d547e8b364ce98\": container with ID starting with 3eed62c34bb403ff209a40eeeb098257d766a8681440034ee7d547e8b364ce98 not found: ID does not exist" Nov 03 13:42:42.558852 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:42.558818 5024 scope.go:122] "RemoveContainer" containerID="a3d9337fc97b523e5eb2e7cbb8c9477d8cb12da0c5c4f282784bc9d62b52283c" Nov 03 13:42:42.572467 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:42.572427 5024 scope.go:122] "RemoveContainer" containerID="a3d9337fc97b523e5eb2e7cbb8c9477d8cb12da0c5c4f282784bc9d62b52283c" Nov 03 13:42:42.572907 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:42:42.572879 5024 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a3d9337fc97b523e5eb2e7cbb8c9477d8cb12da0c5c4f282784bc9d62b52283c\": container with ID starting with a3d9337fc97b523e5eb2e7cbb8c9477d8cb12da0c5c4f282784bc9d62b52283c not found: ID does not exist" containerID="a3d9337fc97b523e5eb2e7cbb8c9477d8cb12da0c5c4f282784bc9d62b52283c" Nov 03 13:42:42.573019 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:42.572915 5024 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3d9337fc97b523e5eb2e7cbb8c9477d8cb12da0c5c4f282784bc9d62b52283c"} err="failed to get container status \"a3d9337fc97b523e5eb2e7cbb8c9477d8cb12da0c5c4f282784bc9d62b52283c\": rpc error: code = NotFound desc = could not find container \"a3d9337fc97b523e5eb2e7cbb8c9477d8cb12da0c5c4f282784bc9d62b52283c\": container with ID starting with a3d9337fc97b523e5eb2e7cbb8c9477d8cb12da0c5c4f282784bc9d62b52283c not found: ID does not exist" Nov 03 13:42:42.574397 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:42.574375 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-2cwvx"] Nov 03 13:42:42.577662 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:42.577634 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-2cwvx"] Nov 03 13:42:42.607303 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:42.605657 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-cf696"] Nov 03 13:42:42.623231 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:42.623185 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-a724a6-openshift-default-876f8db6f-cf696"] Nov 03 13:42:43.523739 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:43.523703 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-lftt4" Nov 03 13:42:43.525814 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:43.525789 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-lftt4" Nov 03 13:42:43.536335 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:43.534436 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-lftt4" Nov 03 13:42:44.122338 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:44.121725 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a69c6f3-da97-4a27-9809-951395987c7c" path="/var/lib/kubelet/pods/2a69c6f3-da97-4a27-9809-951395987c7c/volumes" Nov 03 13:42:44.125065 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:44.125026 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d6b62e0-65bf-4766-93f4-b065c82304fb" path="/var/lib/kubelet/pods/7d6b62e0-65bf-4766-93f4-b065c82304fb/volumes" Nov 03 13:42:49.359341 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:49.350230 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-zpj4t"] Nov 03 13:42:49.359341 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:49.351069 5024 cpu_manager.go:414] "containerMap: removing container" podUID="2a69c6f3-da97-4a27-9809-951395987c7c" containerName="istio-proxy" Nov 03 13:42:49.359341 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:49.351086 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a69c6f3-da97-4a27-9809-951395987c7c" containerName="istio-proxy" Nov 03 13:42:49.359341 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:49.351126 5024 cpu_manager.go:414] "containerMap: removing container" podUID="4f929810-b049-4820-a076-b8f9b78dffb0" containerName="main" Nov 03 13:42:49.359341 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:49.351134 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f929810-b049-4820-a076-b8f9b78dffb0" containerName="main" Nov 03 13:42:49.359341 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:49.351147 5024 cpu_manager.go:414] "containerMap: removing container" podUID="7d6b62e0-65bf-4766-93f4-b065c82304fb" containerName="istio-proxy" Nov 03 13:42:49.359341 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:49.351155 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d6b62e0-65bf-4766-93f4-b065c82304fb" containerName="istio-proxy" Nov 03 13:42:49.359341 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:49.351178 5024 cpu_manager.go:414] "containerMap: removing container" podUID="7d6b62e0-65bf-4766-93f4-b065c82304fb" containerName="istio-proxy" Nov 03 13:42:49.359341 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:49.351184 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d6b62e0-65bf-4766-93f4-b065c82304fb" containerName="istio-proxy" Nov 03 13:42:49.366603 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:49.363526 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a69c6f3-da97-4a27-9809-951395987c7c" containerName="istio-proxy" Nov 03 13:42:49.366603 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:49.363570 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f929810-b049-4820-a076-b8f9b78dffb0" containerName="main" Nov 03 13:42:49.366603 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:49.363582 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d6b62e0-65bf-4766-93f4-b065c82304fb" containerName="istio-proxy" Nov 03 13:42:49.366603 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:49.363599 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d6b62e0-65bf-4766-93f4-b065c82304fb" containerName="istio-proxy" Nov 03 13:42:49.371474 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:49.370438 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-zpj4t" Nov 03 13:42:49.393560 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:49.393499 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-zpj4t"] Nov 03 13:42:49.449127 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:49.449084 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95aa021f-6ec9-4f53-810a-58eb1d059081-workload-socket\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-zpj4t\" (UID: \"95aa021f-6ec9-4f53-810a-58eb1d059081\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-zpj4t" Nov 03 13:42:49.449368 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:49.449138 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95aa021f-6ec9-4f53-810a-58eb1d059081-istiod-ca-cert\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-zpj4t\" (UID: \"95aa021f-6ec9-4f53-810a-58eb1d059081\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-zpj4t" Nov 03 13:42:49.449368 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:49.449169 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/95aa021f-6ec9-4f53-810a-58eb1d059081-istio-token\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-zpj4t\" (UID: \"95aa021f-6ec9-4f53-810a-58eb1d059081\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-zpj4t" Nov 03 13:42:49.449368 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:49.449195 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95aa021f-6ec9-4f53-810a-58eb1d059081-istio-envoy\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-zpj4t\" (UID: \"95aa021f-6ec9-4f53-810a-58eb1d059081\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-zpj4t" Nov 03 13:42:49.449368 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:49.449277 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5kbrh\" (UniqueName: \"kubernetes.io/projected/95aa021f-6ec9-4f53-810a-58eb1d059081-kube-api-access-5kbrh\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-zpj4t\" (UID: \"95aa021f-6ec9-4f53-810a-58eb1d059081\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-zpj4t" Nov 03 13:42:49.449368 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:49.449313 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95aa021f-6ec9-4f53-810a-58eb1d059081-workload-certs\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-zpj4t\" (UID: \"95aa021f-6ec9-4f53-810a-58eb1d059081\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-zpj4t" Nov 03 13:42:49.449677 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:49.449377 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95aa021f-6ec9-4f53-810a-58eb1d059081-credential-socket\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-zpj4t\" (UID: \"95aa021f-6ec9-4f53-810a-58eb1d059081\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-zpj4t" Nov 03 13:42:49.449677 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:49.449396 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95aa021f-6ec9-4f53-810a-58eb1d059081-istio-data\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-zpj4t\" (UID: \"95aa021f-6ec9-4f53-810a-58eb1d059081\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-zpj4t" Nov 03 13:42:49.449677 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:49.449414 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95aa021f-6ec9-4f53-810a-58eb1d059081-istio-podinfo\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-zpj4t\" (UID: \"95aa021f-6ec9-4f53-810a-58eb1d059081\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-zpj4t" Nov 03 13:42:49.550803 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:49.550746 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/95aa021f-6ec9-4f53-810a-58eb1d059081-istio-token\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-zpj4t\" (UID: \"95aa021f-6ec9-4f53-810a-58eb1d059081\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-zpj4t" Nov 03 13:42:49.551025 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:49.550819 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95aa021f-6ec9-4f53-810a-58eb1d059081-istio-envoy\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-zpj4t\" (UID: \"95aa021f-6ec9-4f53-810a-58eb1d059081\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-zpj4t" Nov 03 13:42:49.551025 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:49.550941 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5kbrh\" (UniqueName: \"kubernetes.io/projected/95aa021f-6ec9-4f53-810a-58eb1d059081-kube-api-access-5kbrh\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-zpj4t\" (UID: \"95aa021f-6ec9-4f53-810a-58eb1d059081\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-zpj4t" Nov 03 13:42:49.551025 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:49.550968 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95aa021f-6ec9-4f53-810a-58eb1d059081-workload-certs\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-zpj4t\" (UID: \"95aa021f-6ec9-4f53-810a-58eb1d059081\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-zpj4t" Nov 03 13:42:49.551250 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:49.551069 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95aa021f-6ec9-4f53-810a-58eb1d059081-credential-socket\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-zpj4t\" (UID: \"95aa021f-6ec9-4f53-810a-58eb1d059081\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-zpj4t" Nov 03 13:42:49.551250 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:49.551125 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95aa021f-6ec9-4f53-810a-58eb1d059081-istio-data\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-zpj4t\" (UID: \"95aa021f-6ec9-4f53-810a-58eb1d059081\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-zpj4t" Nov 03 13:42:49.551250 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:49.551174 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95aa021f-6ec9-4f53-810a-58eb1d059081-istio-podinfo\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-zpj4t\" (UID: \"95aa021f-6ec9-4f53-810a-58eb1d059081\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-zpj4t" Nov 03 13:42:49.551439 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:49.551353 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95aa021f-6ec9-4f53-810a-58eb1d059081-workload-socket\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-zpj4t\" (UID: \"95aa021f-6ec9-4f53-810a-58eb1d059081\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-zpj4t" Nov 03 13:42:49.551439 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:49.551388 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95aa021f-6ec9-4f53-810a-58eb1d059081-istiod-ca-cert\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-zpj4t\" (UID: \"95aa021f-6ec9-4f53-810a-58eb1d059081\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-zpj4t" Nov 03 13:42:49.551539 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:49.551503 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95aa021f-6ec9-4f53-810a-58eb1d059081-credential-socket\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-zpj4t\" (UID: \"95aa021f-6ec9-4f53-810a-58eb1d059081\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-zpj4t" Nov 03 13:42:49.551597 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:49.551540 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95aa021f-6ec9-4f53-810a-58eb1d059081-istio-data\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-zpj4t\" (UID: \"95aa021f-6ec9-4f53-810a-58eb1d059081\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-zpj4t" Nov 03 13:42:49.551806 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:49.551784 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95aa021f-6ec9-4f53-810a-58eb1d059081-workload-socket\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-zpj4t\" (UID: \"95aa021f-6ec9-4f53-810a-58eb1d059081\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-zpj4t" Nov 03 13:42:49.551915 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:49.551867 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95aa021f-6ec9-4f53-810a-58eb1d059081-workload-certs\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-zpj4t\" (UID: \"95aa021f-6ec9-4f53-810a-58eb1d059081\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-zpj4t" Nov 03 13:42:49.552118 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:49.552099 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95aa021f-6ec9-4f53-810a-58eb1d059081-istiod-ca-cert\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-zpj4t\" (UID: \"95aa021f-6ec9-4f53-810a-58eb1d059081\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-zpj4t" Nov 03 13:42:49.553785 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:49.553753 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95aa021f-6ec9-4f53-810a-58eb1d059081-istio-envoy\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-zpj4t\" (UID: \"95aa021f-6ec9-4f53-810a-58eb1d059081\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-zpj4t" Nov 03 13:42:49.554455 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:49.554428 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95aa021f-6ec9-4f53-810a-58eb1d059081-istio-podinfo\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-zpj4t\" (UID: \"95aa021f-6ec9-4f53-810a-58eb1d059081\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-zpj4t" Nov 03 13:42:49.560684 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:49.560654 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/95aa021f-6ec9-4f53-810a-58eb1d059081-istio-token\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-zpj4t\" (UID: \"95aa021f-6ec9-4f53-810a-58eb1d059081\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-zpj4t" Nov 03 13:42:49.561181 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:49.561148 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5kbrh\" (UniqueName: \"kubernetes.io/projected/95aa021f-6ec9-4f53-810a-58eb1d059081-kube-api-access-5kbrh\") pod \"e2e-worker-w2-e440bc-openshift-default-d56bb6b89-zpj4t\" (UID: \"95aa021f-6ec9-4f53-810a-58eb1d059081\") " pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-zpj4t" Nov 03 13:42:49.695660 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:49.695571 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-zpj4t" Nov 03 13:42:50.108703 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:42:50.108659 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95aa021f_6ec9_4f53_810a_58eb1d059081.slice/crio-0678890b4999b49b89178ef1a43c7d8eb6da957935bb788eb9ebe183fe0d7fba WatchSource:0}: Error finding container 0678890b4999b49b89178ef1a43c7d8eb6da957935bb788eb9ebe183fe0d7fba: Status 404 returned error can't find the container with id 0678890b4999b49b89178ef1a43c7d8eb6da957935bb788eb9ebe183fe0d7fba Nov 03 13:42:50.112686 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:50.112652 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:42:50.112796 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:50.112721 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:42:50.112796 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:50.112750 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:42:50.112868 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:50.112829 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-zpj4t"] Nov 03 13:42:50.654616 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:50.654567 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-zpj4t" event={"ID":"95aa021f-6ec9-4f53-810a-58eb1d059081","Type":"ContainerStarted","Data":"1065c59e4d759c691604c43c01c5947c75b0be37a4d1303512593a72cf489045"} Nov 03 13:42:50.654994 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:50.654640 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-zpj4t" event={"ID":"95aa021f-6ec9-4f53-810a-58eb1d059081","Type":"ContainerStarted","Data":"0678890b4999b49b89178ef1a43c7d8eb6da957935bb788eb9ebe183fe0d7fba"} Nov 03 13:42:50.679311 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:50.677753 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-zpj4t" podStartSLOduration=1.6777336489999999 podStartE2EDuration="1.677733649s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:42:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:42:50.676813019 +0000 UTC m=+2050.685004297" watchObservedRunningTime="2026-11-03 13:42:50.677733649 +0000 UTC m=+2050.685924928" Nov 03 13:42:50.696477 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:50.696393 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-zpj4t" Nov 03 13:42:51.703636 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:51.703571 5024 patch_prober.go:29] interesting pod/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-zpj4t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" start-of-body= Nov 03 13:42:51.704329 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:51.704271 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-zpj4t" podUID="95aa021f-6ec9-4f53-810a-58eb1d059081" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" Nov 03 13:42:52.698747 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:52.698703 5024 patch_prober.go:29] interesting pod/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-zpj4t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" start-of-body= Nov 03 13:42:52.699056 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:52.699024 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-zpj4t" podUID="95aa021f-6ec9-4f53-810a-58eb1d059081" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" Nov 03 13:42:52.720195 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:52.720156 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-7h97f"] Nov 03 13:42:52.721047 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:52.721008 5024 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-7h97f" podUID="823a705e-26f9-422c-83b0-75dc73349072" containerName="istio-proxy" containerID="cri-o://58dc34962e683ed7b74140e17f6067eba9b1720fb11f3d51cfc0d746e336670d" gracePeriod=30 Nov 03 13:42:52.730748 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:52.730719 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-pznlh"] Nov 03 13:42:52.731307 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:52.731264 5024 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-pznlh" podUID="0757da43-44aa-48e1-b169-00715a78ea02" containerName="istio-proxy" containerID="cri-o://1cd0f17dbeb3270cfe79f3934f9fcbc3b4612cdc3a703cfabbc47c47e6cc1815" gracePeriod=30 Nov 03 13:42:53.696495 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:53.696008 5024 patch_prober.go:29] interesting pod/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-zpj4t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" start-of-body= Nov 03 13:42:53.696495 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:53.696088 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-zpj4t" podUID="95aa021f-6ec9-4f53-810a-58eb1d059081" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" Nov 03 13:42:54.701310 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:54.700808 5024 patch_prober.go:29] interesting pod/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-zpj4t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" start-of-body= Nov 03 13:42:54.701310 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:54.700875 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-zpj4t" podUID="95aa021f-6ec9-4f53-810a-58eb1d059081" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" Nov 03 13:42:55.696828 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:55.696783 5024 patch_prober.go:29] interesting pod/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-zpj4t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" start-of-body= Nov 03 13:42:55.697099 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:55.697070 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-zpj4t" podUID="95aa021f-6ec9-4f53-810a-58eb1d059081" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" Nov 03 13:42:55.862224 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:55.862174 5024 patch_prober.go:29] interesting pod/e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-pznlh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 13:42:55.862653 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:55.862237 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-pznlh" podUID="0757da43-44aa-48e1-b169-00715a78ea02" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 13:42:56.696792 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:56.696738 5024 patch_prober.go:29] interesting pod/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-zpj4t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" start-of-body= Nov 03 13:42:56.697018 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:56.696809 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-zpj4t" podUID="95aa021f-6ec9-4f53-810a-58eb1d059081" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" Nov 03 13:42:57.702106 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:57.702056 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-zpj4t" Nov 03 13:42:57.702711 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:57.702597 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-zpj4t" Nov 03 13:42:57.703705 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:57.703680 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-zpj4t" Nov 03 13:42:57.941421 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:57.941044 5024 patch_prober.go:29] interesting pod/e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-7h97f container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" start-of-body= Nov 03 13:42:57.941421 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:57.941098 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-7h97f" podUID="823a705e-26f9-422c-83b0-75dc73349072" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" Nov 03 13:42:58.077200 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:58.077158 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-pznlh" Nov 03 13:42:58.172969 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:58.172932 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-7h97f" Nov 03 13:42:58.234587 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:58.234482 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0757da43-44aa-48e1-b169-00715a78ea02-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0757da43-44aa-48e1-b169-00715a78ea02-istio-envoy\") pod \"0757da43-44aa-48e1-b169-00715a78ea02\" (UID: \"0757da43-44aa-48e1-b169-00715a78ea02\") " Nov 03 13:42:58.234914 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:58.234875 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0757da43-44aa-48e1-b169-00715a78ea02-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0757da43-44aa-48e1-b169-00715a78ea02-workload-certs\") pod \"0757da43-44aa-48e1-b169-00715a78ea02\" (UID: \"0757da43-44aa-48e1-b169-00715a78ea02\") " Nov 03 13:42:58.235088 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:58.235065 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0757da43-44aa-48e1-b169-00715a78ea02-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0757da43-44aa-48e1-b169-00715a78ea02-istio-podinfo\") pod \"0757da43-44aa-48e1-b169-00715a78ea02\" (UID: \"0757da43-44aa-48e1-b169-00715a78ea02\") " Nov 03 13:42:58.235208 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:58.235162 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0757da43-44aa-48e1-b169-00715a78ea02-workload-certs" pod "0757da43-44aa-48e1-b169-00715a78ea02" (UID: "0757da43-44aa-48e1-b169-00715a78ea02"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:42:58.235681 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:58.235657 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0757da43-44aa-48e1-b169-00715a78ea02-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0757da43-44aa-48e1-b169-00715a78ea02-credential-socket\") pod \"0757da43-44aa-48e1-b169-00715a78ea02\" (UID: \"0757da43-44aa-48e1-b169-00715a78ea02\") " Nov 03 13:42:58.235950 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:58.235927 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0757da43-44aa-48e1-b169-00715a78ea02-credential-socket" pod "0757da43-44aa-48e1-b169-00715a78ea02" (UID: "0757da43-44aa-48e1-b169-00715a78ea02"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:42:58.236042 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:58.236004 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0757da43-44aa-48e1-b169-00715a78ea02-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0757da43-44aa-48e1-b169-00715a78ea02-workload-socket\") pod \"0757da43-44aa-48e1-b169-00715a78ea02\" (UID: \"0757da43-44aa-48e1-b169-00715a78ea02\") " Nov 03 13:42:58.236261 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:58.236237 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0757da43-44aa-48e1-b169-00715a78ea02-workload-socket" pod "0757da43-44aa-48e1-b169-00715a78ea02" (UID: "0757da43-44aa-48e1-b169-00715a78ea02"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:42:58.236364 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:58.236344 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0757da43-44aa-48e1-b169-00715a78ea02-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0757da43-44aa-48e1-b169-00715a78ea02-istiod-ca-cert\") pod \"0757da43-44aa-48e1-b169-00715a78ea02\" (UID: \"0757da43-44aa-48e1-b169-00715a78ea02\") " Nov 03 13:42:58.236762 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:58.236684 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0757da43-44aa-48e1-b169-00715a78ea02-istiod-ca-cert" pod "0757da43-44aa-48e1-b169-00715a78ea02" (UID: "0757da43-44aa-48e1-b169-00715a78ea02"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:42:58.236847 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:58.236788 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0757da43-44aa-48e1-b169-00715a78ea02-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0757da43-44aa-48e1-b169-00715a78ea02-istio-data\") pod \"0757da43-44aa-48e1-b169-00715a78ea02\" (UID: \"0757da43-44aa-48e1-b169-00715a78ea02\") " Nov 03 13:42:58.237528 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:58.237499 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0757da43-44aa-48e1-b169-00715a78ea02-istio-envoy" pod "0757da43-44aa-48e1-b169-00715a78ea02" (UID: "0757da43-44aa-48e1-b169-00715a78ea02"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:42:58.237528 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:58.237516 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0757da43-44aa-48e1-b169-00715a78ea02-istio-data" pod "0757da43-44aa-48e1-b169-00715a78ea02" (UID: "0757da43-44aa-48e1-b169-00715a78ea02"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:42:58.237681 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:58.237544 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0757da43-44aa-48e1-b169-00715a78ea02-istio-podinfo" pod "0757da43-44aa-48e1-b169-00715a78ea02" (UID: "0757da43-44aa-48e1-b169-00715a78ea02"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 13:42:58.237681 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:58.236820 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0757da43-44aa-48e1-b169-00715a78ea02-istio-token\" (UniqueName: \"kubernetes.io/projected/0757da43-44aa-48e1-b169-00715a78ea02-istio-token\") pod \"0757da43-44aa-48e1-b169-00715a78ea02\" (UID: \"0757da43-44aa-48e1-b169-00715a78ea02\") " Nov 03 13:42:58.237681 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:58.237635 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0757da43-44aa-48e1-b169-00715a78ea02-kube-api-access-zw559\" (UniqueName: \"kubernetes.io/projected/0757da43-44aa-48e1-b169-00715a78ea02-kube-api-access-zw559\") pod \"0757da43-44aa-48e1-b169-00715a78ea02\" (UID: \"0757da43-44aa-48e1-b169-00715a78ea02\") " Nov 03 13:42:58.238329 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:58.238301 5024 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0757da43-44aa-48e1-b169-00715a78ea02-istio-envoy\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:42:58.238425 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:58.238333 5024 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0757da43-44aa-48e1-b169-00715a78ea02-workload-certs\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:42:58.238425 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:58.238349 5024 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0757da43-44aa-48e1-b169-00715a78ea02-istio-podinfo\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:42:58.238425 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:58.238363 5024 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0757da43-44aa-48e1-b169-00715a78ea02-credential-socket\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:42:58.238425 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:58.238376 5024 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0757da43-44aa-48e1-b169-00715a78ea02-workload-socket\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:42:58.238425 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:58.238390 5024 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0757da43-44aa-48e1-b169-00715a78ea02-istiod-ca-cert\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:42:58.238425 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:58.238404 5024 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0757da43-44aa-48e1-b169-00715a78ea02-istio-data\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:42:58.239473 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:58.239443 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0757da43-44aa-48e1-b169-00715a78ea02-istio-token" pod "0757da43-44aa-48e1-b169-00715a78ea02" (UID: "0757da43-44aa-48e1-b169-00715a78ea02"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:42:58.239944 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:58.239927 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0757da43-44aa-48e1-b169-00715a78ea02-kube-api-access-zw559" pod "0757da43-44aa-48e1-b169-00715a78ea02" (UID: "0757da43-44aa-48e1-b169-00715a78ea02"). InnerVolumeSpecName "kube-api-access-zw559". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:42:58.339160 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:58.339118 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/823a705e-26f9-422c-83b0-75dc73349072-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/823a705e-26f9-422c-83b0-75dc73349072-istio-data\") pod \"823a705e-26f9-422c-83b0-75dc73349072\" (UID: \"823a705e-26f9-422c-83b0-75dc73349072\") " Nov 03 13:42:58.339398 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:58.339176 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/823a705e-26f9-422c-83b0-75dc73349072-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/823a705e-26f9-422c-83b0-75dc73349072-credential-socket\") pod \"823a705e-26f9-422c-83b0-75dc73349072\" (UID: \"823a705e-26f9-422c-83b0-75dc73349072\") " Nov 03 13:42:58.339398 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:58.339260 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/823a705e-26f9-422c-83b0-75dc73349072-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/823a705e-26f9-422c-83b0-75dc73349072-workload-certs\") pod \"823a705e-26f9-422c-83b0-75dc73349072\" (UID: \"823a705e-26f9-422c-83b0-75dc73349072\") " Nov 03 13:42:58.339398 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:58.339340 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/823a705e-26f9-422c-83b0-75dc73349072-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/823a705e-26f9-422c-83b0-75dc73349072-istiod-ca-cert\") pod \"823a705e-26f9-422c-83b0-75dc73349072\" (UID: \"823a705e-26f9-422c-83b0-75dc73349072\") " Nov 03 13:42:58.339586 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:58.339401 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/823a705e-26f9-422c-83b0-75dc73349072-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/823a705e-26f9-422c-83b0-75dc73349072-workload-socket\") pod \"823a705e-26f9-422c-83b0-75dc73349072\" (UID: \"823a705e-26f9-422c-83b0-75dc73349072\") " Nov 03 13:42:58.339586 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:58.339455 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/823a705e-26f9-422c-83b0-75dc73349072-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/823a705e-26f9-422c-83b0-75dc73349072-istio-envoy\") pod \"823a705e-26f9-422c-83b0-75dc73349072\" (UID: \"823a705e-26f9-422c-83b0-75dc73349072\") " Nov 03 13:42:58.339586 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:58.339468 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/823a705e-26f9-422c-83b0-75dc73349072-istio-data" pod "823a705e-26f9-422c-83b0-75dc73349072" (UID: "823a705e-26f9-422c-83b0-75dc73349072"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:42:58.339586 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:58.339484 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/823a705e-26f9-422c-83b0-75dc73349072-credential-socket" pod "823a705e-26f9-422c-83b0-75dc73349072" (UID: "823a705e-26f9-422c-83b0-75dc73349072"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:42:58.339586 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:58.339537 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/823a705e-26f9-422c-83b0-75dc73349072-workload-certs" pod "823a705e-26f9-422c-83b0-75dc73349072" (UID: "823a705e-26f9-422c-83b0-75dc73349072"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:42:58.339809 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:58.339667 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/823a705e-26f9-422c-83b0-75dc73349072-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/823a705e-26f9-422c-83b0-75dc73349072-istio-podinfo\") pod \"823a705e-26f9-422c-83b0-75dc73349072\" (UID: \"823a705e-26f9-422c-83b0-75dc73349072\") " Nov 03 13:42:58.339809 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:58.339685 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/823a705e-26f9-422c-83b0-75dc73349072-workload-socket" pod "823a705e-26f9-422c-83b0-75dc73349072" (UID: "823a705e-26f9-422c-83b0-75dc73349072"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:42:58.339809 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:58.339699 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/823a705e-26f9-422c-83b0-75dc73349072-istiod-ca-cert" pod "823a705e-26f9-422c-83b0-75dc73349072" (UID: "823a705e-26f9-422c-83b0-75dc73349072"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:42:58.339809 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:58.339772 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/823a705e-26f9-422c-83b0-75dc73349072-istio-token\" (UniqueName: \"kubernetes.io/projected/823a705e-26f9-422c-83b0-75dc73349072-istio-token\") pod \"823a705e-26f9-422c-83b0-75dc73349072\" (UID: \"823a705e-26f9-422c-83b0-75dc73349072\") " Nov 03 13:42:58.339809 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:58.339803 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/823a705e-26f9-422c-83b0-75dc73349072-kube-api-access-7s97c\" (UniqueName: \"kubernetes.io/projected/823a705e-26f9-422c-83b0-75dc73349072-kube-api-access-7s97c\") pod \"823a705e-26f9-422c-83b0-75dc73349072\" (UID: \"823a705e-26f9-422c-83b0-75dc73349072\") " Nov 03 13:42:58.340489 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:58.340462 5024 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/823a705e-26f9-422c-83b0-75dc73349072-istio-data\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:42:58.340586 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:58.340495 5024 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/823a705e-26f9-422c-83b0-75dc73349072-credential-socket\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:42:58.340586 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:58.340512 5024 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0757da43-44aa-48e1-b169-00715a78ea02-istio-token\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:42:58.340586 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:58.340529 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zw559\" (UniqueName: \"kubernetes.io/projected/0757da43-44aa-48e1-b169-00715a78ea02-kube-api-access-zw559\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:42:58.340586 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:58.340545 5024 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/823a705e-26f9-422c-83b0-75dc73349072-workload-certs\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:42:58.340586 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:58.340561 5024 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/823a705e-26f9-422c-83b0-75dc73349072-istiod-ca-cert\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:42:58.340586 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:58.340577 5024 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/823a705e-26f9-422c-83b0-75dc73349072-workload-socket\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:42:58.341834 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:58.341803 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/823a705e-26f9-422c-83b0-75dc73349072-istio-envoy" pod "823a705e-26f9-422c-83b0-75dc73349072" (UID: "823a705e-26f9-422c-83b0-75dc73349072"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:42:58.342016 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:58.341998 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/823a705e-26f9-422c-83b0-75dc73349072-istio-token" pod "823a705e-26f9-422c-83b0-75dc73349072" (UID: "823a705e-26f9-422c-83b0-75dc73349072"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:42:58.342182 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:58.342157 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/823a705e-26f9-422c-83b0-75dc73349072-istio-podinfo" pod "823a705e-26f9-422c-83b0-75dc73349072" (UID: "823a705e-26f9-422c-83b0-75dc73349072"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 13:42:58.342182 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:58.342168 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/823a705e-26f9-422c-83b0-75dc73349072-kube-api-access-7s97c" pod "823a705e-26f9-422c-83b0-75dc73349072" (UID: "823a705e-26f9-422c-83b0-75dc73349072"). InnerVolumeSpecName "kube-api-access-7s97c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:42:58.441559 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:58.441525 5024 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/823a705e-26f9-422c-83b0-75dc73349072-istio-envoy\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:42:58.441559 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:58.441556 5024 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/823a705e-26f9-422c-83b0-75dc73349072-istio-podinfo\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:42:58.441559 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:58.441565 5024 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/823a705e-26f9-422c-83b0-75dc73349072-istio-token\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:42:58.441820 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:58.441575 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7s97c\" (UniqueName: \"kubernetes.io/projected/823a705e-26f9-422c-83b0-75dc73349072-kube-api-access-7s97c\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:42:58.811526 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:58.811481 5024 generic.go:356] "Generic (PLEG): container finished" podID="0757da43-44aa-48e1-b169-00715a78ea02" containerID="1cd0f17dbeb3270cfe79f3934f9fcbc3b4612cdc3a703cfabbc47c47e6cc1815" exitCode=0 Nov 03 13:42:58.812067 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:58.811791 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-pznlh" Nov 03 13:42:58.812909 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:58.812877 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-pznlh" event={"ID":"0757da43-44aa-48e1-b169-00715a78ea02","Type":"ContainerDied","Data":"1cd0f17dbeb3270cfe79f3934f9fcbc3b4612cdc3a703cfabbc47c47e6cc1815"} Nov 03 13:42:58.813710 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:58.813683 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-pznlh" event={"ID":"0757da43-44aa-48e1-b169-00715a78ea02","Type":"ContainerDied","Data":"eaf6a2b02c4aecdcd60875cd2f4bcd050383f86c3d3cf151196843ef934cc94a"} Nov 03 13:42:58.814225 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:58.813771 5024 scope.go:122] "RemoveContainer" containerID="1cd0f17dbeb3270cfe79f3934f9fcbc3b4612cdc3a703cfabbc47c47e6cc1815" Nov 03 13:42:58.814980 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:58.814952 5024 generic.go:356] "Generic (PLEG): container finished" podID="823a705e-26f9-422c-83b0-75dc73349072" containerID="58dc34962e683ed7b74140e17f6067eba9b1720fb11f3d51cfc0d746e336670d" exitCode=0 Nov 03 13:42:58.815134 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:58.815118 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-7h97f" Nov 03 13:42:58.815557 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:58.815538 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-7h97f" event={"ID":"823a705e-26f9-422c-83b0-75dc73349072","Type":"ContainerDied","Data":"58dc34962e683ed7b74140e17f6067eba9b1720fb11f3d51cfc0d746e336670d"} Nov 03 13:42:58.815821 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:58.815801 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-7h97f" event={"ID":"823a705e-26f9-422c-83b0-75dc73349072","Type":"ContainerDied","Data":"c98955c7bc5e38bb4596a68e0610846b2ddd8a94a7eefc4415d5c700886e9a1f"} Nov 03 13:42:58.829973 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:58.829951 5024 scope.go:122] "RemoveContainer" containerID="33974b23654fbff102111f8a70f02b8e2a0ae24b9e456b780a29ae4c7684b063" Nov 03 13:42:58.840154 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:58.840132 5024 scope.go:122] "RemoveContainer" containerID="1cd0f17dbeb3270cfe79f3934f9fcbc3b4612cdc3a703cfabbc47c47e6cc1815" Nov 03 13:42:58.840495 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:42:58.840466 5024 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1cd0f17dbeb3270cfe79f3934f9fcbc3b4612cdc3a703cfabbc47c47e6cc1815\": container with ID starting with 1cd0f17dbeb3270cfe79f3934f9fcbc3b4612cdc3a703cfabbc47c47e6cc1815 not found: ID does not exist" containerID="1cd0f17dbeb3270cfe79f3934f9fcbc3b4612cdc3a703cfabbc47c47e6cc1815" Nov 03 13:42:58.840554 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:58.840502 5024 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1cd0f17dbeb3270cfe79f3934f9fcbc3b4612cdc3a703cfabbc47c47e6cc1815"} err="failed to get container status \"1cd0f17dbeb3270cfe79f3934f9fcbc3b4612cdc3a703cfabbc47c47e6cc1815\": rpc error: code = NotFound desc = could not find container \"1cd0f17dbeb3270cfe79f3934f9fcbc3b4612cdc3a703cfabbc47c47e6cc1815\": container with ID starting with 1cd0f17dbeb3270cfe79f3934f9fcbc3b4612cdc3a703cfabbc47c47e6cc1815 not found: ID does not exist" Nov 03 13:42:58.840554 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:58.840519 5024 scope.go:122] "RemoveContainer" containerID="33974b23654fbff102111f8a70f02b8e2a0ae24b9e456b780a29ae4c7684b063" Nov 03 13:42:58.840732 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:42:58.840718 5024 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"33974b23654fbff102111f8a70f02b8e2a0ae24b9e456b780a29ae4c7684b063\": container with ID starting with 33974b23654fbff102111f8a70f02b8e2a0ae24b9e456b780a29ae4c7684b063 not found: ID does not exist" containerID="33974b23654fbff102111f8a70f02b8e2a0ae24b9e456b780a29ae4c7684b063" Nov 03 13:42:58.840776 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:58.840734 5024 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33974b23654fbff102111f8a70f02b8e2a0ae24b9e456b780a29ae4c7684b063"} err="failed to get container status \"33974b23654fbff102111f8a70f02b8e2a0ae24b9e456b780a29ae4c7684b063\": rpc error: code = NotFound desc = could not find container \"33974b23654fbff102111f8a70f02b8e2a0ae24b9e456b780a29ae4c7684b063\": container with ID starting with 33974b23654fbff102111f8a70f02b8e2a0ae24b9e456b780a29ae4c7684b063 not found: ID does not exist" Nov 03 13:42:58.840776 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:58.840746 5024 scope.go:122] "RemoveContainer" containerID="58dc34962e683ed7b74140e17f6067eba9b1720fb11f3d51cfc0d746e336670d" Nov 03 13:42:58.849573 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:58.849545 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-pznlh"] Nov 03 13:42:58.851400 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:58.851339 5024 scope.go:122] "RemoveContainer" containerID="58dc34962e683ed7b74140e17f6067eba9b1720fb11f3d51cfc0d746e336670d" Nov 03 13:42:58.851661 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:42:58.851634 5024 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"58dc34962e683ed7b74140e17f6067eba9b1720fb11f3d51cfc0d746e336670d\": container with ID starting with 58dc34962e683ed7b74140e17f6067eba9b1720fb11f3d51cfc0d746e336670d not found: ID does not exist" containerID="58dc34962e683ed7b74140e17f6067eba9b1720fb11f3d51cfc0d746e336670d" Nov 03 13:42:58.851754 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:58.851663 5024 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58dc34962e683ed7b74140e17f6067eba9b1720fb11f3d51cfc0d746e336670d"} err="failed to get container status \"58dc34962e683ed7b74140e17f6067eba9b1720fb11f3d51cfc0d746e336670d\": rpc error: code = NotFound desc = could not find container \"58dc34962e683ed7b74140e17f6067eba9b1720fb11f3d51cfc0d746e336670d\": container with ID starting with 58dc34962e683ed7b74140e17f6067eba9b1720fb11f3d51cfc0d746e336670d not found: ID does not exist" Nov 03 13:42:58.854217 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:58.854195 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-pznlh"] Nov 03 13:42:58.867961 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:58.867923 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-7h97f"] Nov 03 13:42:58.876576 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:42:58.876543 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-2b8a08-openshift-default-77998db5b7-7h97f"] Nov 03 13:43:00.109667 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:43:00.109622 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0757da43-44aa-48e1-b169-00715a78ea02" path="/var/lib/kubelet/pods/0757da43-44aa-48e1-b169-00715a78ea02/volumes" Nov 03 13:43:00.110395 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:43:00.110371 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="823a705e-26f9-422c-83b0-75dc73349072" path="/var/lib/kubelet/pods/823a705e-26f9-422c-83b0-75dc73349072/volumes" Nov 03 13:43:02.547725 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:43:02.547686 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-ef1693-86df94dcf8-nllf6"] Nov 03 13:43:02.548120 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:43:02.547884 5024 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ef1693-86df94dcf8-nllf6" podUID="0104d1e3-0aa2-489b-85de-c95cf4c85524" containerName="maas-api" containerID="cri-o://01647972e148180edda0b8f3893a5fe354d8d5d4db6ebbe53aa9122e1d41c82f" gracePeriod=30 Nov 03 13:43:03.023836 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:43:03.023590 5024 generic.go:356] "Generic (PLEG): container finished" podID="0104d1e3-0aa2-489b-85de-c95cf4c85524" containerID="01647972e148180edda0b8f3893a5fe354d8d5d4db6ebbe53aa9122e1d41c82f" exitCode=0 Nov 03 13:43:03.023836 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:43:03.023753 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ef1693-86df94dcf8-nllf6" event={"ID":"0104d1e3-0aa2-489b-85de-c95cf4c85524","Type":"ContainerDied","Data":"01647972e148180edda0b8f3893a5fe354d8d5d4db6ebbe53aa9122e1d41c82f"} Nov 03 13:43:03.080792 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:43:03.080765 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ef1693-86df94dcf8-nllf6" Nov 03 13:43:03.146265 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:43:03.146224 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0104d1e3-0aa2-489b-85de-c95cf4c85524-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0104d1e3-0aa2-489b-85de-c95cf4c85524-metrics-certs\") pod \"0104d1e3-0aa2-489b-85de-c95cf4c85524\" (UID: \"0104d1e3-0aa2-489b-85de-c95cf4c85524\") " Nov 03 13:43:03.146484 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:43:03.146459 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0104d1e3-0aa2-489b-85de-c95cf4c85524-kube-api-access-qp52t\" (UniqueName: \"kubernetes.io/projected/0104d1e3-0aa2-489b-85de-c95cf4c85524-kube-api-access-qp52t\") pod \"0104d1e3-0aa2-489b-85de-c95cf4c85524\" (UID: \"0104d1e3-0aa2-489b-85de-c95cf4c85524\") " Nov 03 13:43:03.146607 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:43:03.146589 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0104d1e3-0aa2-489b-85de-c95cf4c85524-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0104d1e3-0aa2-489b-85de-c95cf4c85524-maas-api-tls\") pod \"0104d1e3-0aa2-489b-85de-c95cf4c85524\" (UID: \"0104d1e3-0aa2-489b-85de-c95cf4c85524\") " Nov 03 13:43:03.146697 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:43:03.146682 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0104d1e3-0aa2-489b-85de-c95cf4c85524-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0104d1e3-0aa2-489b-85de-c95cf4c85524-openshift-service-ca\") pod \"0104d1e3-0aa2-489b-85de-c95cf4c85524\" (UID: \"0104d1e3-0aa2-489b-85de-c95cf4c85524\") " Nov 03 13:43:03.147276 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:43:03.147246 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0104d1e3-0aa2-489b-85de-c95cf4c85524-openshift-service-ca" pod "0104d1e3-0aa2-489b-85de-c95cf4c85524" (UID: "0104d1e3-0aa2-489b-85de-c95cf4c85524"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:43:03.148848 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:43:03.148796 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0104d1e3-0aa2-489b-85de-c95cf4c85524-maas-api-tls" pod "0104d1e3-0aa2-489b-85de-c95cf4c85524" (UID: "0104d1e3-0aa2-489b-85de-c95cf4c85524"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 13:43:03.149060 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:43:03.149041 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0104d1e3-0aa2-489b-85de-c95cf4c85524-kube-api-access-qp52t" pod "0104d1e3-0aa2-489b-85de-c95cf4c85524" (UID: "0104d1e3-0aa2-489b-85de-c95cf4c85524"). InnerVolumeSpecName "kube-api-access-qp52t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:43:03.149182 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:43:03.149152 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0104d1e3-0aa2-489b-85de-c95cf4c85524-metrics-certs" pod "0104d1e3-0aa2-489b-85de-c95cf4c85524" (UID: "0104d1e3-0aa2-489b-85de-c95cf4c85524"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 13:43:03.248088 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:43:03.248054 5024 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0104d1e3-0aa2-489b-85de-c95cf4c85524-maas-api-tls\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:43:03.248088 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:43:03.248084 5024 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0104d1e3-0aa2-489b-85de-c95cf4c85524-openshift-service-ca\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:43:03.248088 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:43:03.248093 5024 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0104d1e3-0aa2-489b-85de-c95cf4c85524-metrics-certs\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:43:03.248352 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:43:03.248102 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qp52t\" (UniqueName: \"kubernetes.io/projected/0104d1e3-0aa2-489b-85de-c95cf4c85524-kube-api-access-qp52t\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:43:04.032700 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:43:04.032667 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ef1693-86df94dcf8-nllf6" Nov 03 13:43:04.032700 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:43:04.032691 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ef1693-86df94dcf8-nllf6" event={"ID":"0104d1e3-0aa2-489b-85de-c95cf4c85524","Type":"ContainerDied","Data":"10a4553ffa35daa5cbd87b83faea4c7ae4a0336df5575d883b0d3913e71cdc54"} Nov 03 13:43:04.033246 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:43:04.032755 5024 scope.go:122] "RemoveContainer" containerID="01647972e148180edda0b8f3893a5fe354d8d5d4db6ebbe53aa9122e1d41c82f" Nov 03 13:43:04.059791 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:43:04.059762 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-ef1693-86df94dcf8-nllf6"] Nov 03 13:43:04.063766 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:43:04.063741 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-ef1693-86df94dcf8-nllf6"] Nov 03 13:43:04.109426 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:43:04.109393 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0104d1e3-0aa2-489b-85de-c95cf4c85524" path="/var/lib/kubelet/pods/0104d1e3-0aa2-489b-85de-c95cf4c85524/volumes" Nov 03 13:43:09.382170 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:43:09.382135 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-ef1693-openshift-default-66b997ddf7-fqzk5"] Nov 03 13:43:09.382678 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:43:09.382406 5024 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-ef1693-openshift-default-66b997ddf7-fqzk5" podUID="26bee2e8-835c-49d8-9e88-f959627e0ec9" containerName="istio-proxy" containerID="cri-o://ecb99fcca1ad99be501b687c96e52f3fceb74db263d52460b29f79fafd21d8c1" gracePeriod=30 Nov 03 13:43:09.633150 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:43:09.633025 5024 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d7mwz" containerName="extractor" Nov 03 13:43:13.527950 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:43:13.527910 5024 patch_prober.go:29] interesting pod/e2e-extmod-a-ef1693-openshift-default-66b997ddf7-fqzk5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 13:43:13.528346 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:43:13.527971 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-ef1693-openshift-default-66b997ddf7-fqzk5" podUID="26bee2e8-835c-49d8-9e88-f959627e0ec9" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 13:43:14.730168 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:43:14.730131 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-ef1693-openshift-default-66b997ddf7-fqzk5" Nov 03 13:43:14.774357 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:43:14.774272 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26bee2e8-835c-49d8-9e88-f959627e0ec9-istio-token\" (UniqueName: \"kubernetes.io/projected/26bee2e8-835c-49d8-9e88-f959627e0ec9-istio-token\") pod \"26bee2e8-835c-49d8-9e88-f959627e0ec9\" (UID: \"26bee2e8-835c-49d8-9e88-f959627e0ec9\") " Nov 03 13:43:14.774549 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:43:14.774528 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26bee2e8-835c-49d8-9e88-f959627e0ec9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26bee2e8-835c-49d8-9e88-f959627e0ec9-istio-data\") pod \"26bee2e8-835c-49d8-9e88-f959627e0ec9\" (UID: \"26bee2e8-835c-49d8-9e88-f959627e0ec9\") " Nov 03 13:43:14.774602 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:43:14.774580 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26bee2e8-835c-49d8-9e88-f959627e0ec9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26bee2e8-835c-49d8-9e88-f959627e0ec9-istio-envoy\") pod \"26bee2e8-835c-49d8-9e88-f959627e0ec9\" (UID: \"26bee2e8-835c-49d8-9e88-f959627e0ec9\") " Nov 03 13:43:14.774663 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:43:14.774646 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26bee2e8-835c-49d8-9e88-f959627e0ec9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26bee2e8-835c-49d8-9e88-f959627e0ec9-workload-certs\") pod \"26bee2e8-835c-49d8-9e88-f959627e0ec9\" (UID: \"26bee2e8-835c-49d8-9e88-f959627e0ec9\") " Nov 03 13:43:14.774705 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:43:14.774680 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26bee2e8-835c-49d8-9e88-f959627e0ec9-kube-api-access-7x2lh\" (UniqueName: \"kubernetes.io/projected/26bee2e8-835c-49d8-9e88-f959627e0ec9-kube-api-access-7x2lh\") pod \"26bee2e8-835c-49d8-9e88-f959627e0ec9\" (UID: \"26bee2e8-835c-49d8-9e88-f959627e0ec9\") " Nov 03 13:43:14.774750 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:43:14.774741 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/26bee2e8-835c-49d8-9e88-f959627e0ec9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26bee2e8-835c-49d8-9e88-f959627e0ec9-istiod-ca-cert\") pod \"26bee2e8-835c-49d8-9e88-f959627e0ec9\" (UID: \"26bee2e8-835c-49d8-9e88-f959627e0ec9\") " Nov 03 13:43:14.774787 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:43:14.774778 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26bee2e8-835c-49d8-9e88-f959627e0ec9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26bee2e8-835c-49d8-9e88-f959627e0ec9-credential-socket\") pod \"26bee2e8-835c-49d8-9e88-f959627e0ec9\" (UID: \"26bee2e8-835c-49d8-9e88-f959627e0ec9\") " Nov 03 13:43:14.774899 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:43:14.774879 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/26bee2e8-835c-49d8-9e88-f959627e0ec9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26bee2e8-835c-49d8-9e88-f959627e0ec9-istio-podinfo\") pod \"26bee2e8-835c-49d8-9e88-f959627e0ec9\" (UID: \"26bee2e8-835c-49d8-9e88-f959627e0ec9\") " Nov 03 13:43:14.775982 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:43:14.775107 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26bee2e8-835c-49d8-9e88-f959627e0ec9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26bee2e8-835c-49d8-9e88-f959627e0ec9-workload-socket\") pod \"26bee2e8-835c-49d8-9e88-f959627e0ec9\" (UID: \"26bee2e8-835c-49d8-9e88-f959627e0ec9\") " Nov 03 13:43:14.776183 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:43:14.776146 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26bee2e8-835c-49d8-9e88-f959627e0ec9-workload-socket" pod "26bee2e8-835c-49d8-9e88-f959627e0ec9" (UID: "26bee2e8-835c-49d8-9e88-f959627e0ec9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:43:14.776814 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:43:14.776787 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26bee2e8-835c-49d8-9e88-f959627e0ec9-istio-data" pod "26bee2e8-835c-49d8-9e88-f959627e0ec9" (UID: "26bee2e8-835c-49d8-9e88-f959627e0ec9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:43:14.776981 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:43:14.776963 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26bee2e8-835c-49d8-9e88-f959627e0ec9-credential-socket" pod "26bee2e8-835c-49d8-9e88-f959627e0ec9" (UID: "26bee2e8-835c-49d8-9e88-f959627e0ec9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:43:14.777195 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:43:14.777150 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26bee2e8-835c-49d8-9e88-f959627e0ec9-workload-certs" pod "26bee2e8-835c-49d8-9e88-f959627e0ec9" (UID: "26bee2e8-835c-49d8-9e88-f959627e0ec9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:43:14.777796 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:43:14.777771 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26bee2e8-835c-49d8-9e88-f959627e0ec9-istiod-ca-cert" pod "26bee2e8-835c-49d8-9e88-f959627e0ec9" (UID: "26bee2e8-835c-49d8-9e88-f959627e0ec9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:43:14.778921 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:43:14.778900 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26bee2e8-835c-49d8-9e88-f959627e0ec9-kube-api-access-7x2lh" pod "26bee2e8-835c-49d8-9e88-f959627e0ec9" (UID: "26bee2e8-835c-49d8-9e88-f959627e0ec9"). InnerVolumeSpecName "kube-api-access-7x2lh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:43:14.779056 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:43:14.779034 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26bee2e8-835c-49d8-9e88-f959627e0ec9-istio-token" pod "26bee2e8-835c-49d8-9e88-f959627e0ec9" (UID: "26bee2e8-835c-49d8-9e88-f959627e0ec9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:43:14.779376 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:43:14.779353 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/26bee2e8-835c-49d8-9e88-f959627e0ec9-istio-podinfo" pod "26bee2e8-835c-49d8-9e88-f959627e0ec9" (UID: "26bee2e8-835c-49d8-9e88-f959627e0ec9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 13:43:14.780422 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:43:14.780401 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26bee2e8-835c-49d8-9e88-f959627e0ec9-istio-envoy" pod "26bee2e8-835c-49d8-9e88-f959627e0ec9" (UID: "26bee2e8-835c-49d8-9e88-f959627e0ec9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:43:14.876976 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:43:14.876937 5024 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26bee2e8-835c-49d8-9e88-f959627e0ec9-istio-data\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:43:14.876976 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:43:14.876969 5024 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26bee2e8-835c-49d8-9e88-f959627e0ec9-istio-envoy\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:43:14.876976 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:43:14.876983 5024 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26bee2e8-835c-49d8-9e88-f959627e0ec9-workload-certs\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:43:14.877311 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:43:14.876994 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7x2lh\" (UniqueName: \"kubernetes.io/projected/26bee2e8-835c-49d8-9e88-f959627e0ec9-kube-api-access-7x2lh\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:43:14.877311 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:43:14.877006 5024 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26bee2e8-835c-49d8-9e88-f959627e0ec9-istiod-ca-cert\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:43:14.877311 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:43:14.877016 5024 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26bee2e8-835c-49d8-9e88-f959627e0ec9-credential-socket\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:43:14.877311 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:43:14.877024 5024 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26bee2e8-835c-49d8-9e88-f959627e0ec9-istio-podinfo\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:43:14.877311 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:43:14.877033 5024 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26bee2e8-835c-49d8-9e88-f959627e0ec9-workload-socket\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:43:14.877311 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:43:14.877041 5024 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26bee2e8-835c-49d8-9e88-f959627e0ec9-istio-token\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:43:15.226389 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:43:15.226340 5024 generic.go:356] "Generic (PLEG): container finished" podID="26bee2e8-835c-49d8-9e88-f959627e0ec9" containerID="ecb99fcca1ad99be501b687c96e52f3fceb74db263d52460b29f79fafd21d8c1" exitCode=0 Nov 03 13:43:15.226986 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:43:15.226949 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-ef1693-openshift-default-66b997ddf7-fqzk5" event={"ID":"26bee2e8-835c-49d8-9e88-f959627e0ec9","Type":"ContainerDied","Data":"ecb99fcca1ad99be501b687c96e52f3fceb74db263d52460b29f79fafd21d8c1"} Nov 03 13:43:15.227194 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:43:15.227169 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-ef1693-openshift-default-66b997ddf7-fqzk5" event={"ID":"26bee2e8-835c-49d8-9e88-f959627e0ec9","Type":"ContainerDied","Data":"ed42652c0866458c33a278a85dab6da65bb773b22d7d7599b7a60d21bdc783f4"} Nov 03 13:43:15.227339 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:43:15.227325 5024 scope.go:122] "RemoveContainer" containerID="ecb99fcca1ad99be501b687c96e52f3fceb74db263d52460b29f79fafd21d8c1" Nov 03 13:43:15.227693 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:43:15.227658 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-ef1693-openshift-default-66b997ddf7-fqzk5" Nov 03 13:43:15.255907 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:43:15.255802 5024 scope.go:122] "RemoveContainer" containerID="ecb99fcca1ad99be501b687c96e52f3fceb74db263d52460b29f79fafd21d8c1" Nov 03 13:43:15.257747 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:43:15.257677 5024 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ecb99fcca1ad99be501b687c96e52f3fceb74db263d52460b29f79fafd21d8c1\": container with ID starting with ecb99fcca1ad99be501b687c96e52f3fceb74db263d52460b29f79fafd21d8c1 not found: ID does not exist" containerID="ecb99fcca1ad99be501b687c96e52f3fceb74db263d52460b29f79fafd21d8c1" Nov 03 13:43:15.257747 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:43:15.257719 5024 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ecb99fcca1ad99be501b687c96e52f3fceb74db263d52460b29f79fafd21d8c1"} err="failed to get container status \"ecb99fcca1ad99be501b687c96e52f3fceb74db263d52460b29f79fafd21d8c1\": rpc error: code = NotFound desc = could not find container \"ecb99fcca1ad99be501b687c96e52f3fceb74db263d52460b29f79fafd21d8c1\": container with ID starting with ecb99fcca1ad99be501b687c96e52f3fceb74db263d52460b29f79fafd21d8c1 not found: ID does not exist" Nov 03 13:43:15.272639 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:43:15.272611 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-ef1693-openshift-default-66b997ddf7-fqzk5"] Nov 03 13:43:15.280504 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:43:15.280479 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-ef1693-openshift-default-66b997ddf7-fqzk5"] Nov 03 13:43:15.603124 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:43:15.603090 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-6865svtfs"] Nov 03 13:43:15.603997 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:43:15.603969 5024 cpu_manager.go:414] "containerMap: removing container" podUID="0757da43-44aa-48e1-b169-00715a78ea02" containerName="istio-proxy" Nov 03 13:43:15.604067 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:43:15.603998 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0757da43-44aa-48e1-b169-00715a78ea02" containerName="istio-proxy" Nov 03 13:43:15.604067 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:43:15.604022 5024 cpu_manager.go:414] "containerMap: removing container" podUID="0757da43-44aa-48e1-b169-00715a78ea02" containerName="istio-proxy" Nov 03 13:43:15.604067 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:43:15.604030 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0757da43-44aa-48e1-b169-00715a78ea02" containerName="istio-proxy" Nov 03 13:43:15.604067 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:43:15.604042 5024 cpu_manager.go:414] "containerMap: removing container" podUID="823a705e-26f9-422c-83b0-75dc73349072" containerName="istio-proxy" Nov 03 13:43:15.604067 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:43:15.604050 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="823a705e-26f9-422c-83b0-75dc73349072" containerName="istio-proxy" Nov 03 13:43:15.604349 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:43:15.604089 5024 cpu_manager.go:414] "containerMap: removing container" podUID="0104d1e3-0aa2-489b-85de-c95cf4c85524" containerName="maas-api" Nov 03 13:43:15.604349 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:43:15.604098 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0104d1e3-0aa2-489b-85de-c95cf4c85524" containerName="maas-api" Nov 03 13:43:15.604349 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:43:15.604118 5024 cpu_manager.go:414] "containerMap: removing container" podUID="26bee2e8-835c-49d8-9e88-f959627e0ec9" containerName="istio-proxy" Nov 03 13:43:15.604349 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:43:15.604125 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26bee2e8-835c-49d8-9e88-f959627e0ec9" containerName="istio-proxy" Nov 03 13:43:15.604349 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:43:15.604312 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="0757da43-44aa-48e1-b169-00715a78ea02" containerName="istio-proxy" Nov 03 13:43:15.604349 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:43:15.604330 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="0757da43-44aa-48e1-b169-00715a78ea02" containerName="istio-proxy" Nov 03 13:43:15.604349 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:43:15.604338 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="823a705e-26f9-422c-83b0-75dc73349072" containerName="istio-proxy" Nov 03 13:43:15.604696 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:43:15.604354 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="0104d1e3-0aa2-489b-85de-c95cf4c85524" containerName="maas-api" Nov 03 13:43:15.604696 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:43:15.604364 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="26bee2e8-835c-49d8-9e88-f959627e0ec9" containerName="istio-proxy" Nov 03 13:43:15.609889 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:43:15.609854 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-6865svtfs" Nov 03 13:43:15.618749 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:43:15.618709 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-7c192b\"/\"routing-overlay\"" Nov 03 13:43:15.618910 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:43:15.618820 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-7c192b\"/\"payload-processing-external-model-e2e--cd1da41a-dockercfg-gxlvm\"" Nov 03 13:43:15.618976 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:43:15.618908 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-7c192b\"/\"kube-root-ca.crt\"" Nov 03 13:43:15.618976 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:43:15.618947 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-7c192b\"/\"payload-processing-external-model-plugins-e2e-extmod-b-7c192b\"" Nov 03 13:43:15.619087 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:43:15.618997 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-7c192b\"/\"openshift-service-ca.crt\"" Nov 03 13:43:15.619087 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:43:15.619059 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-7c192b\"/\"agc-cred-5a8e154da4af\"" Nov 03 13:43:15.619354 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:43:15.619330 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-6865svtfs"] Nov 03 13:43:15.768956 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:43:15.768905 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d34e5887-ae31-43ce-8d24-9973d8ba447c-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-7c192b-6865svtfs\" (UID: \"d34e5887-ae31-43ce-8d24-9973d8ba447c\") " pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-6865svtfs" Nov 03 13:43:15.769477 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:43:15.768998 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d34e5887-ae31-43ce-8d24-9973d8ba447c-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-7c192b-6865svtfs\" (UID: \"d34e5887-ae31-43ce-8d24-9973d8ba447c\") " pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-6865svtfs" Nov 03 13:43:15.769477 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:43:15.769026 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hpkhh\" (UniqueName: \"kubernetes.io/projected/d34e5887-ae31-43ce-8d24-9973d8ba447c-kube-api-access-hpkhh\") pod \"payload-processing-external-model-e2e-extmod-b-7c192b-6865svtfs\" (UID: \"d34e5887-ae31-43ce-8d24-9973d8ba447c\") " pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-6865svtfs" Nov 03 13:43:15.769477 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:43:15.769139 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d34e5887-ae31-43ce-8d24-9973d8ba447c-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-7c192b-6865svtfs\" (UID: \"d34e5887-ae31-43ce-8d24-9973d8ba447c\") " pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-6865svtfs" Nov 03 13:43:15.870890 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:43:15.870793 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d34e5887-ae31-43ce-8d24-9973d8ba447c-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-7c192b-6865svtfs\" (UID: \"d34e5887-ae31-43ce-8d24-9973d8ba447c\") " pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-6865svtfs" Nov 03 13:43:15.871077 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:43:15.870963 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d34e5887-ae31-43ce-8d24-9973d8ba447c-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-7c192b-6865svtfs\" (UID: \"d34e5887-ae31-43ce-8d24-9973d8ba447c\") " pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-6865svtfs" Nov 03 13:43:15.871077 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:43:15.871010 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d34e5887-ae31-43ce-8d24-9973d8ba447c-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-7c192b-6865svtfs\" (UID: \"d34e5887-ae31-43ce-8d24-9973d8ba447c\") " pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-6865svtfs" Nov 03 13:43:15.871077 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:43:15.871028 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hpkhh\" (UniqueName: \"kubernetes.io/projected/d34e5887-ae31-43ce-8d24-9973d8ba447c-kube-api-access-hpkhh\") pod \"payload-processing-external-model-e2e-extmod-b-7c192b-6865svtfs\" (UID: \"d34e5887-ae31-43ce-8d24-9973d8ba447c\") " pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-6865svtfs" Nov 03 13:43:15.871729 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:43:15.871699 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d34e5887-ae31-43ce-8d24-9973d8ba447c-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-7c192b-6865svtfs\" (UID: \"d34e5887-ae31-43ce-8d24-9973d8ba447c\") " pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-6865svtfs" Nov 03 13:43:15.871838 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:43:15.871813 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d34e5887-ae31-43ce-8d24-9973d8ba447c-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-7c192b-6865svtfs\" (UID: \"d34e5887-ae31-43ce-8d24-9973d8ba447c\") " pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-6865svtfs" Nov 03 13:43:15.873376 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:43:15.873350 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d34e5887-ae31-43ce-8d24-9973d8ba447c-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-7c192b-6865svtfs\" (UID: \"d34e5887-ae31-43ce-8d24-9973d8ba447c\") " pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-6865svtfs" Nov 03 13:43:15.880712 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:43:15.880690 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hpkhh\" (UniqueName: \"kubernetes.io/projected/d34e5887-ae31-43ce-8d24-9973d8ba447c-kube-api-access-hpkhh\") pod \"payload-processing-external-model-e2e-extmod-b-7c192b-6865svtfs\" (UID: \"d34e5887-ae31-43ce-8d24-9973d8ba447c\") " pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-6865svtfs" Nov 03 13:43:15.932997 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:43:15.932960 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-6865svtfs" Nov 03 13:43:16.110815 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:43:16.110774 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="26bee2e8-835c-49d8-9e88-f959627e0ec9" path="/var/lib/kubelet/pods/26bee2e8-835c-49d8-9e88-f959627e0ec9/volumes" Nov 03 13:43:16.218518 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:43:16.218480 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-6865svtfs"] Nov 03 13:43:16.219321 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:43:16.219278 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd34e5887_ae31_43ce_8d24_9973d8ba447c.slice/crio-94ec91ca4418dc01aa6636b0663c2f8bdd2715f8f9bb631eb71127f7495b9a20 WatchSource:0}: Error finding container 94ec91ca4418dc01aa6636b0663c2f8bdd2715f8f9bb631eb71127f7495b9a20: Status 404 returned error can't find the container with id 94ec91ca4418dc01aa6636b0663c2f8bdd2715f8f9bb631eb71127f7495b9a20 Nov 03 13:43:16.236776 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:43:16.236745 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-6865svtfs" event={"ID":"d34e5887-ae31-43ce-8d24-9973d8ba447c","Type":"ContainerStarted","Data":"94ec91ca4418dc01aa6636b0663c2f8bdd2715f8f9bb631eb71127f7495b9a20"} Nov 03 13:43:17.245704 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:43:17.245658 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-6865svtfs" event={"ID":"d34e5887-ae31-43ce-8d24-9973d8ba447c","Type":"ContainerStarted","Data":"eb95531032faf67b58545f82375d2cabf0dba2a81bc135b9ab9b09603a071801"} Nov 03 13:43:17.246156 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:43:17.245794 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-6865svtfs" Nov 03 13:43:17.273324 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:43:17.273245 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-6865svtfs" podStartSLOduration=2.273228236 podStartE2EDuration="2.273228236s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:43:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:43:17.270239047 +0000 UTC m=+2077.278430316" watchObservedRunningTime="2026-11-03 13:43:17.273228236 +0000 UTC m=+2077.281419513" Nov 03 13:43:19.131753 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:43:19.131714 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-1a18c8-d6c758bcc-pmrxw"] Nov 03 13:43:19.132341 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:43:19.132264 5024 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-1a18c8-d6c758bcc-pmrxw" podUID="937216cd-5ccd-43e1-8b64-a4d31af284d6" containerName="maas-api" containerID="cri-o://f1b5973ec3f4f63798b5167ecce777ae1841dd246b521f25db2ab7e1db0f7538" gracePeriod=30 Nov 03 13:43:19.260479 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:43:19.260449 5024 generic.go:356] "Generic (PLEG): container finished" podID="937216cd-5ccd-43e1-8b64-a4d31af284d6" containerID="f1b5973ec3f4f63798b5167ecce777ae1841dd246b521f25db2ab7e1db0f7538" exitCode=0 Nov 03 13:43:19.260613 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:43:19.260508 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-1a18c8-d6c758bcc-pmrxw" event={"ID":"937216cd-5ccd-43e1-8b64-a4d31af284d6","Type":"ContainerDied","Data":"f1b5973ec3f4f63798b5167ecce777ae1841dd246b521f25db2ab7e1db0f7538"} Nov 03 13:43:19.663842 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:43:19.663809 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-1a18c8-d6c758bcc-pmrxw" Nov 03 13:43:19.760705 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:43:19.755354 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/937216cd-5ccd-43e1-8b64-a4d31af284d6-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/937216cd-5ccd-43e1-8b64-a4d31af284d6-openshift-service-ca\") pod \"937216cd-5ccd-43e1-8b64-a4d31af284d6\" (UID: \"937216cd-5ccd-43e1-8b64-a4d31af284d6\") " Nov 03 13:43:19.760705 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:43:19.755445 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/937216cd-5ccd-43e1-8b64-a4d31af284d6-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/937216cd-5ccd-43e1-8b64-a4d31af284d6-maas-api-tls\") pod \"937216cd-5ccd-43e1-8b64-a4d31af284d6\" (UID: \"937216cd-5ccd-43e1-8b64-a4d31af284d6\") " Nov 03 13:43:19.760705 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:43:19.755486 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/937216cd-5ccd-43e1-8b64-a4d31af284d6-metrics-certs\" (UniqueName: \"kubernetes.io/secret/937216cd-5ccd-43e1-8b64-a4d31af284d6-metrics-certs\") pod \"937216cd-5ccd-43e1-8b64-a4d31af284d6\" (UID: \"937216cd-5ccd-43e1-8b64-a4d31af284d6\") " Nov 03 13:43:19.760705 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:43:19.755658 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/937216cd-5ccd-43e1-8b64-a4d31af284d6-kube-api-access-mqzb7\" (UniqueName: \"kubernetes.io/projected/937216cd-5ccd-43e1-8b64-a4d31af284d6-kube-api-access-mqzb7\") pod \"937216cd-5ccd-43e1-8b64-a4d31af284d6\" (UID: \"937216cd-5ccd-43e1-8b64-a4d31af284d6\") " Nov 03 13:43:19.760705 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:43:19.759243 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/937216cd-5ccd-43e1-8b64-a4d31af284d6-kube-api-access-mqzb7" pod "937216cd-5ccd-43e1-8b64-a4d31af284d6" (UID: "937216cd-5ccd-43e1-8b64-a4d31af284d6"). InnerVolumeSpecName "kube-api-access-mqzb7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:43:19.760705 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:43:19.759713 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/937216cd-5ccd-43e1-8b64-a4d31af284d6-openshift-service-ca" pod "937216cd-5ccd-43e1-8b64-a4d31af284d6" (UID: "937216cd-5ccd-43e1-8b64-a4d31af284d6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:43:19.761987 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:43:19.761950 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/937216cd-5ccd-43e1-8b64-a4d31af284d6-maas-api-tls" pod "937216cd-5ccd-43e1-8b64-a4d31af284d6" (UID: "937216cd-5ccd-43e1-8b64-a4d31af284d6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 13:43:19.764075 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:43:19.764046 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/937216cd-5ccd-43e1-8b64-a4d31af284d6-metrics-certs" pod "937216cd-5ccd-43e1-8b64-a4d31af284d6" (UID: "937216cd-5ccd-43e1-8b64-a4d31af284d6"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 13:43:19.858342 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:43:19.858271 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mqzb7\" (UniqueName: \"kubernetes.io/projected/937216cd-5ccd-43e1-8b64-a4d31af284d6-kube-api-access-mqzb7\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:43:19.858342 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:43:19.858344 5024 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/937216cd-5ccd-43e1-8b64-a4d31af284d6-openshift-service-ca\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:43:19.858558 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:43:19.858359 5024 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/937216cd-5ccd-43e1-8b64-a4d31af284d6-maas-api-tls\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:43:19.858558 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:43:19.858368 5024 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/937216cd-5ccd-43e1-8b64-a4d31af284d6-metrics-certs\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:43:20.268602 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:43:20.268559 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-1a18c8-d6c758bcc-pmrxw" event={"ID":"937216cd-5ccd-43e1-8b64-a4d31af284d6","Type":"ContainerDied","Data":"413a30f11b8aba299dd4b9111ce51845820ce43cea1527af60e7c82f521dbb0a"} Nov 03 13:43:20.269091 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:43:20.268629 5024 scope.go:122] "RemoveContainer" containerID="f1b5973ec3f4f63798b5167ecce777ae1841dd246b521f25db2ab7e1db0f7538" Nov 03 13:43:20.269091 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:43:20.268584 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-1a18c8-d6c758bcc-pmrxw" Nov 03 13:43:20.290032 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:43:20.289990 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-1a18c8-d6c758bcc-pmrxw"] Nov 03 13:43:20.295930 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:43:20.295905 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-1a18c8-d6c758bcc-pmrxw"] Nov 03 13:43:22.109324 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:43:22.109275 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="937216cd-5ccd-43e1-8b64-a4d31af284d6" path="/var/lib/kubelet/pods/937216cd-5ccd-43e1-8b64-a4d31af284d6/volumes" Nov 03 13:43:24.816419 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:43:24.816379 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-1a18c8-openshift-default-67d7b5995c-z4csj"] Nov 03 13:43:24.817315 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:43:24.816631 5024 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-1a18c8-openshift-default-67d7b5995c-z4csj" podUID="168564ee-9452-4aba-8391-d4f0bdaafa7b" containerName="istio-proxy" containerID="cri-o://538c794a1ce0474466bbbc53c119dac920c1451c9b9ab21f523c5f9e02e4a4fa" gracePeriod=30 Nov 03 13:43:28.255231 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:43:28.255187 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-6865svtfs" Nov 03 13:43:30.105044 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:43:30.105018 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-1a18c8-openshift-default-67d7b5995c-z4csj" Nov 03 13:43:30.173129 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:43:30.173088 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/168564ee-9452-4aba-8391-d4f0bdaafa7b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/168564ee-9452-4aba-8391-d4f0bdaafa7b-istio-envoy\") pod \"168564ee-9452-4aba-8391-d4f0bdaafa7b\" (UID: \"168564ee-9452-4aba-8391-d4f0bdaafa7b\") " Nov 03 13:43:30.173336 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:43:30.173154 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/168564ee-9452-4aba-8391-d4f0bdaafa7b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/168564ee-9452-4aba-8391-d4f0bdaafa7b-workload-certs\") pod \"168564ee-9452-4aba-8391-d4f0bdaafa7b\" (UID: \"168564ee-9452-4aba-8391-d4f0bdaafa7b\") " Nov 03 13:43:30.173336 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:43:30.173197 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/168564ee-9452-4aba-8391-d4f0bdaafa7b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/168564ee-9452-4aba-8391-d4f0bdaafa7b-credential-socket\") pod \"168564ee-9452-4aba-8391-d4f0bdaafa7b\" (UID: \"168564ee-9452-4aba-8391-d4f0bdaafa7b\") " Nov 03 13:43:30.173336 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:43:30.173315 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/168564ee-9452-4aba-8391-d4f0bdaafa7b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/168564ee-9452-4aba-8391-d4f0bdaafa7b-istiod-ca-cert\") pod \"168564ee-9452-4aba-8391-d4f0bdaafa7b\" (UID: \"168564ee-9452-4aba-8391-d4f0bdaafa7b\") " Nov 03 13:43:30.173543 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:43:30.173363 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/168564ee-9452-4aba-8391-d4f0bdaafa7b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/168564ee-9452-4aba-8391-d4f0bdaafa7b-istio-podinfo\") pod \"168564ee-9452-4aba-8391-d4f0bdaafa7b\" (UID: \"168564ee-9452-4aba-8391-d4f0bdaafa7b\") " Nov 03 13:43:30.173543 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:43:30.173435 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/168564ee-9452-4aba-8391-d4f0bdaafa7b-kube-api-access-lrg9v\" (UniqueName: \"kubernetes.io/projected/168564ee-9452-4aba-8391-d4f0bdaafa7b-kube-api-access-lrg9v\") pod \"168564ee-9452-4aba-8391-d4f0bdaafa7b\" (UID: \"168564ee-9452-4aba-8391-d4f0bdaafa7b\") " Nov 03 13:43:30.173543 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:43:30.173479 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/168564ee-9452-4aba-8391-d4f0bdaafa7b-workload-certs" pod "168564ee-9452-4aba-8391-d4f0bdaafa7b" (UID: "168564ee-9452-4aba-8391-d4f0bdaafa7b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:43:30.173543 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:43:30.173504 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/168564ee-9452-4aba-8391-d4f0bdaafa7b-istio-token\" (UniqueName: \"kubernetes.io/projected/168564ee-9452-4aba-8391-d4f0bdaafa7b-istio-token\") pod \"168564ee-9452-4aba-8391-d4f0bdaafa7b\" (UID: \"168564ee-9452-4aba-8391-d4f0bdaafa7b\") " Nov 03 13:43:30.173756 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:43:30.173683 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/168564ee-9452-4aba-8391-d4f0bdaafa7b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/168564ee-9452-4aba-8391-d4f0bdaafa7b-istio-data\") pod \"168564ee-9452-4aba-8391-d4f0bdaafa7b\" (UID: \"168564ee-9452-4aba-8391-d4f0bdaafa7b\") " Nov 03 13:43:30.173756 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:43:30.173726 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/168564ee-9452-4aba-8391-d4f0bdaafa7b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/168564ee-9452-4aba-8391-d4f0bdaafa7b-workload-socket\") pod \"168564ee-9452-4aba-8391-d4f0bdaafa7b\" (UID: \"168564ee-9452-4aba-8391-d4f0bdaafa7b\") " Nov 03 13:43:30.173871 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:43:30.173841 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/168564ee-9452-4aba-8391-d4f0bdaafa7b-istiod-ca-cert" pod "168564ee-9452-4aba-8391-d4f0bdaafa7b" (UID: "168564ee-9452-4aba-8391-d4f0bdaafa7b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:43:30.174183 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:43:30.174154 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/168564ee-9452-4aba-8391-d4f0bdaafa7b-credential-socket" pod "168564ee-9452-4aba-8391-d4f0bdaafa7b" (UID: "168564ee-9452-4aba-8391-d4f0bdaafa7b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:43:30.181589 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:43:30.181264 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/168564ee-9452-4aba-8391-d4f0bdaafa7b-workload-socket" pod "168564ee-9452-4aba-8391-d4f0bdaafa7b" (UID: "168564ee-9452-4aba-8391-d4f0bdaafa7b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:43:30.187845 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:43:30.187722 5024 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/168564ee-9452-4aba-8391-d4f0bdaafa7b-workload-socket\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:43:30.187845 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:43:30.187761 5024 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/168564ee-9452-4aba-8391-d4f0bdaafa7b-workload-certs\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:43:30.187845 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:43:30.187778 5024 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/168564ee-9452-4aba-8391-d4f0bdaafa7b-credential-socket\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:43:30.187845 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:43:30.187793 5024 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/168564ee-9452-4aba-8391-d4f0bdaafa7b-istiod-ca-cert\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:43:30.195009 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:43:30.194975 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/168564ee-9452-4aba-8391-d4f0bdaafa7b-istio-data" pod "168564ee-9452-4aba-8391-d4f0bdaafa7b" (UID: "168564ee-9452-4aba-8391-d4f0bdaafa7b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:43:30.195120 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:43:30.195032 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/168564ee-9452-4aba-8391-d4f0bdaafa7b-istio-token" pod "168564ee-9452-4aba-8391-d4f0bdaafa7b" (UID: "168564ee-9452-4aba-8391-d4f0bdaafa7b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:43:30.199563 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:43:30.198944 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/168564ee-9452-4aba-8391-d4f0bdaafa7b-istio-envoy" pod "168564ee-9452-4aba-8391-d4f0bdaafa7b" (UID: "168564ee-9452-4aba-8391-d4f0bdaafa7b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:43:30.202613 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:43:30.202582 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/168564ee-9452-4aba-8391-d4f0bdaafa7b-kube-api-access-lrg9v" pod "168564ee-9452-4aba-8391-d4f0bdaafa7b" (UID: "168564ee-9452-4aba-8391-d4f0bdaafa7b"). InnerVolumeSpecName "kube-api-access-lrg9v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:43:30.210617 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:43:30.210586 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/168564ee-9452-4aba-8391-d4f0bdaafa7b-istio-podinfo" pod "168564ee-9452-4aba-8391-d4f0bdaafa7b" (UID: "168564ee-9452-4aba-8391-d4f0bdaafa7b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 13:43:30.288791 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:43:30.288749 5024 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/168564ee-9452-4aba-8391-d4f0bdaafa7b-istio-podinfo\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:43:30.288791 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:43:30.288784 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lrg9v\" (UniqueName: \"kubernetes.io/projected/168564ee-9452-4aba-8391-d4f0bdaafa7b-kube-api-access-lrg9v\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:43:30.288791 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:43:30.288796 5024 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/168564ee-9452-4aba-8391-d4f0bdaafa7b-istio-token\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:43:30.289022 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:43:30.288805 5024 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/168564ee-9452-4aba-8391-d4f0bdaafa7b-istio-data\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:43:30.289022 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:43:30.288814 5024 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/168564ee-9452-4aba-8391-d4f0bdaafa7b-istio-envoy\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:43:30.372742 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:43:30.372642 5024 generic.go:356] "Generic (PLEG): container finished" podID="168564ee-9452-4aba-8391-d4f0bdaafa7b" containerID="538c794a1ce0474466bbbc53c119dac920c1451c9b9ab21f523c5f9e02e4a4fa" exitCode=0 Nov 03 13:43:30.372742 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:43:30.372696 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-1a18c8-openshift-default-67d7b5995c-z4csj" Nov 03 13:43:30.372742 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:43:30.372710 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-1a18c8-openshift-default-67d7b5995c-z4csj" event={"ID":"168564ee-9452-4aba-8391-d4f0bdaafa7b","Type":"ContainerDied","Data":"538c794a1ce0474466bbbc53c119dac920c1451c9b9ab21f523c5f9e02e4a4fa"} Nov 03 13:43:30.373021 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:43:30.372806 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-1a18c8-openshift-default-67d7b5995c-z4csj" event={"ID":"168564ee-9452-4aba-8391-d4f0bdaafa7b","Type":"ContainerDied","Data":"4f72ab5f299049e65c8efd2cb529c1aecfabbb1d8980d03c4aad6ffbe9ed1679"} Nov 03 13:43:30.373021 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:43:30.372837 5024 scope.go:122] "RemoveContainer" containerID="538c794a1ce0474466bbbc53c119dac920c1451c9b9ab21f523c5f9e02e4a4fa" Nov 03 13:43:30.394123 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:43:30.394100 5024 scope.go:122] "RemoveContainer" containerID="954b253b9e11610a4a8efa783f173c6d9a344093f76f73815f5b175465317dcf" Nov 03 13:43:30.406532 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:43:30.406512 5024 scope.go:122] "RemoveContainer" containerID="538c794a1ce0474466bbbc53c119dac920c1451c9b9ab21f523c5f9e02e4a4fa" Nov 03 13:43:30.406820 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:43:30.406799 5024 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"538c794a1ce0474466bbbc53c119dac920c1451c9b9ab21f523c5f9e02e4a4fa\": container with ID starting with 538c794a1ce0474466bbbc53c119dac920c1451c9b9ab21f523c5f9e02e4a4fa not found: ID does not exist" containerID="538c794a1ce0474466bbbc53c119dac920c1451c9b9ab21f523c5f9e02e4a4fa" Nov 03 13:43:30.406885 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:43:30.406832 5024 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"538c794a1ce0474466bbbc53c119dac920c1451c9b9ab21f523c5f9e02e4a4fa"} err="failed to get container status \"538c794a1ce0474466bbbc53c119dac920c1451c9b9ab21f523c5f9e02e4a4fa\": rpc error: code = NotFound desc = could not find container \"538c794a1ce0474466bbbc53c119dac920c1451c9b9ab21f523c5f9e02e4a4fa\": container with ID starting with 538c794a1ce0474466bbbc53c119dac920c1451c9b9ab21f523c5f9e02e4a4fa not found: ID does not exist" Nov 03 13:43:30.406885 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:43:30.406851 5024 scope.go:122] "RemoveContainer" containerID="954b253b9e11610a4a8efa783f173c6d9a344093f76f73815f5b175465317dcf" Nov 03 13:43:30.407113 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:43:30.407093 5024 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"954b253b9e11610a4a8efa783f173c6d9a344093f76f73815f5b175465317dcf\": container with ID starting with 954b253b9e11610a4a8efa783f173c6d9a344093f76f73815f5b175465317dcf not found: ID does not exist" containerID="954b253b9e11610a4a8efa783f173c6d9a344093f76f73815f5b175465317dcf" Nov 03 13:43:30.407171 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:43:30.407119 5024 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"954b253b9e11610a4a8efa783f173c6d9a344093f76f73815f5b175465317dcf"} err="failed to get container status \"954b253b9e11610a4a8efa783f173c6d9a344093f76f73815f5b175465317dcf\": rpc error: code = NotFound desc = could not find container \"954b253b9e11610a4a8efa783f173c6d9a344093f76f73815f5b175465317dcf\": container with ID starting with 954b253b9e11610a4a8efa783f173c6d9a344093f76f73815f5b175465317dcf not found: ID does not exist" Nov 03 13:43:30.430799 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:43:30.430771 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-1a18c8-openshift-default-67d7b5995c-z4csj"] Nov 03 13:43:30.433337 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:43:30.433311 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-1a18c8-openshift-default-67d7b5995c-z4csj"] Nov 03 13:43:32.112818 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:43:32.112781 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="168564ee-9452-4aba-8391-d4f0bdaafa7b" path="/var/lib/kubelet/pods/168564ee-9452-4aba-8391-d4f0bdaafa7b/volumes" Nov 03 13:43:39.513853 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:43:39.513767 5024 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d7mwz" containerName="extractor" Nov 03 13:43:41.385368 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:43:41.385336 5024 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6htht_cd91de08-084e-4667-b13f-2e07b3cff624/console-operator/1.log" Nov 03 13:43:41.396078 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:43:41.396050 5024 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6htht_cd91de08-084e-4667-b13f-2e07b3cff624/console-operator/1.log" Nov 03 13:43:45.937934 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:43:45.937886 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w0-8f8aa8/rate-test-model-8f8aa8-kserve-7786f67f6f-mbztf"] Nov 03 13:43:55.140306 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:43:55.134746 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-e440bc-8664c78f67-p6j9d"] Nov 03 13:43:55.140306 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:43:55.135413 5024 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-e440bc-8664c78f67-p6j9d" podUID="afc7261b-bfde-425a-8c19-f2f1d521f50a" containerName="maas-api" containerID="cri-o://41fbf33cbb9d5a3006a0c550c96fc8d034fecd8f08fc889c357699ea4e7a024d" gracePeriod=30 Nov 03 13:43:55.670815 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:43:55.670786 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-e440bc-8664c78f67-p6j9d" Nov 03 13:43:55.697473 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:43:55.697437 5024 generic.go:356] "Generic (PLEG): container finished" podID="afc7261b-bfde-425a-8c19-f2f1d521f50a" containerID="41fbf33cbb9d5a3006a0c550c96fc8d034fecd8f08fc889c357699ea4e7a024d" exitCode=0 Nov 03 13:43:55.697843 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:43:55.697800 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-e440bc-8664c78f67-p6j9d" event={"ID":"afc7261b-bfde-425a-8c19-f2f1d521f50a","Type":"ContainerDied","Data":"41fbf33cbb9d5a3006a0c550c96fc8d034fecd8f08fc889c357699ea4e7a024d"} Nov 03 13:43:55.698107 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:43:55.698086 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-e440bc-8664c78f67-p6j9d" event={"ID":"afc7261b-bfde-425a-8c19-f2f1d521f50a","Type":"ContainerDied","Data":"b47315165ab5b58e356ef4c6a4abe3cbd54afd2a0aacf525fa2e2428e6de99f3"} Nov 03 13:43:55.698279 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:43:55.698265 5024 scope.go:122] "RemoveContainer" containerID="41fbf33cbb9d5a3006a0c550c96fc8d034fecd8f08fc889c357699ea4e7a024d" Nov 03 13:43:55.698667 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:43:55.698652 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-e440bc-8664c78f67-p6j9d" Nov 03 13:43:55.760306 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:43:55.755433 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/afc7261b-bfde-425a-8c19-f2f1d521f50a-kube-api-access-7kj2q\" (UniqueName: \"kubernetes.io/projected/afc7261b-bfde-425a-8c19-f2f1d521f50a-kube-api-access-7kj2q\") pod \"afc7261b-bfde-425a-8c19-f2f1d521f50a\" (UID: \"afc7261b-bfde-425a-8c19-f2f1d521f50a\") " Nov 03 13:43:55.760306 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:43:55.755490 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/afc7261b-bfde-425a-8c19-f2f1d521f50a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/afc7261b-bfde-425a-8c19-f2f1d521f50a-metrics-certs\") pod \"afc7261b-bfde-425a-8c19-f2f1d521f50a\" (UID: \"afc7261b-bfde-425a-8c19-f2f1d521f50a\") " Nov 03 13:43:55.760306 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:43:55.755653 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/afc7261b-bfde-425a-8c19-f2f1d521f50a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/afc7261b-bfde-425a-8c19-f2f1d521f50a-maas-api-tls\") pod \"afc7261b-bfde-425a-8c19-f2f1d521f50a\" (UID: \"afc7261b-bfde-425a-8c19-f2f1d521f50a\") " Nov 03 13:43:55.760306 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:43:55.755688 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/afc7261b-bfde-425a-8c19-f2f1d521f50a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/afc7261b-bfde-425a-8c19-f2f1d521f50a-openshift-service-ca\") pod \"afc7261b-bfde-425a-8c19-f2f1d521f50a\" (UID: \"afc7261b-bfde-425a-8c19-f2f1d521f50a\") " Nov 03 13:43:55.831335 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:43:55.829486 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/afc7261b-bfde-425a-8c19-f2f1d521f50a-kube-api-access-7kj2q" pod "afc7261b-bfde-425a-8c19-f2f1d521f50a" (UID: "afc7261b-bfde-425a-8c19-f2f1d521f50a"). InnerVolumeSpecName "kube-api-access-7kj2q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:43:55.849580 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:43:55.845868 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/afc7261b-bfde-425a-8c19-f2f1d521f50a-maas-api-tls" pod "afc7261b-bfde-425a-8c19-f2f1d521f50a" (UID: "afc7261b-bfde-425a-8c19-f2f1d521f50a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 13:43:55.850079 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:43:55.850049 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/afc7261b-bfde-425a-8c19-f2f1d521f50a-metrics-certs" pod "afc7261b-bfde-425a-8c19-f2f1d521f50a" (UID: "afc7261b-bfde-425a-8c19-f2f1d521f50a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 13:43:55.850423 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:43:55.850374 5024 scope.go:122] "RemoveContainer" containerID="41fbf33cbb9d5a3006a0c550c96fc8d034fecd8f08fc889c357699ea4e7a024d" Nov 03 13:43:55.850686 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:43:55.757055 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/afc7261b-bfde-425a-8c19-f2f1d521f50a-openshift-service-ca" pod "afc7261b-bfde-425a-8c19-f2f1d521f50a" (UID: "afc7261b-bfde-425a-8c19-f2f1d521f50a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:43:55.857327 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:43:55.855729 5024 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41fbf33cbb9d5a3006a0c550c96fc8d034fecd8f08fc889c357699ea4e7a024d\": container with ID starting with 41fbf33cbb9d5a3006a0c550c96fc8d034fecd8f08fc889c357699ea4e7a024d not found: ID does not exist" containerID="41fbf33cbb9d5a3006a0c550c96fc8d034fecd8f08fc889c357699ea4e7a024d" Nov 03 13:43:55.857327 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:43:55.855791 5024 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41fbf33cbb9d5a3006a0c550c96fc8d034fecd8f08fc889c357699ea4e7a024d"} err="failed to get container status \"41fbf33cbb9d5a3006a0c550c96fc8d034fecd8f08fc889c357699ea4e7a024d\": rpc error: code = NotFound desc = could not find container \"41fbf33cbb9d5a3006a0c550c96fc8d034fecd8f08fc889c357699ea4e7a024d\": container with ID starting with 41fbf33cbb9d5a3006a0c550c96fc8d034fecd8f08fc889c357699ea4e7a024d not found: ID does not exist" Nov 03 13:43:55.875261 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:43:55.872926 5024 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/afc7261b-bfde-425a-8c19-f2f1d521f50a-maas-api-tls\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:43:55.875261 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:43:55.872974 5024 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/afc7261b-bfde-425a-8c19-f2f1d521f50a-openshift-service-ca\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:43:55.875261 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:43:55.872989 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7kj2q\" (UniqueName: \"kubernetes.io/projected/afc7261b-bfde-425a-8c19-f2f1d521f50a-kube-api-access-7kj2q\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:43:55.875261 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:43:55.873006 5024 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/afc7261b-bfde-425a-8c19-f2f1d521f50a-metrics-certs\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:43:56.209044 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:43:56.203115 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-e440bc-8664c78f67-p6j9d"] Nov 03 13:43:56.209044 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:43:56.203170 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-e440bc-8664c78f67-p6j9d"] Nov 03 13:43:56.209044 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:43:56.203197 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-c47670-54c8bb9b89-fzdnn"] Nov 03 13:43:58.109896 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:43:58.109865 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="afc7261b-bfde-425a-8c19-f2f1d521f50a" path="/var/lib/kubelet/pods/afc7261b-bfde-425a-8c19-f2f1d521f50a/volumes" Nov 03 13:44:00.962578 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:00.962531 5024 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w0-8f8aa8/rate-test-model-8f8aa8-kserve-7786f67f6f-mbztf" podUID="5fabbac7-721a-4f35-bd22-e86df0cecc62" containerName="main" containerID="cri-o://5605d9b3c7a996f21b2a1c473766ca440a976ce0ca56aa746b80040cca17721c" gracePeriod=45 Nov 03 13:44:01.203781 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:01.203733 5024 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-c47670-54c8bb9b89-fzdnn" podUID="7a30a13a-965d-4291-9acd-f109a6ead710" containerName="payload-processing" containerID="cri-o://ccdfa1856a6bba533687ddc2d36f10d15651ea5354a78e8a75ad8fb767081af5" gracePeriod=25 Nov 03 13:44:01.246426 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:01.246396 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-8f8aa8/rate-test-model-8f8aa8-kserve-7786f67f6f-mbztf" Nov 03 13:44:01.338234 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:01.338197 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5fabbac7-721a-4f35-bd22-e86df0cecc62-tls-certs\" (UniqueName: \"kubernetes.io/secret/5fabbac7-721a-4f35-bd22-e86df0cecc62-tls-certs\") pod \"5fabbac7-721a-4f35-bd22-e86df0cecc62\" (UID: \"5fabbac7-721a-4f35-bd22-e86df0cecc62\") " Nov 03 13:44:01.338234 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:01.338241 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5fabbac7-721a-4f35-bd22-e86df0cecc62-dshm\" (UniqueName: \"kubernetes.io/empty-dir/5fabbac7-721a-4f35-bd22-e86df0cecc62-dshm\") pod \"5fabbac7-721a-4f35-bd22-e86df0cecc62\" (UID: \"5fabbac7-721a-4f35-bd22-e86df0cecc62\") " Nov 03 13:44:01.338506 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:01.338405 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5fabbac7-721a-4f35-bd22-e86df0cecc62-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5fabbac7-721a-4f35-bd22-e86df0cecc62-model-cache\") pod \"5fabbac7-721a-4f35-bd22-e86df0cecc62\" (UID: \"5fabbac7-721a-4f35-bd22-e86df0cecc62\") " Nov 03 13:44:01.338506 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:01.338473 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5fabbac7-721a-4f35-bd22-e86df0cecc62-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5fabbac7-721a-4f35-bd22-e86df0cecc62-tmp-dir\") pod \"5fabbac7-721a-4f35-bd22-e86df0cecc62\" (UID: \"5fabbac7-721a-4f35-bd22-e86df0cecc62\") " Nov 03 13:44:01.338666 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:01.338522 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5fabbac7-721a-4f35-bd22-e86df0cecc62-kube-api-access-dmvw6\" (UniqueName: \"kubernetes.io/projected/5fabbac7-721a-4f35-bd22-e86df0cecc62-kube-api-access-dmvw6\") pod \"5fabbac7-721a-4f35-bd22-e86df0cecc62\" (UID: \"5fabbac7-721a-4f35-bd22-e86df0cecc62\") " Nov 03 13:44:01.338666 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:01.338622 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5fabbac7-721a-4f35-bd22-e86df0cecc62-home\" (UniqueName: \"kubernetes.io/empty-dir/5fabbac7-721a-4f35-bd22-e86df0cecc62-home\") pod \"5fabbac7-721a-4f35-bd22-e86df0cecc62\" (UID: \"5fabbac7-721a-4f35-bd22-e86df0cecc62\") " Nov 03 13:44:01.338782 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:01.338707 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fabbac7-721a-4f35-bd22-e86df0cecc62-model-cache" pod "5fabbac7-721a-4f35-bd22-e86df0cecc62" (UID: "5fabbac7-721a-4f35-bd22-e86df0cecc62"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:44:01.338782 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:01.338727 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fabbac7-721a-4f35-bd22-e86df0cecc62-tmp-dir" pod "5fabbac7-721a-4f35-bd22-e86df0cecc62" (UID: "5fabbac7-721a-4f35-bd22-e86df0cecc62"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:44:01.338893 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:01.338871 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fabbac7-721a-4f35-bd22-e86df0cecc62-home" pod "5fabbac7-721a-4f35-bd22-e86df0cecc62" (UID: "5fabbac7-721a-4f35-bd22-e86df0cecc62"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:44:01.339207 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:01.339188 5024 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5fabbac7-721a-4f35-bd22-e86df0cecc62-home\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:44:01.339248 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:01.339216 5024 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5fabbac7-721a-4f35-bd22-e86df0cecc62-model-cache\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:44:01.339248 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:01.339234 5024 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5fabbac7-721a-4f35-bd22-e86df0cecc62-tmp-dir\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:44:01.340963 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:01.340937 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5fabbac7-721a-4f35-bd22-e86df0cecc62-tls-certs" pod "5fabbac7-721a-4f35-bd22-e86df0cecc62" (UID: "5fabbac7-721a-4f35-bd22-e86df0cecc62"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 13:44:01.341140 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:01.341124 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fabbac7-721a-4f35-bd22-e86df0cecc62-dshm" pod "5fabbac7-721a-4f35-bd22-e86df0cecc62" (UID: "5fabbac7-721a-4f35-bd22-e86df0cecc62"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:44:01.341245 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:01.341226 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5fabbac7-721a-4f35-bd22-e86df0cecc62-kube-api-access-dmvw6" pod "5fabbac7-721a-4f35-bd22-e86df0cecc62" (UID: "5fabbac7-721a-4f35-bd22-e86df0cecc62"). InnerVolumeSpecName "kube-api-access-dmvw6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:44:01.440607 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:01.440567 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dmvw6\" (UniqueName: \"kubernetes.io/projected/5fabbac7-721a-4f35-bd22-e86df0cecc62-kube-api-access-dmvw6\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:44:01.440607 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:01.440601 5024 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5fabbac7-721a-4f35-bd22-e86df0cecc62-tls-certs\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:44:01.440607 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:01.440611 5024 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5fabbac7-721a-4f35-bd22-e86df0cecc62-dshm\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:44:01.780118 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:01.780084 5024 generic.go:356] "Generic (PLEG): container finished" podID="5fabbac7-721a-4f35-bd22-e86df0cecc62" containerID="5605d9b3c7a996f21b2a1c473766ca440a976ce0ca56aa746b80040cca17721c" exitCode=0 Nov 03 13:44:01.780364 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:01.780144 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-8f8aa8/rate-test-model-8f8aa8-kserve-7786f67f6f-mbztf" Nov 03 13:44:01.780364 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:01.780174 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-8f8aa8/rate-test-model-8f8aa8-kserve-7786f67f6f-mbztf" event={"ID":"5fabbac7-721a-4f35-bd22-e86df0cecc62","Type":"ContainerDied","Data":"5605d9b3c7a996f21b2a1c473766ca440a976ce0ca56aa746b80040cca17721c"} Nov 03 13:44:01.780364 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:01.780229 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-8f8aa8/rate-test-model-8f8aa8-kserve-7786f67f6f-mbztf" event={"ID":"5fabbac7-721a-4f35-bd22-e86df0cecc62","Type":"ContainerDied","Data":"e464dbcc4536d2ceaca73ff3194216de5519155e5750f7e28b3182f7835e4de6"} Nov 03 13:44:01.780364 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:01.780241 5024 scope.go:122] "RemoveContainer" containerID="5605d9b3c7a996f21b2a1c473766ca440a976ce0ca56aa746b80040cca17721c" Nov 03 13:44:01.790351 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:01.790334 5024 scope.go:122] "RemoveContainer" containerID="5605d9b3c7a996f21b2a1c473766ca440a976ce0ca56aa746b80040cca17721c" Nov 03 13:44:01.790658 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:44:01.790631 5024 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5605d9b3c7a996f21b2a1c473766ca440a976ce0ca56aa746b80040cca17721c\": container with ID starting with 5605d9b3c7a996f21b2a1c473766ca440a976ce0ca56aa746b80040cca17721c not found: ID does not exist" containerID="5605d9b3c7a996f21b2a1c473766ca440a976ce0ca56aa746b80040cca17721c" Nov 03 13:44:01.790778 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:01.790666 5024 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5605d9b3c7a996f21b2a1c473766ca440a976ce0ca56aa746b80040cca17721c"} err="failed to get container status \"5605d9b3c7a996f21b2a1c473766ca440a976ce0ca56aa746b80040cca17721c\": rpc error: code = NotFound desc = could not find container \"5605d9b3c7a996f21b2a1c473766ca440a976ce0ca56aa746b80040cca17721c\": container with ID starting with 5605d9b3c7a996f21b2a1c473766ca440a976ce0ca56aa746b80040cca17721c not found: ID does not exist" Nov 03 13:44:01.820109 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:01.820081 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w0-8f8aa8/rate-test-model-8f8aa8-kserve-7786f67f6f-mbztf"] Nov 03 13:44:01.828522 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:01.828498 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w0-8f8aa8/rate-test-model-8f8aa8-kserve-7786f67f6f-mbztf"] Nov 03 13:44:02.108670 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:02.108641 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5fabbac7-721a-4f35-bd22-e86df0cecc62" path="/var/lib/kubelet/pods/5fabbac7-721a-4f35-bd22-e86df0cecc62/volumes" Nov 03 13:44:04.910981 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:04.910947 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-lftt4"] Nov 03 13:44:04.911480 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:04.911167 5024 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-lftt4" podUID="99af7f3f-6179-495a-a526-4cce51075588" containerName="istio-proxy" containerID="cri-o://b2d7fc25419ec27517c330dbbbea8e457c53a3c0f1f4ed49c153f9b9bcfeb43d" gracePeriod=30 Nov 03 13:44:04.929750 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:04.929721 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-zpj4t"] Nov 03 13:44:04.929952 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:04.929931 5024 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-zpj4t" podUID="95aa021f-6ec9-4f53-810a-58eb1d059081" containerName="istio-proxy" containerID="cri-o://1065c59e4d759c691604c43c01c5947c75b0be37a4d1303512593a72cf489045" gracePeriod=30 Nov 03 13:44:04.932278 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:04.932254 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-d2rlz"] Nov 03 13:44:04.932460 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:04.932440 5024 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-d2rlz" podUID="af43a1c8-fad2-4b80-a0e9-65ab23c3612e" containerName="istio-proxy" containerID="cri-o://638d0cf057c395a54491109576c2829707ad1164879abf8ed2ff73b9afcd49d1" gracePeriod=30 Nov 03 13:44:05.670930 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:05.670886 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-c47670-54c8bb9b89-fzdnn" podUID="7a30a13a-965d-4291-9acd-f109a6ead710" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 13:44:06.942567 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:06.942532 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-c47670-openshift-default-7f67b9bfc-rv4l7"] Nov 03 13:44:06.942980 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:06.942789 5024 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-c47670-openshift-default-7f67b9bfc-rv4l7" podUID="160d4b07-2558-4e4d-afb7-a7d509cc68ca" containerName="istio-proxy" containerID="cri-o://4245b3cae40c16aaa5bb911f6a4236f951bba5bd44441134ccc19ae9ba252f86" gracePeriod=30 Nov 03 13:44:07.189555 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:07.189514 5024 patch_prober.go:29] interesting pod/e2e-extmod-a-c47670-openshift-default-7f67b9bfc-rv4l7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 13:44:07.189715 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:07.189580 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-c47670-openshift-default-7f67b9bfc-rv4l7" podUID="160d4b07-2558-4e4d-afb7-a7d509cc68ca" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 13:44:09.119501 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:09.119430 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-6865svtfs"] Nov 03 13:44:09.519452 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:44:09.519353 5024 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d7mwz" containerName="extractor" Nov 03 13:44:09.527317 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:09.527247 5024 patch_prober.go:29] interesting pod/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-d2rlz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 13:44:09.527485 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:09.527347 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-d2rlz" podUID="af43a1c8-fad2-4b80-a0e9-65ab23c3612e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 13:44:10.248673 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:10.248641 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-lftt4" Nov 03 13:44:10.335878 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:10.335846 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-zpj4t" Nov 03 13:44:10.337952 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:10.337927 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-d2rlz" Nov 03 13:44:10.384457 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:10.384423 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99af7f3f-6179-495a-a526-4cce51075588-kube-api-access-724pq\" (UniqueName: \"kubernetes.io/projected/99af7f3f-6179-495a-a526-4cce51075588-kube-api-access-724pq\") pod \"99af7f3f-6179-495a-a526-4cce51075588\" (UID: \"99af7f3f-6179-495a-a526-4cce51075588\") " Nov 03 13:44:10.385476 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:10.385449 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99af7f3f-6179-495a-a526-4cce51075588-istio-token\" (UniqueName: \"kubernetes.io/projected/99af7f3f-6179-495a-a526-4cce51075588-istio-token\") pod \"99af7f3f-6179-495a-a526-4cce51075588\" (UID: \"99af7f3f-6179-495a-a526-4cce51075588\") " Nov 03 13:44:10.385747 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:10.385730 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/99af7f3f-6179-495a-a526-4cce51075588-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99af7f3f-6179-495a-a526-4cce51075588-istio-podinfo\") pod \"99af7f3f-6179-495a-a526-4cce51075588\" (UID: \"99af7f3f-6179-495a-a526-4cce51075588\") " Nov 03 13:44:10.385940 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:10.385925 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99af7f3f-6179-495a-a526-4cce51075588-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99af7f3f-6179-495a-a526-4cce51075588-istio-envoy\") pod \"99af7f3f-6179-495a-a526-4cce51075588\" (UID: \"99af7f3f-6179-495a-a526-4cce51075588\") " Nov 03 13:44:10.386123 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:10.386093 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99af7f3f-6179-495a-a526-4cce51075588-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99af7f3f-6179-495a-a526-4cce51075588-workload-certs\") pod \"99af7f3f-6179-495a-a526-4cce51075588\" (UID: \"99af7f3f-6179-495a-a526-4cce51075588\") " Nov 03 13:44:10.386473 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:10.386441 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99af7f3f-6179-495a-a526-4cce51075588-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99af7f3f-6179-495a-a526-4cce51075588-istio-data\") pod \"99af7f3f-6179-495a-a526-4cce51075588\" (UID: \"99af7f3f-6179-495a-a526-4cce51075588\") " Nov 03 13:44:10.386705 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:10.386675 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99af7f3f-6179-495a-a526-4cce51075588-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99af7f3f-6179-495a-a526-4cce51075588-credential-socket\") pod \"99af7f3f-6179-495a-a526-4cce51075588\" (UID: \"99af7f3f-6179-495a-a526-4cce51075588\") " Nov 03 13:44:10.386891 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:10.386876 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/99af7f3f-6179-495a-a526-4cce51075588-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99af7f3f-6179-495a-a526-4cce51075588-istiod-ca-cert\") pod \"99af7f3f-6179-495a-a526-4cce51075588\" (UID: \"99af7f3f-6179-495a-a526-4cce51075588\") " Nov 03 13:44:10.387036 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:10.387021 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99af7f3f-6179-495a-a526-4cce51075588-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99af7f3f-6179-495a-a526-4cce51075588-workload-socket\") pod \"99af7f3f-6179-495a-a526-4cce51075588\" (UID: \"99af7f3f-6179-495a-a526-4cce51075588\") " Nov 03 13:44:10.388485 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:10.388016 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99af7f3f-6179-495a-a526-4cce51075588-workload-certs" pod "99af7f3f-6179-495a-a526-4cce51075588" (UID: "99af7f3f-6179-495a-a526-4cce51075588"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:44:10.388485 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:10.388036 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99af7f3f-6179-495a-a526-4cce51075588-credential-socket" pod "99af7f3f-6179-495a-a526-4cce51075588" (UID: "99af7f3f-6179-495a-a526-4cce51075588"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:44:10.388485 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:10.388244 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99af7f3f-6179-495a-a526-4cce51075588-istio-data" pod "99af7f3f-6179-495a-a526-4cce51075588" (UID: "99af7f3f-6179-495a-a526-4cce51075588"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:44:10.388485 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:10.388392 5024 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99af7f3f-6179-495a-a526-4cce51075588-workload-certs\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:44:10.388485 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:10.388416 5024 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99af7f3f-6179-495a-a526-4cce51075588-credential-socket\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:44:10.388485 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:10.388449 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99af7f3f-6179-495a-a526-4cce51075588-istiod-ca-cert" pod "99af7f3f-6179-495a-a526-4cce51075588" (UID: "99af7f3f-6179-495a-a526-4cce51075588"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:44:10.388977 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:10.388808 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99af7f3f-6179-495a-a526-4cce51075588-workload-socket" pod "99af7f3f-6179-495a-a526-4cce51075588" (UID: "99af7f3f-6179-495a-a526-4cce51075588"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:44:10.390777 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:10.390743 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99af7f3f-6179-495a-a526-4cce51075588-istio-envoy" pod "99af7f3f-6179-495a-a526-4cce51075588" (UID: "99af7f3f-6179-495a-a526-4cce51075588"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:44:10.390876 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:10.390827 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99af7f3f-6179-495a-a526-4cce51075588-istio-token" pod "99af7f3f-6179-495a-a526-4cce51075588" (UID: "99af7f3f-6179-495a-a526-4cce51075588"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:44:10.395453 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:10.395420 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/99af7f3f-6179-495a-a526-4cce51075588-istio-podinfo" pod "99af7f3f-6179-495a-a526-4cce51075588" (UID: "99af7f3f-6179-495a-a526-4cce51075588"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 13:44:10.396181 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:10.396158 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99af7f3f-6179-495a-a526-4cce51075588-kube-api-access-724pq" pod "99af7f3f-6179-495a-a526-4cce51075588" (UID: "99af7f3f-6179-495a-a526-4cce51075588"). InnerVolumeSpecName "kube-api-access-724pq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:44:10.489773 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:10.489673 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af43a1c8-fad2-4b80-a0e9-65ab23c3612e-istio-token\" (UniqueName: \"kubernetes.io/projected/af43a1c8-fad2-4b80-a0e9-65ab23c3612e-istio-token\") pod \"af43a1c8-fad2-4b80-a0e9-65ab23c3612e\" (UID: \"af43a1c8-fad2-4b80-a0e9-65ab23c3612e\") " Nov 03 13:44:10.489773 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:10.489746 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af43a1c8-fad2-4b80-a0e9-65ab23c3612e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af43a1c8-fad2-4b80-a0e9-65ab23c3612e-istio-data\") pod \"af43a1c8-fad2-4b80-a0e9-65ab23c3612e\" (UID: \"af43a1c8-fad2-4b80-a0e9-65ab23c3612e\") " Nov 03 13:44:10.489970 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:10.489854 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/95aa021f-6ec9-4f53-810a-58eb1d059081-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95aa021f-6ec9-4f53-810a-58eb1d059081-istiod-ca-cert\") pod \"95aa021f-6ec9-4f53-810a-58eb1d059081\" (UID: \"95aa021f-6ec9-4f53-810a-58eb1d059081\") " Nov 03 13:44:10.489970 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:10.489910 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95aa021f-6ec9-4f53-810a-58eb1d059081-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95aa021f-6ec9-4f53-810a-58eb1d059081-workload-socket\") pod \"95aa021f-6ec9-4f53-810a-58eb1d059081\" (UID: \"95aa021f-6ec9-4f53-810a-58eb1d059081\") " Nov 03 13:44:10.489970 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:10.489944 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/af43a1c8-fad2-4b80-a0e9-65ab23c3612e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af43a1c8-fad2-4b80-a0e9-65ab23c3612e-istio-podinfo\") pod \"af43a1c8-fad2-4b80-a0e9-65ab23c3612e\" (UID: \"af43a1c8-fad2-4b80-a0e9-65ab23c3612e\") " Nov 03 13:44:10.490092 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:10.490002 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af43a1c8-fad2-4b80-a0e9-65ab23c3612e-kube-api-access-m55fg\" (UniqueName: \"kubernetes.io/projected/af43a1c8-fad2-4b80-a0e9-65ab23c3612e-kube-api-access-m55fg\") pod \"af43a1c8-fad2-4b80-a0e9-65ab23c3612e\" (UID: \"af43a1c8-fad2-4b80-a0e9-65ab23c3612e\") " Nov 03 13:44:10.490092 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:10.490021 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af43a1c8-fad2-4b80-a0e9-65ab23c3612e-istio-data" pod "af43a1c8-fad2-4b80-a0e9-65ab23c3612e" (UID: "af43a1c8-fad2-4b80-a0e9-65ab23c3612e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:44:10.490092 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:10.490043 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af43a1c8-fad2-4b80-a0e9-65ab23c3612e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af43a1c8-fad2-4b80-a0e9-65ab23c3612e-credential-socket\") pod \"af43a1c8-fad2-4b80-a0e9-65ab23c3612e\" (UID: \"af43a1c8-fad2-4b80-a0e9-65ab23c3612e\") " Nov 03 13:44:10.490092 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:10.490073 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af43a1c8-fad2-4b80-a0e9-65ab23c3612e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af43a1c8-fad2-4b80-a0e9-65ab23c3612e-workload-certs\") pod \"af43a1c8-fad2-4b80-a0e9-65ab23c3612e\" (UID: \"af43a1c8-fad2-4b80-a0e9-65ab23c3612e\") " Nov 03 13:44:10.490303 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:10.490115 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95aa021f-6ec9-4f53-810a-58eb1d059081-kube-api-access-5kbrh\" (UniqueName: \"kubernetes.io/projected/95aa021f-6ec9-4f53-810a-58eb1d059081-kube-api-access-5kbrh\") pod \"95aa021f-6ec9-4f53-810a-58eb1d059081\" (UID: \"95aa021f-6ec9-4f53-810a-58eb1d059081\") " Nov 03 13:44:10.490303 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:10.490147 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95aa021f-6ec9-4f53-810a-58eb1d059081-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95aa021f-6ec9-4f53-810a-58eb1d059081-workload-certs\") pod \"95aa021f-6ec9-4f53-810a-58eb1d059081\" (UID: \"95aa021f-6ec9-4f53-810a-58eb1d059081\") " Nov 03 13:44:10.490303 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:10.490249 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95aa021f-6ec9-4f53-810a-58eb1d059081-workload-socket" pod "95aa021f-6ec9-4f53-810a-58eb1d059081" (UID: "95aa021f-6ec9-4f53-810a-58eb1d059081"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:44:10.490467 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:10.490449 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95aa021f-6ec9-4f53-810a-58eb1d059081-workload-certs" pod "95aa021f-6ec9-4f53-810a-58eb1d059081" (UID: "95aa021f-6ec9-4f53-810a-58eb1d059081"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:44:10.490977 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:10.490516 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95aa021f-6ec9-4f53-810a-58eb1d059081-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95aa021f-6ec9-4f53-810a-58eb1d059081-istio-envoy\") pod \"95aa021f-6ec9-4f53-810a-58eb1d059081\" (UID: \"95aa021f-6ec9-4f53-810a-58eb1d059081\") " Nov 03 13:44:10.490977 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:10.490542 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95aa021f-6ec9-4f53-810a-58eb1d059081-istiod-ca-cert" pod "95aa021f-6ec9-4f53-810a-58eb1d059081" (UID: "95aa021f-6ec9-4f53-810a-58eb1d059081"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:44:10.490977 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:10.490579 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af43a1c8-fad2-4b80-a0e9-65ab23c3612e-credential-socket" pod "af43a1c8-fad2-4b80-a0e9-65ab23c3612e" (UID: "af43a1c8-fad2-4b80-a0e9-65ab23c3612e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:44:10.490977 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:10.490633 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af43a1c8-fad2-4b80-a0e9-65ab23c3612e-workload-certs" pod "af43a1c8-fad2-4b80-a0e9-65ab23c3612e" (UID: "af43a1c8-fad2-4b80-a0e9-65ab23c3612e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:44:10.490977 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:10.490644 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af43a1c8-fad2-4b80-a0e9-65ab23c3612e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af43a1c8-fad2-4b80-a0e9-65ab23c3612e-workload-socket\") pod \"af43a1c8-fad2-4b80-a0e9-65ab23c3612e\" (UID: \"af43a1c8-fad2-4b80-a0e9-65ab23c3612e\") " Nov 03 13:44:10.490977 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:10.490760 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af43a1c8-fad2-4b80-a0e9-65ab23c3612e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af43a1c8-fad2-4b80-a0e9-65ab23c3612e-istio-envoy\") pod \"af43a1c8-fad2-4b80-a0e9-65ab23c3612e\" (UID: \"af43a1c8-fad2-4b80-a0e9-65ab23c3612e\") " Nov 03 13:44:10.490977 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:10.490832 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af43a1c8-fad2-4b80-a0e9-65ab23c3612e-workload-socket" pod "af43a1c8-fad2-4b80-a0e9-65ab23c3612e" (UID: "af43a1c8-fad2-4b80-a0e9-65ab23c3612e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:44:10.490977 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:10.490890 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/95aa021f-6ec9-4f53-810a-58eb1d059081-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95aa021f-6ec9-4f53-810a-58eb1d059081-istio-podinfo\") pod \"95aa021f-6ec9-4f53-810a-58eb1d059081\" (UID: \"95aa021f-6ec9-4f53-810a-58eb1d059081\") " Nov 03 13:44:10.490977 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:10.490931 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/af43a1c8-fad2-4b80-a0e9-65ab23c3612e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af43a1c8-fad2-4b80-a0e9-65ab23c3612e-istiod-ca-cert\") pod \"af43a1c8-fad2-4b80-a0e9-65ab23c3612e\" (UID: \"af43a1c8-fad2-4b80-a0e9-65ab23c3612e\") " Nov 03 13:44:10.491496 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:10.490993 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95aa021f-6ec9-4f53-810a-58eb1d059081-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95aa021f-6ec9-4f53-810a-58eb1d059081-istio-data\") pod \"95aa021f-6ec9-4f53-810a-58eb1d059081\" (UID: \"95aa021f-6ec9-4f53-810a-58eb1d059081\") " Nov 03 13:44:10.491496 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:10.491055 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95aa021f-6ec9-4f53-810a-58eb1d059081-istio-token\" (UniqueName: \"kubernetes.io/projected/95aa021f-6ec9-4f53-810a-58eb1d059081-istio-token\") pod \"95aa021f-6ec9-4f53-810a-58eb1d059081\" (UID: \"95aa021f-6ec9-4f53-810a-58eb1d059081\") " Nov 03 13:44:10.491496 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:10.491107 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95aa021f-6ec9-4f53-810a-58eb1d059081-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95aa021f-6ec9-4f53-810a-58eb1d059081-credential-socket\") pod \"95aa021f-6ec9-4f53-810a-58eb1d059081\" (UID: \"95aa021f-6ec9-4f53-810a-58eb1d059081\") " Nov 03 13:44:10.491705 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:10.491685 5024 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95aa021f-6ec9-4f53-810a-58eb1d059081-istiod-ca-cert\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:44:10.491770 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:10.491711 5024 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95aa021f-6ec9-4f53-810a-58eb1d059081-workload-socket\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:44:10.491770 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:10.491726 5024 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99af7f3f-6179-495a-a526-4cce51075588-istio-data\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:44:10.491770 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:10.491741 5024 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af43a1c8-fad2-4b80-a0e9-65ab23c3612e-credential-socket\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:44:10.491770 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:10.491755 5024 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af43a1c8-fad2-4b80-a0e9-65ab23c3612e-workload-certs\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:44:10.491972 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:10.491769 5024 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95aa021f-6ec9-4f53-810a-58eb1d059081-workload-certs\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:44:10.491972 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:10.491783 5024 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af43a1c8-fad2-4b80-a0e9-65ab23c3612e-workload-socket\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:44:10.491972 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:10.491797 5024 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99af7f3f-6179-495a-a526-4cce51075588-istiod-ca-cert\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:44:10.491972 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:10.491811 5024 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99af7f3f-6179-495a-a526-4cce51075588-workload-socket\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:44:10.491972 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:10.491826 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-724pq\" (UniqueName: \"kubernetes.io/projected/99af7f3f-6179-495a-a526-4cce51075588-kube-api-access-724pq\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:44:10.491972 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:10.491841 5024 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99af7f3f-6179-495a-a526-4cce51075588-istio-token\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:44:10.491972 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:10.491855 5024 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99af7f3f-6179-495a-a526-4cce51075588-istio-podinfo\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:44:10.491972 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:10.491868 5024 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99af7f3f-6179-495a-a526-4cce51075588-istio-envoy\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:44:10.491972 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:10.491882 5024 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af43a1c8-fad2-4b80-a0e9-65ab23c3612e-istio-data\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:44:10.492547 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:10.492406 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af43a1c8-fad2-4b80-a0e9-65ab23c3612e-istiod-ca-cert" pod "af43a1c8-fad2-4b80-a0e9-65ab23c3612e" (UID: "af43a1c8-fad2-4b80-a0e9-65ab23c3612e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:44:10.492698 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:10.492668 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af43a1c8-fad2-4b80-a0e9-65ab23c3612e-istio-token" pod "af43a1c8-fad2-4b80-a0e9-65ab23c3612e" (UID: "af43a1c8-fad2-4b80-a0e9-65ab23c3612e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:44:10.492838 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:10.492718 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95aa021f-6ec9-4f53-810a-58eb1d059081-credential-socket" pod "95aa021f-6ec9-4f53-810a-58eb1d059081" (UID: "95aa021f-6ec9-4f53-810a-58eb1d059081"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:44:10.492838 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:10.492745 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/af43a1c8-fad2-4b80-a0e9-65ab23c3612e-istio-podinfo" pod "af43a1c8-fad2-4b80-a0e9-65ab23c3612e" (UID: "af43a1c8-fad2-4b80-a0e9-65ab23c3612e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 13:44:10.492838 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:10.492721 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95aa021f-6ec9-4f53-810a-58eb1d059081-kube-api-access-5kbrh" pod "95aa021f-6ec9-4f53-810a-58eb1d059081" (UID: "95aa021f-6ec9-4f53-810a-58eb1d059081"). InnerVolumeSpecName "kube-api-access-5kbrh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:44:10.493042 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:10.492968 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95aa021f-6ec9-4f53-810a-58eb1d059081-istio-data" pod "95aa021f-6ec9-4f53-810a-58eb1d059081" (UID: "95aa021f-6ec9-4f53-810a-58eb1d059081"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:44:10.493115 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:10.493047 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95aa021f-6ec9-4f53-810a-58eb1d059081-istio-envoy" pod "95aa021f-6ec9-4f53-810a-58eb1d059081" (UID: "95aa021f-6ec9-4f53-810a-58eb1d059081"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:44:10.493236 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:10.493216 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af43a1c8-fad2-4b80-a0e9-65ab23c3612e-kube-api-access-m55fg" pod "af43a1c8-fad2-4b80-a0e9-65ab23c3612e" (UID: "af43a1c8-fad2-4b80-a0e9-65ab23c3612e"). InnerVolumeSpecName "kube-api-access-m55fg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:44:10.493584 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:10.493568 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af43a1c8-fad2-4b80-a0e9-65ab23c3612e-istio-envoy" pod "af43a1c8-fad2-4b80-a0e9-65ab23c3612e" (UID: "af43a1c8-fad2-4b80-a0e9-65ab23c3612e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:44:10.494411 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:10.494386 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/95aa021f-6ec9-4f53-810a-58eb1d059081-istio-podinfo" pod "95aa021f-6ec9-4f53-810a-58eb1d059081" (UID: "95aa021f-6ec9-4f53-810a-58eb1d059081"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 13:44:10.494631 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:10.494612 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95aa021f-6ec9-4f53-810a-58eb1d059081-istio-token" pod "95aa021f-6ec9-4f53-810a-58eb1d059081" (UID: "95aa021f-6ec9-4f53-810a-58eb1d059081"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:44:10.593169 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:10.593129 5024 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af43a1c8-fad2-4b80-a0e9-65ab23c3612e-istio-envoy\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:44:10.593169 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:10.593165 5024 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95aa021f-6ec9-4f53-810a-58eb1d059081-istio-podinfo\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:44:10.593169 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:10.593174 5024 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af43a1c8-fad2-4b80-a0e9-65ab23c3612e-istiod-ca-cert\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:44:10.593446 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:10.593183 5024 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95aa021f-6ec9-4f53-810a-58eb1d059081-istio-data\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:44:10.593446 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:10.593194 5024 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/95aa021f-6ec9-4f53-810a-58eb1d059081-istio-token\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:44:10.593446 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:10.593202 5024 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95aa021f-6ec9-4f53-810a-58eb1d059081-credential-socket\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:44:10.593446 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:10.593214 5024 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af43a1c8-fad2-4b80-a0e9-65ab23c3612e-istio-token\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:44:10.593446 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:10.593226 5024 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af43a1c8-fad2-4b80-a0e9-65ab23c3612e-istio-podinfo\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:44:10.593446 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:10.593236 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m55fg\" (UniqueName: \"kubernetes.io/projected/af43a1c8-fad2-4b80-a0e9-65ab23c3612e-kube-api-access-m55fg\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:44:10.593446 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:10.593245 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5kbrh\" (UniqueName: \"kubernetes.io/projected/95aa021f-6ec9-4f53-810a-58eb1d059081-kube-api-access-5kbrh\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:44:10.593446 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:10.593253 5024 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95aa021f-6ec9-4f53-810a-58eb1d059081-istio-envoy\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:44:10.866078 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:10.866041 5024 generic.go:356] "Generic (PLEG): container finished" podID="99af7f3f-6179-495a-a526-4cce51075588" containerID="b2d7fc25419ec27517c330dbbbea8e457c53a3c0f1f4ed49c153f9b9bcfeb43d" exitCode=0 Nov 03 13:44:10.866246 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:10.866100 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-lftt4" Nov 03 13:44:10.866246 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:10.866128 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-lftt4" event={"ID":"99af7f3f-6179-495a-a526-4cce51075588","Type":"ContainerDied","Data":"b2d7fc25419ec27517c330dbbbea8e457c53a3c0f1f4ed49c153f9b9bcfeb43d"} Nov 03 13:44:10.866246 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:10.866183 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-lftt4" event={"ID":"99af7f3f-6179-495a-a526-4cce51075588","Type":"ContainerDied","Data":"a2aafd7ddec973236d4d96dfc439d40c88144b6db061b1f298b48300a37fd6d2"} Nov 03 13:44:10.866246 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:10.866204 5024 scope.go:122] "RemoveContainer" containerID="b2d7fc25419ec27517c330dbbbea8e457c53a3c0f1f4ed49c153f9b9bcfeb43d" Nov 03 13:44:10.867954 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:10.867924 5024 generic.go:356] "Generic (PLEG): container finished" podID="af43a1c8-fad2-4b80-a0e9-65ab23c3612e" containerID="638d0cf057c395a54491109576c2829707ad1164879abf8ed2ff73b9afcd49d1" exitCode=0 Nov 03 13:44:10.868080 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:10.867967 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-d2rlz" Nov 03 13:44:10.868080 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:10.867965 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-d2rlz" event={"ID":"af43a1c8-fad2-4b80-a0e9-65ab23c3612e","Type":"ContainerDied","Data":"638d0cf057c395a54491109576c2829707ad1164879abf8ed2ff73b9afcd49d1"} Nov 03 13:44:10.868184 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:10.868125 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-d2rlz" event={"ID":"af43a1c8-fad2-4b80-a0e9-65ab23c3612e","Type":"ContainerDied","Data":"4a1615012fa67730fd4a0c11d3e10117077b9ffe5d8f2feb0a8feea72c20a1da"} Nov 03 13:44:10.878210 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:10.876576 5024 generic.go:356] "Generic (PLEG): container finished" podID="95aa021f-6ec9-4f53-810a-58eb1d059081" containerID="1065c59e4d759c691604c43c01c5947c75b0be37a4d1303512593a72cf489045" exitCode=0 Nov 03 13:44:10.878210 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:10.876663 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-zpj4t" event={"ID":"95aa021f-6ec9-4f53-810a-58eb1d059081","Type":"ContainerDied","Data":"1065c59e4d759c691604c43c01c5947c75b0be37a4d1303512593a72cf489045"} Nov 03 13:44:10.878210 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:10.876751 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-zpj4t" event={"ID":"95aa021f-6ec9-4f53-810a-58eb1d059081","Type":"ContainerDied","Data":"0678890b4999b49b89178ef1a43c7d8eb6da957935bb788eb9ebe183fe0d7fba"} Nov 03 13:44:10.878210 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:10.876861 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-zpj4t" Nov 03 13:44:10.904026 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:10.903999 5024 scope.go:122] "RemoveContainer" containerID="b2d7fc25419ec27517c330dbbbea8e457c53a3c0f1f4ed49c153f9b9bcfeb43d" Nov 03 13:44:10.904425 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:44:10.904396 5024 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2d7fc25419ec27517c330dbbbea8e457c53a3c0f1f4ed49c153f9b9bcfeb43d\": container with ID starting with b2d7fc25419ec27517c330dbbbea8e457c53a3c0f1f4ed49c153f9b9bcfeb43d not found: ID does not exist" containerID="b2d7fc25419ec27517c330dbbbea8e457c53a3c0f1f4ed49c153f9b9bcfeb43d" Nov 03 13:44:10.904542 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:10.904432 5024 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2d7fc25419ec27517c330dbbbea8e457c53a3c0f1f4ed49c153f9b9bcfeb43d"} err="failed to get container status \"b2d7fc25419ec27517c330dbbbea8e457c53a3c0f1f4ed49c153f9b9bcfeb43d\": rpc error: code = NotFound desc = could not find container \"b2d7fc25419ec27517c330dbbbea8e457c53a3c0f1f4ed49c153f9b9bcfeb43d\": container with ID starting with b2d7fc25419ec27517c330dbbbea8e457c53a3c0f1f4ed49c153f9b9bcfeb43d not found: ID does not exist" Nov 03 13:44:10.904542 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:10.904451 5024 scope.go:122] "RemoveContainer" containerID="638d0cf057c395a54491109576c2829707ad1164879abf8ed2ff73b9afcd49d1" Nov 03 13:44:10.914119 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:10.914091 5024 scope.go:122] "RemoveContainer" containerID="638d0cf057c395a54491109576c2829707ad1164879abf8ed2ff73b9afcd49d1" Nov 03 13:44:10.914510 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:44:10.914475 5024 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"638d0cf057c395a54491109576c2829707ad1164879abf8ed2ff73b9afcd49d1\": container with ID starting with 638d0cf057c395a54491109576c2829707ad1164879abf8ed2ff73b9afcd49d1 not found: ID does not exist" containerID="638d0cf057c395a54491109576c2829707ad1164879abf8ed2ff73b9afcd49d1" Nov 03 13:44:10.914587 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:10.914524 5024 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"638d0cf057c395a54491109576c2829707ad1164879abf8ed2ff73b9afcd49d1"} err="failed to get container status \"638d0cf057c395a54491109576c2829707ad1164879abf8ed2ff73b9afcd49d1\": rpc error: code = NotFound desc = could not find container \"638d0cf057c395a54491109576c2829707ad1164879abf8ed2ff73b9afcd49d1\": container with ID starting with 638d0cf057c395a54491109576c2829707ad1164879abf8ed2ff73b9afcd49d1 not found: ID does not exist" Nov 03 13:44:10.914587 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:10.914547 5024 scope.go:122] "RemoveContainer" containerID="1065c59e4d759c691604c43c01c5947c75b0be37a4d1303512593a72cf489045" Nov 03 13:44:10.922739 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:10.922702 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-zpj4t"] Nov 03 13:44:10.927593 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:10.927569 5024 scope.go:122] "RemoveContainer" containerID="1065c59e4d759c691604c43c01c5947c75b0be37a4d1303512593a72cf489045" Nov 03 13:44:10.927897 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:10.927878 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-zpj4t"] Nov 03 13:44:10.927938 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:44:10.927888 5024 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1065c59e4d759c691604c43c01c5947c75b0be37a4d1303512593a72cf489045\": container with ID starting with 1065c59e4d759c691604c43c01c5947c75b0be37a4d1303512593a72cf489045 not found: ID does not exist" containerID="1065c59e4d759c691604c43c01c5947c75b0be37a4d1303512593a72cf489045" Nov 03 13:44:10.927938 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:10.927913 5024 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1065c59e4d759c691604c43c01c5947c75b0be37a4d1303512593a72cf489045"} err="failed to get container status \"1065c59e4d759c691604c43c01c5947c75b0be37a4d1303512593a72cf489045\": rpc error: code = NotFound desc = could not find container \"1065c59e4d759c691604c43c01c5947c75b0be37a4d1303512593a72cf489045\": container with ID starting with 1065c59e4d759c691604c43c01c5947c75b0be37a4d1303512593a72cf489045 not found: ID does not exist" Nov 03 13:44:10.945115 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:10.945088 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-d2rlz"] Nov 03 13:44:10.952022 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:10.951998 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-d2rlz"] Nov 03 13:44:10.974753 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:10.974721 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-lftt4"] Nov 03 13:44:10.986476 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:10.986445 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-e440bc-openshift-default-d56bb6b89-lftt4"] Nov 03 13:44:12.112690 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:12.112652 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="95aa021f-6ec9-4f53-810a-58eb1d059081" path="/var/lib/kubelet/pods/95aa021f-6ec9-4f53-810a-58eb1d059081/volumes" Nov 03 13:44:12.113349 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:12.113325 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="99af7f3f-6179-495a-a526-4cce51075588" path="/var/lib/kubelet/pods/99af7f3f-6179-495a-a526-4cce51075588/volumes" Nov 03 13:44:12.113865 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:12.113840 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af43a1c8-fad2-4b80-a0e9-65ab23c3612e" path="/var/lib/kubelet/pods/af43a1c8-fad2-4b80-a0e9-65ab23c3612e/volumes" Nov 03 13:44:12.261893 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:12.261861 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-c47670-openshift-default-7f67b9bfc-rv4l7" Nov 03 13:44:12.320878 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:12.320833 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/160d4b07-2558-4e4d-afb7-a7d509cc68ca-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/160d4b07-2558-4e4d-afb7-a7d509cc68ca-workload-certs\") pod \"160d4b07-2558-4e4d-afb7-a7d509cc68ca\" (UID: \"160d4b07-2558-4e4d-afb7-a7d509cc68ca\") " Nov 03 13:44:12.321063 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:12.320921 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/160d4b07-2558-4e4d-afb7-a7d509cc68ca-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/160d4b07-2558-4e4d-afb7-a7d509cc68ca-workload-socket\") pod \"160d4b07-2558-4e4d-afb7-a7d509cc68ca\" (UID: \"160d4b07-2558-4e4d-afb7-a7d509cc68ca\") " Nov 03 13:44:12.321063 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:12.320986 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/160d4b07-2558-4e4d-afb7-a7d509cc68ca-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/160d4b07-2558-4e4d-afb7-a7d509cc68ca-istio-data\") pod \"160d4b07-2558-4e4d-afb7-a7d509cc68ca\" (UID: \"160d4b07-2558-4e4d-afb7-a7d509cc68ca\") " Nov 03 13:44:12.321063 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:12.321015 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/160d4b07-2558-4e4d-afb7-a7d509cc68ca-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/160d4b07-2558-4e4d-afb7-a7d509cc68ca-istiod-ca-cert\") pod \"160d4b07-2558-4e4d-afb7-a7d509cc68ca\" (UID: \"160d4b07-2558-4e4d-afb7-a7d509cc68ca\") " Nov 03 13:44:12.321353 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:12.321067 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/160d4b07-2558-4e4d-afb7-a7d509cc68ca-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/160d4b07-2558-4e4d-afb7-a7d509cc68ca-istio-envoy\") pod \"160d4b07-2558-4e4d-afb7-a7d509cc68ca\" (UID: \"160d4b07-2558-4e4d-afb7-a7d509cc68ca\") " Nov 03 13:44:12.321353 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:12.321111 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/160d4b07-2558-4e4d-afb7-a7d509cc68ca-workload-certs" pod "160d4b07-2558-4e4d-afb7-a7d509cc68ca" (UID: "160d4b07-2558-4e4d-afb7-a7d509cc68ca"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:44:12.321353 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:12.321116 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/160d4b07-2558-4e4d-afb7-a7d509cc68ca-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/160d4b07-2558-4e4d-afb7-a7d509cc68ca-istio-podinfo\") pod \"160d4b07-2558-4e4d-afb7-a7d509cc68ca\" (UID: \"160d4b07-2558-4e4d-afb7-a7d509cc68ca\") " Nov 03 13:44:12.321353 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:12.321267 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/160d4b07-2558-4e4d-afb7-a7d509cc68ca-workload-socket" pod "160d4b07-2558-4e4d-afb7-a7d509cc68ca" (UID: "160d4b07-2558-4e4d-afb7-a7d509cc68ca"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:44:12.321353 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:12.321272 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/160d4b07-2558-4e4d-afb7-a7d509cc68ca-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/160d4b07-2558-4e4d-afb7-a7d509cc68ca-credential-socket\") pod \"160d4b07-2558-4e4d-afb7-a7d509cc68ca\" (UID: \"160d4b07-2558-4e4d-afb7-a7d509cc68ca\") " Nov 03 13:44:12.321627 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:12.321395 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/160d4b07-2558-4e4d-afb7-a7d509cc68ca-istio-token\" (UniqueName: \"kubernetes.io/projected/160d4b07-2558-4e4d-afb7-a7d509cc68ca-istio-token\") pod \"160d4b07-2558-4e4d-afb7-a7d509cc68ca\" (UID: \"160d4b07-2558-4e4d-afb7-a7d509cc68ca\") " Nov 03 13:44:12.321627 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:12.321313 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/160d4b07-2558-4e4d-afb7-a7d509cc68ca-istio-data" pod "160d4b07-2558-4e4d-afb7-a7d509cc68ca" (UID: "160d4b07-2558-4e4d-afb7-a7d509cc68ca"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:44:12.321627 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:12.321426 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/160d4b07-2558-4e4d-afb7-a7d509cc68ca-kube-api-access-z74vs\" (UniqueName: \"kubernetes.io/projected/160d4b07-2558-4e4d-afb7-a7d509cc68ca-kube-api-access-z74vs\") pod \"160d4b07-2558-4e4d-afb7-a7d509cc68ca\" (UID: \"160d4b07-2558-4e4d-afb7-a7d509cc68ca\") " Nov 03 13:44:12.321627 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:12.321478 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/160d4b07-2558-4e4d-afb7-a7d509cc68ca-istiod-ca-cert" pod "160d4b07-2558-4e4d-afb7-a7d509cc68ca" (UID: "160d4b07-2558-4e4d-afb7-a7d509cc68ca"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:44:12.321627 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:12.321531 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/160d4b07-2558-4e4d-afb7-a7d509cc68ca-credential-socket" pod "160d4b07-2558-4e4d-afb7-a7d509cc68ca" (UID: "160d4b07-2558-4e4d-afb7-a7d509cc68ca"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:44:12.321964 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:12.321937 5024 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/160d4b07-2558-4e4d-afb7-a7d509cc68ca-istio-data\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:44:12.322055 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:12.321966 5024 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/160d4b07-2558-4e4d-afb7-a7d509cc68ca-istiod-ca-cert\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:44:12.322055 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:12.321981 5024 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/160d4b07-2558-4e4d-afb7-a7d509cc68ca-credential-socket\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:44:12.322055 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:12.321997 5024 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/160d4b07-2558-4e4d-afb7-a7d509cc68ca-workload-certs\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:44:12.322055 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:12.322010 5024 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/160d4b07-2558-4e4d-afb7-a7d509cc68ca-workload-socket\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:44:12.323741 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:12.323713 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/160d4b07-2558-4e4d-afb7-a7d509cc68ca-istio-token" pod "160d4b07-2558-4e4d-afb7-a7d509cc68ca" (UID: "160d4b07-2558-4e4d-afb7-a7d509cc68ca"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:44:12.323911 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:12.323886 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/160d4b07-2558-4e4d-afb7-a7d509cc68ca-istio-podinfo" pod "160d4b07-2558-4e4d-afb7-a7d509cc68ca" (UID: "160d4b07-2558-4e4d-afb7-a7d509cc68ca"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 13:44:12.324052 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:12.324036 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/160d4b07-2558-4e4d-afb7-a7d509cc68ca-kube-api-access-z74vs" pod "160d4b07-2558-4e4d-afb7-a7d509cc68ca" (UID: "160d4b07-2558-4e4d-afb7-a7d509cc68ca"). InnerVolumeSpecName "kube-api-access-z74vs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:44:12.324368 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:12.324335 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/160d4b07-2558-4e4d-afb7-a7d509cc68ca-istio-envoy" pod "160d4b07-2558-4e4d-afb7-a7d509cc68ca" (UID: "160d4b07-2558-4e4d-afb7-a7d509cc68ca"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:44:12.423122 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:12.423089 5024 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/160d4b07-2558-4e4d-afb7-a7d509cc68ca-istio-token\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:44:12.423122 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:12.423119 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z74vs\" (UniqueName: \"kubernetes.io/projected/160d4b07-2558-4e4d-afb7-a7d509cc68ca-kube-api-access-z74vs\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:44:12.423411 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:12.423129 5024 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/160d4b07-2558-4e4d-afb7-a7d509cc68ca-istio-envoy\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:44:12.423411 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:12.423138 5024 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/160d4b07-2558-4e4d-afb7-a7d509cc68ca-istio-podinfo\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:44:12.895981 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:12.895947 5024 generic.go:356] "Generic (PLEG): container finished" podID="160d4b07-2558-4e4d-afb7-a7d509cc68ca" containerID="4245b3cae40c16aaa5bb911f6a4236f951bba5bd44441134ccc19ae9ba252f86" exitCode=0 Nov 03 13:44:12.896317 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:12.896014 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-c47670-openshift-default-7f67b9bfc-rv4l7" Nov 03 13:44:12.896317 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:12.896036 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-c47670-openshift-default-7f67b9bfc-rv4l7" event={"ID":"160d4b07-2558-4e4d-afb7-a7d509cc68ca","Type":"ContainerDied","Data":"4245b3cae40c16aaa5bb911f6a4236f951bba5bd44441134ccc19ae9ba252f86"} Nov 03 13:44:12.896317 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:12.896105 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-c47670-openshift-default-7f67b9bfc-rv4l7" event={"ID":"160d4b07-2558-4e4d-afb7-a7d509cc68ca","Type":"ContainerDied","Data":"0858c5f08ab458bb3116acbe14801152ccd7d0227cf6ffe9afdc7be5423aca8f"} Nov 03 13:44:12.896317 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:12.896140 5024 scope.go:122] "RemoveContainer" containerID="4245b3cae40c16aaa5bb911f6a4236f951bba5bd44441134ccc19ae9ba252f86" Nov 03 13:44:12.907217 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:12.907200 5024 scope.go:122] "RemoveContainer" containerID="9b8b4fed8f49f179cf7e812cb0767af05932b07fc896bd7af1f6ccf37503d2f8" Nov 03 13:44:12.916135 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:12.916115 5024 scope.go:122] "RemoveContainer" containerID="4245b3cae40c16aaa5bb911f6a4236f951bba5bd44441134ccc19ae9ba252f86" Nov 03 13:44:12.916443 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:44:12.916419 5024 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4245b3cae40c16aaa5bb911f6a4236f951bba5bd44441134ccc19ae9ba252f86\": container with ID starting with 4245b3cae40c16aaa5bb911f6a4236f951bba5bd44441134ccc19ae9ba252f86 not found: ID does not exist" containerID="4245b3cae40c16aaa5bb911f6a4236f951bba5bd44441134ccc19ae9ba252f86" Nov 03 13:44:12.916545 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:12.916453 5024 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4245b3cae40c16aaa5bb911f6a4236f951bba5bd44441134ccc19ae9ba252f86"} err="failed to get container status \"4245b3cae40c16aaa5bb911f6a4236f951bba5bd44441134ccc19ae9ba252f86\": rpc error: code = NotFound desc = could not find container \"4245b3cae40c16aaa5bb911f6a4236f951bba5bd44441134ccc19ae9ba252f86\": container with ID starting with 4245b3cae40c16aaa5bb911f6a4236f951bba5bd44441134ccc19ae9ba252f86 not found: ID does not exist" Nov 03 13:44:12.916545 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:12.916478 5024 scope.go:122] "RemoveContainer" containerID="9b8b4fed8f49f179cf7e812cb0767af05932b07fc896bd7af1f6ccf37503d2f8" Nov 03 13:44:12.916777 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:44:12.916758 5024 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b8b4fed8f49f179cf7e812cb0767af05932b07fc896bd7af1f6ccf37503d2f8\": container with ID starting with 9b8b4fed8f49f179cf7e812cb0767af05932b07fc896bd7af1f6ccf37503d2f8 not found: ID does not exist" containerID="9b8b4fed8f49f179cf7e812cb0767af05932b07fc896bd7af1f6ccf37503d2f8" Nov 03 13:44:12.916817 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:12.916782 5024 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b8b4fed8f49f179cf7e812cb0767af05932b07fc896bd7af1f6ccf37503d2f8"} err="failed to get container status \"9b8b4fed8f49f179cf7e812cb0767af05932b07fc896bd7af1f6ccf37503d2f8\": rpc error: code = NotFound desc = could not find container \"9b8b4fed8f49f179cf7e812cb0767af05932b07fc896bd7af1f6ccf37503d2f8\": container with ID starting with 9b8b4fed8f49f179cf7e812cb0767af05932b07fc896bd7af1f6ccf37503d2f8 not found: ID does not exist" Nov 03 13:44:12.934746 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:12.934719 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-c47670-openshift-default-7f67b9bfc-rv4l7"] Nov 03 13:44:12.947957 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:12.947936 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-c47670-openshift-default-7f67b9bfc-rv4l7"] Nov 03 13:44:14.111171 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:14.111137 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="160d4b07-2558-4e4d-afb7-a7d509cc68ca" path="/var/lib/kubelet/pods/160d4b07-2558-4e4d-afb7-a7d509cc68ca/volumes" Nov 03 13:44:14.121246 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:14.121213 5024 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-6865svtfs" podUID="d34e5887-ae31-43ce-8d24-9973d8ba447c" containerName="payload-processing" containerID="cri-o://eb95531032faf67b58545f82375d2cabf0dba2a81bc135b9ab9b09603a071801" gracePeriod=25 Nov 03 13:44:14.395000 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:14.394970 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-6865svtfs" Nov 03 13:44:14.452605 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:14.452576 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d34e5887-ae31-43ce-8d24-9973d8ba447c-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d34e5887-ae31-43ce-8d24-9973d8ba447c-routing-overlay-volume\") pod \"d34e5887-ae31-43ce-8d24-9973d8ba447c\" (UID: \"d34e5887-ae31-43ce-8d24-9973d8ba447c\") " Nov 03 13:44:14.452772 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:14.452620 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d34e5887-ae31-43ce-8d24-9973d8ba447c-provider-credentials\" (UniqueName: \"kubernetes.io/projected/d34e5887-ae31-43ce-8d24-9973d8ba447c-provider-credentials\") pod \"d34e5887-ae31-43ce-8d24-9973d8ba447c\" (UID: \"d34e5887-ae31-43ce-8d24-9973d8ba447c\") " Nov 03 13:44:14.452772 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:14.452677 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d34e5887-ae31-43ce-8d24-9973d8ba447c-kube-api-access-hpkhh\" (UniqueName: \"kubernetes.io/projected/d34e5887-ae31-43ce-8d24-9973d8ba447c-kube-api-access-hpkhh\") pod \"d34e5887-ae31-43ce-8d24-9973d8ba447c\" (UID: \"d34e5887-ae31-43ce-8d24-9973d8ba447c\") " Nov 03 13:44:14.452951 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:14.452910 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d34e5887-ae31-43ce-8d24-9973d8ba447c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d34e5887-ae31-43ce-8d24-9973d8ba447c-plugins-config-volume\") pod \"d34e5887-ae31-43ce-8d24-9973d8ba447c\" (UID: \"d34e5887-ae31-43ce-8d24-9973d8ba447c\") " Nov 03 13:44:14.453379 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:14.453351 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d34e5887-ae31-43ce-8d24-9973d8ba447c-routing-overlay-volume" pod "d34e5887-ae31-43ce-8d24-9973d8ba447c" (UID: "d34e5887-ae31-43ce-8d24-9973d8ba447c"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:44:14.453478 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:14.453359 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d34e5887-ae31-43ce-8d24-9973d8ba447c-plugins-config-volume" pod "d34e5887-ae31-43ce-8d24-9973d8ba447c" (UID: "d34e5887-ae31-43ce-8d24-9973d8ba447c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:44:14.453546 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:14.453519 5024 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d34e5887-ae31-43ce-8d24-9973d8ba447c-routing-overlay-volume\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:44:14.455803 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:14.455769 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d34e5887-ae31-43ce-8d24-9973d8ba447c-provider-credentials" pod "d34e5887-ae31-43ce-8d24-9973d8ba447c" (UID: "d34e5887-ae31-43ce-8d24-9973d8ba447c"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:44:14.457444 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:14.457419 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d34e5887-ae31-43ce-8d24-9973d8ba447c-kube-api-access-hpkhh" pod "d34e5887-ae31-43ce-8d24-9973d8ba447c" (UID: "d34e5887-ae31-43ce-8d24-9973d8ba447c"). InnerVolumeSpecName "kube-api-access-hpkhh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:44:14.555013 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:14.554979 5024 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d34e5887-ae31-43ce-8d24-9973d8ba447c-plugins-config-volume\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:44:14.555013 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:14.555009 5024 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d34e5887-ae31-43ce-8d24-9973d8ba447c-provider-credentials\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:44:14.555218 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:14.555021 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hpkhh\" (UniqueName: \"kubernetes.io/projected/d34e5887-ae31-43ce-8d24-9973d8ba447c-kube-api-access-hpkhh\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:44:14.818200 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:14.818121 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-9f71f2-5b8844dc8c-6xt6h"] Nov 03 13:44:14.913691 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:14.913658 5024 generic.go:356] "Generic (PLEG): container finished" podID="d34e5887-ae31-43ce-8d24-9973d8ba447c" containerID="eb95531032faf67b58545f82375d2cabf0dba2a81bc135b9ab9b09603a071801" exitCode=0 Nov 03 13:44:14.913865 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:14.913709 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-6865svtfs" Nov 03 13:44:14.913865 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:14.913739 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-6865svtfs" event={"ID":"d34e5887-ae31-43ce-8d24-9973d8ba447c","Type":"ContainerDied","Data":"eb95531032faf67b58545f82375d2cabf0dba2a81bc135b9ab9b09603a071801"} Nov 03 13:44:14.913865 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:14.913804 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-6865svtfs" event={"ID":"d34e5887-ae31-43ce-8d24-9973d8ba447c","Type":"ContainerDied","Data":"94ec91ca4418dc01aa6636b0663c2f8bdd2715f8f9bb631eb71127f7495b9a20"} Nov 03 13:44:14.913865 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:14.913834 5024 scope.go:122] "RemoveContainer" containerID="eb95531032faf67b58545f82375d2cabf0dba2a81bc135b9ab9b09603a071801" Nov 03 13:44:14.925860 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:14.925821 5024 scope.go:122] "RemoveContainer" containerID="eb95531032faf67b58545f82375d2cabf0dba2a81bc135b9ab9b09603a071801" Nov 03 13:44:14.934840 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:44:14.934811 5024 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb95531032faf67b58545f82375d2cabf0dba2a81bc135b9ab9b09603a071801\": container with ID starting with eb95531032faf67b58545f82375d2cabf0dba2a81bc135b9ab9b09603a071801 not found: ID does not exist" containerID="eb95531032faf67b58545f82375d2cabf0dba2a81bc135b9ab9b09603a071801" Nov 03 13:44:14.934945 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:14.934848 5024 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb95531032faf67b58545f82375d2cabf0dba2a81bc135b9ab9b09603a071801"} err="failed to get container status \"eb95531032faf67b58545f82375d2cabf0dba2a81bc135b9ab9b09603a071801\": rpc error: code = NotFound desc = could not find container \"eb95531032faf67b58545f82375d2cabf0dba2a81bc135b9ab9b09603a071801\": container with ID starting with eb95531032faf67b58545f82375d2cabf0dba2a81bc135b9ab9b09603a071801 not found: ID does not exist" Nov 03 13:44:14.946161 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:14.946133 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-6865svtfs"] Nov 03 13:44:14.954190 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:14.954166 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-7c192b/payload-processing-external-model-e2e-extmod-b-7c192b-6865svtfs"] Nov 03 13:44:15.670982 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:15.670945 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-c47670-54c8bb9b89-fzdnn" podUID="7a30a13a-965d-4291-9acd-f109a6ead710" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 13:44:16.112713 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:16.112669 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d34e5887-ae31-43ce-8d24-9973d8ba447c" path="/var/lib/kubelet/pods/d34e5887-ae31-43ce-8d24-9973d8ba447c/volumes" Nov 03 13:44:19.819249 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:19.819201 5024 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-9f71f2-5b8844dc8c-6xt6h" podUID="f47e8fd5-9a67-45ca-a817-05b9e20654cc" containerName="payload-processing" containerID="cri-o://9e5f2654511b7a99377d16600dae81be35cfb95d749a69bcf9a601ca0d177ac3" gracePeriod=25 Nov 03 13:44:19.952318 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:19.952259 5024 generic.go:356] "Generic (PLEG): container finished" podID="f47e8fd5-9a67-45ca-a817-05b9e20654cc" containerID="9e5f2654511b7a99377d16600dae81be35cfb95d749a69bcf9a601ca0d177ac3" exitCode=0 Nov 03 13:44:19.952517 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:19.952346 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-9f71f2-5b8844dc8c-6xt6h" event={"ID":"f47e8fd5-9a67-45ca-a817-05b9e20654cc","Type":"ContainerDied","Data":"9e5f2654511b7a99377d16600dae81be35cfb95d749a69bcf9a601ca0d177ac3"} Nov 03 13:44:20.110794 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:20.110764 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-9f71f2-5b8844dc8c-6xt6h" Nov 03 13:44:20.246376 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:20.246328 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f47e8fd5-9a67-45ca-a817-05b9e20654cc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f47e8fd5-9a67-45ca-a817-05b9e20654cc-plugins-config-volume\") pod \"f47e8fd5-9a67-45ca-a817-05b9e20654cc\" (UID: \"f47e8fd5-9a67-45ca-a817-05b9e20654cc\") " Nov 03 13:44:20.246591 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:20.246573 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f47e8fd5-9a67-45ca-a817-05b9e20654cc-kube-api-access-9hlqh\" (UniqueName: \"kubernetes.io/projected/f47e8fd5-9a67-45ca-a817-05b9e20654cc-kube-api-access-9hlqh\") pod \"f47e8fd5-9a67-45ca-a817-05b9e20654cc\" (UID: \"f47e8fd5-9a67-45ca-a817-05b9e20654cc\") " Nov 03 13:44:20.246765 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:20.246740 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f47e8fd5-9a67-45ca-a817-05b9e20654cc-plugins-config-volume" pod "f47e8fd5-9a67-45ca-a817-05b9e20654cc" (UID: "f47e8fd5-9a67-45ca-a817-05b9e20654cc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:44:20.247037 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:20.247015 5024 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f47e8fd5-9a67-45ca-a817-05b9e20654cc-plugins-config-volume\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:44:20.248749 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:20.248727 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f47e8fd5-9a67-45ca-a817-05b9e20654cc-kube-api-access-9hlqh" pod "f47e8fd5-9a67-45ca-a817-05b9e20654cc" (UID: "f47e8fd5-9a67-45ca-a817-05b9e20654cc"). InnerVolumeSpecName "kube-api-access-9hlqh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:44:20.348849 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:20.348759 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9hlqh\" (UniqueName: \"kubernetes.io/projected/f47e8fd5-9a67-45ca-a817-05b9e20654cc-kube-api-access-9hlqh\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:44:20.962057 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:20.962022 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-9f71f2-5b8844dc8c-6xt6h" Nov 03 13:44:20.962057 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:20.962039 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-9f71f2-5b8844dc8c-6xt6h" event={"ID":"f47e8fd5-9a67-45ca-a817-05b9e20654cc","Type":"ContainerDied","Data":"c252fa015508deeb50fe885bcf8387d8010ac6995c34025e6315aa29c03c8291"} Nov 03 13:44:20.962660 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:20.962115 5024 scope.go:122] "RemoveContainer" containerID="9e5f2654511b7a99377d16600dae81be35cfb95d749a69bcf9a601ca0d177ac3" Nov 03 13:44:20.991435 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:20.991406 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-9f71f2-5b8844dc8c-6xt6h"] Nov 03 13:44:20.997581 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:20.997555 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-9f71f2-5b8844dc8c-6xt6h"] Nov 03 13:44:21.508440 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:21.508408 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-c47670-54c8bb9b89-fzdnn" Nov 03 13:44:21.562851 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:21.562810 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7a30a13a-965d-4291-9acd-f109a6ead710-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a30a13a-965d-4291-9acd-f109a6ead710-plugins-config-volume\") pod \"7a30a13a-965d-4291-9acd-f109a6ead710\" (UID: \"7a30a13a-965d-4291-9acd-f109a6ead710\") " Nov 03 13:44:21.563039 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:21.563002 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a30a13a-965d-4291-9acd-f109a6ead710-kube-api-access-m8sfg\" (UniqueName: \"kubernetes.io/projected/7a30a13a-965d-4291-9acd-f109a6ead710-kube-api-access-m8sfg\") pod \"7a30a13a-965d-4291-9acd-f109a6ead710\" (UID: \"7a30a13a-965d-4291-9acd-f109a6ead710\") " Nov 03 13:44:21.563278 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:21.563247 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a30a13a-965d-4291-9acd-f109a6ead710-plugins-config-volume" pod "7a30a13a-965d-4291-9acd-f109a6ead710" (UID: "7a30a13a-965d-4291-9acd-f109a6ead710"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:44:21.563503 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:21.563480 5024 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a30a13a-965d-4291-9acd-f109a6ead710-plugins-config-volume\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:44:21.565279 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:21.565249 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a30a13a-965d-4291-9acd-f109a6ead710-kube-api-access-m8sfg" pod "7a30a13a-965d-4291-9acd-f109a6ead710" (UID: "7a30a13a-965d-4291-9acd-f109a6ead710"). InnerVolumeSpecName "kube-api-access-m8sfg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:44:21.664473 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:21.664439 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m8sfg\" (UniqueName: \"kubernetes.io/projected/7a30a13a-965d-4291-9acd-f109a6ead710-kube-api-access-m8sfg\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:44:21.970384 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:21.970339 5024 generic.go:356] "Generic (PLEG): container finished" podID="7a30a13a-965d-4291-9acd-f109a6ead710" containerID="ccdfa1856a6bba533687ddc2d36f10d15651ea5354a78e8a75ad8fb767081af5" exitCode=0 Nov 03 13:44:21.970892 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:21.970380 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-c47670-54c8bb9b89-fzdnn" event={"ID":"7a30a13a-965d-4291-9acd-f109a6ead710","Type":"ContainerDied","Data":"ccdfa1856a6bba533687ddc2d36f10d15651ea5354a78e8a75ad8fb767081af5"} Nov 03 13:44:21.970892 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:21.970403 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-c47670-54c8bb9b89-fzdnn" Nov 03 13:44:21.970892 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:21.970452 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-c47670-54c8bb9b89-fzdnn" event={"ID":"7a30a13a-965d-4291-9acd-f109a6ead710","Type":"ContainerDied","Data":"12b72fc0ff14678f253add4dce17d31b74fc172c0d1bdcc40ead75ecb6cd79f9"} Nov 03 13:44:21.970892 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:21.970497 5024 scope.go:122] "RemoveContainer" containerID="ccdfa1856a6bba533687ddc2d36f10d15651ea5354a78e8a75ad8fb767081af5" Nov 03 13:44:21.981336 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:21.981310 5024 scope.go:122] "RemoveContainer" containerID="ccdfa1856a6bba533687ddc2d36f10d15651ea5354a78e8a75ad8fb767081af5" Nov 03 13:44:21.981763 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:44:21.981635 5024 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ccdfa1856a6bba533687ddc2d36f10d15651ea5354a78e8a75ad8fb767081af5\": container with ID starting with ccdfa1856a6bba533687ddc2d36f10d15651ea5354a78e8a75ad8fb767081af5 not found: ID does not exist" containerID="ccdfa1856a6bba533687ddc2d36f10d15651ea5354a78e8a75ad8fb767081af5" Nov 03 13:44:21.981763 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:21.981670 5024 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ccdfa1856a6bba533687ddc2d36f10d15651ea5354a78e8a75ad8fb767081af5"} err="failed to get container status \"ccdfa1856a6bba533687ddc2d36f10d15651ea5354a78e8a75ad8fb767081af5\": rpc error: code = NotFound desc = could not find container \"ccdfa1856a6bba533687ddc2d36f10d15651ea5354a78e8a75ad8fb767081af5\": container with ID starting with ccdfa1856a6bba533687ddc2d36f10d15651ea5354a78e8a75ad8fb767081af5 not found: ID does not exist" Nov 03 13:44:22.008948 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:22.008850 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-c47670-54c8bb9b89-fzdnn"] Nov 03 13:44:22.011856 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:22.011829 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-c47670-54c8bb9b89-fzdnn"] Nov 03 13:44:22.109953 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:22.109916 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7a30a13a-965d-4291-9acd-f109a6ead710" path="/var/lib/kubelet/pods/7a30a13a-965d-4291-9acd-f109a6ead710/volumes" Nov 03 13:44:22.110575 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:22.110548 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f47e8fd5-9a67-45ca-a817-05b9e20654cc" path="/var/lib/kubelet/pods/f47e8fd5-9a67-45ca-a817-05b9e20654cc/volumes" Nov 03 13:44:35.623097 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:35.623058 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-08111b/premium-w1-08111b-kserve-57555d598-dx9g5"] Nov 03 13:44:35.630922 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:35.630894 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-08111b/simulator-w1-08111b-kserve-7f65c5867-btf5b"] Nov 03 13:44:39.509061 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:44:39.509016 5024 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d7mwz" containerName="extractor" Nov 03 13:44:42.578200 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:42.578159 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-84587756dc-h2ns8"] Nov 03 13:44:42.578690 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:42.578414 5024 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-h2ns8" podUID="155ba865-dcfd-4203-82b9-5399ca58d988" containerName="istio-proxy" containerID="cri-o://b497e622f8a4363b340c1d0fe74937acbcd77a6120d6e2e07dbfd73c7c50b244" gracePeriod=30 Nov 03 13:44:42.580771 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:42.580737 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-84587756dc-nr5hn"] Nov 03 13:44:42.580973 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:42.580950 5024 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-nr5hn" podUID="3cd0d23e-9823-4988-ac79-0820dcc4c028" containerName="istio-proxy" containerID="cri-o://8cef8680dcbf34f6e439e6c1d96440fe7a53d66759de93261895d1190c9769ed" gracePeriod=30 Nov 03 13:44:42.583195 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:42.583170 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-84587756dc-xtsb8"] Nov 03 13:44:42.583397 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:42.583369 5024 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-xtsb8" podUID="b46130ce-1911-475e-8ec7-5204881eb5ea" containerName="istio-proxy" containerID="cri-o://e525aba39eee600786bacf1d3950338c7cc4baaa122bc1eb2d7be06c938f0d0d" gracePeriod=30 Nov 03 13:44:43.312097 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:43.312048 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-08111b-9cc448f95-n6ttb"] Nov 03 13:44:43.325998 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:43.325963 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-08111b-744df4c7d7-b5xcs"] Nov 03 13:44:47.900037 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:47.900001 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-h2ns8" Nov 03 13:44:47.914041 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:47.914008 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-xtsb8" Nov 03 13:44:47.923538 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:47.923513 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-nr5hn" Nov 03 13:44:47.958745 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:47.957883 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/155ba865-dcfd-4203-82b9-5399ca58d988-istio-token\" (UniqueName: \"kubernetes.io/projected/155ba865-dcfd-4203-82b9-5399ca58d988-istio-token\") pod \"155ba865-dcfd-4203-82b9-5399ca58d988\" (UID: \"155ba865-dcfd-4203-82b9-5399ca58d988\") " Nov 03 13:44:47.959453 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:47.959421 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/155ba865-dcfd-4203-82b9-5399ca58d988-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/155ba865-dcfd-4203-82b9-5399ca58d988-istiod-ca-cert\") pod \"155ba865-dcfd-4203-82b9-5399ca58d988\" (UID: \"155ba865-dcfd-4203-82b9-5399ca58d988\") " Nov 03 13:44:47.959598 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:47.959484 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/155ba865-dcfd-4203-82b9-5399ca58d988-kube-api-access-sqbdt\" (UniqueName: \"kubernetes.io/projected/155ba865-dcfd-4203-82b9-5399ca58d988-kube-api-access-sqbdt\") pod \"155ba865-dcfd-4203-82b9-5399ca58d988\" (UID: \"155ba865-dcfd-4203-82b9-5399ca58d988\") " Nov 03 13:44:47.959658 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:47.959629 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/155ba865-dcfd-4203-82b9-5399ca58d988-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/155ba865-dcfd-4203-82b9-5399ca58d988-istio-data\") pod \"155ba865-dcfd-4203-82b9-5399ca58d988\" (UID: \"155ba865-dcfd-4203-82b9-5399ca58d988\") " Nov 03 13:44:47.959704 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:47.959659 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/155ba865-dcfd-4203-82b9-5399ca58d988-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/155ba865-dcfd-4203-82b9-5399ca58d988-workload-socket\") pod \"155ba865-dcfd-4203-82b9-5399ca58d988\" (UID: \"155ba865-dcfd-4203-82b9-5399ca58d988\") " Nov 03 13:44:47.959747 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:47.959707 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/155ba865-dcfd-4203-82b9-5399ca58d988-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/155ba865-dcfd-4203-82b9-5399ca58d988-workload-certs\") pod \"155ba865-dcfd-4203-82b9-5399ca58d988\" (UID: \"155ba865-dcfd-4203-82b9-5399ca58d988\") " Nov 03 13:44:47.959796 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:47.959754 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/155ba865-dcfd-4203-82b9-5399ca58d988-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/155ba865-dcfd-4203-82b9-5399ca58d988-istio-podinfo\") pod \"155ba865-dcfd-4203-82b9-5399ca58d988\" (UID: \"155ba865-dcfd-4203-82b9-5399ca58d988\") " Nov 03 13:44:47.959848 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:47.959821 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/155ba865-dcfd-4203-82b9-5399ca58d988-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/155ba865-dcfd-4203-82b9-5399ca58d988-credential-socket\") pod \"155ba865-dcfd-4203-82b9-5399ca58d988\" (UID: \"155ba865-dcfd-4203-82b9-5399ca58d988\") " Nov 03 13:44:47.959893 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:47.959877 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/155ba865-dcfd-4203-82b9-5399ca58d988-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/155ba865-dcfd-4203-82b9-5399ca58d988-istio-envoy\") pod \"155ba865-dcfd-4203-82b9-5399ca58d988\" (UID: \"155ba865-dcfd-4203-82b9-5399ca58d988\") " Nov 03 13:44:47.961019 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:47.960397 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/155ba865-dcfd-4203-82b9-5399ca58d988-istio-data" pod "155ba865-dcfd-4203-82b9-5399ca58d988" (UID: "155ba865-dcfd-4203-82b9-5399ca58d988"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:44:47.961019 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:47.960412 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/155ba865-dcfd-4203-82b9-5399ca58d988-workload-certs" pod "155ba865-dcfd-4203-82b9-5399ca58d988" (UID: "155ba865-dcfd-4203-82b9-5399ca58d988"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:44:47.961019 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:47.960629 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/155ba865-dcfd-4203-82b9-5399ca58d988-workload-socket" pod "155ba865-dcfd-4203-82b9-5399ca58d988" (UID: "155ba865-dcfd-4203-82b9-5399ca58d988"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:44:47.961019 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:47.960727 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/155ba865-dcfd-4203-82b9-5399ca58d988-istiod-ca-cert" pod "155ba865-dcfd-4203-82b9-5399ca58d988" (UID: "155ba865-dcfd-4203-82b9-5399ca58d988"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:44:47.961504 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:47.961202 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/155ba865-dcfd-4203-82b9-5399ca58d988-istio-token" pod "155ba865-dcfd-4203-82b9-5399ca58d988" (UID: "155ba865-dcfd-4203-82b9-5399ca58d988"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:44:47.961504 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:47.961213 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/155ba865-dcfd-4203-82b9-5399ca58d988-credential-socket" pod "155ba865-dcfd-4203-82b9-5399ca58d988" (UID: "155ba865-dcfd-4203-82b9-5399ca58d988"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:44:47.965496 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:47.965463 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/155ba865-dcfd-4203-82b9-5399ca58d988-kube-api-access-sqbdt" pod "155ba865-dcfd-4203-82b9-5399ca58d988" (UID: "155ba865-dcfd-4203-82b9-5399ca58d988"). InnerVolumeSpecName "kube-api-access-sqbdt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:44:47.971443 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:47.970447 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/155ba865-dcfd-4203-82b9-5399ca58d988-istio-envoy" pod "155ba865-dcfd-4203-82b9-5399ca58d988" (UID: "155ba865-dcfd-4203-82b9-5399ca58d988"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:44:47.971443 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:47.970457 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/155ba865-dcfd-4203-82b9-5399ca58d988-istio-podinfo" pod "155ba865-dcfd-4203-82b9-5399ca58d988" (UID: "155ba865-dcfd-4203-82b9-5399ca58d988"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 13:44:48.061413 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:48.061366 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3cd0d23e-9823-4988-ac79-0820dcc4c028-kube-api-access-rhbfk\" (UniqueName: \"kubernetes.io/projected/3cd0d23e-9823-4988-ac79-0820dcc4c028-kube-api-access-rhbfk\") pod \"3cd0d23e-9823-4988-ac79-0820dcc4c028\" (UID: \"3cd0d23e-9823-4988-ac79-0820dcc4c028\") " Nov 03 13:44:48.061641 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:48.061425 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b46130ce-1911-475e-8ec7-5204881eb5ea-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b46130ce-1911-475e-8ec7-5204881eb5ea-istio-envoy\") pod \"b46130ce-1911-475e-8ec7-5204881eb5ea\" (UID: \"b46130ce-1911-475e-8ec7-5204881eb5ea\") " Nov 03 13:44:48.061641 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:48.061468 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b46130ce-1911-475e-8ec7-5204881eb5ea-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b46130ce-1911-475e-8ec7-5204881eb5ea-workload-socket\") pod \"b46130ce-1911-475e-8ec7-5204881eb5ea\" (UID: \"b46130ce-1911-475e-8ec7-5204881eb5ea\") " Nov 03 13:44:48.061641 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:48.061496 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b46130ce-1911-475e-8ec7-5204881eb5ea-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b46130ce-1911-475e-8ec7-5204881eb5ea-workload-certs\") pod \"b46130ce-1911-475e-8ec7-5204881eb5ea\" (UID: \"b46130ce-1911-475e-8ec7-5204881eb5ea\") " Nov 03 13:44:48.061641 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:48.061545 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3cd0d23e-9823-4988-ac79-0820dcc4c028-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3cd0d23e-9823-4988-ac79-0820dcc4c028-istio-envoy\") pod \"3cd0d23e-9823-4988-ac79-0820dcc4c028\" (UID: \"3cd0d23e-9823-4988-ac79-0820dcc4c028\") " Nov 03 13:44:48.061641 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:48.061580 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b46130ce-1911-475e-8ec7-5204881eb5ea-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b46130ce-1911-475e-8ec7-5204881eb5ea-istio-data\") pod \"b46130ce-1911-475e-8ec7-5204881eb5ea\" (UID: \"b46130ce-1911-475e-8ec7-5204881eb5ea\") " Nov 03 13:44:48.061641 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:48.061621 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b46130ce-1911-475e-8ec7-5204881eb5ea-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b46130ce-1911-475e-8ec7-5204881eb5ea-istio-podinfo\") pod \"b46130ce-1911-475e-8ec7-5204881eb5ea\" (UID: \"b46130ce-1911-475e-8ec7-5204881eb5ea\") " Nov 03 13:44:48.062019 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:48.061652 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3cd0d23e-9823-4988-ac79-0820dcc4c028-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3cd0d23e-9823-4988-ac79-0820dcc4c028-istio-data\") pod \"3cd0d23e-9823-4988-ac79-0820dcc4c028\" (UID: \"3cd0d23e-9823-4988-ac79-0820dcc4c028\") " Nov 03 13:44:48.062019 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:48.061688 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b46130ce-1911-475e-8ec7-5204881eb5ea-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b46130ce-1911-475e-8ec7-5204881eb5ea-credential-socket\") pod \"b46130ce-1911-475e-8ec7-5204881eb5ea\" (UID: \"b46130ce-1911-475e-8ec7-5204881eb5ea\") " Nov 03 13:44:48.062019 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:48.061724 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b46130ce-1911-475e-8ec7-5204881eb5ea-istio-token\" (UniqueName: \"kubernetes.io/projected/b46130ce-1911-475e-8ec7-5204881eb5ea-istio-token\") pod \"b46130ce-1911-475e-8ec7-5204881eb5ea\" (UID: \"b46130ce-1911-475e-8ec7-5204881eb5ea\") " Nov 03 13:44:48.062019 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:48.061772 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3cd0d23e-9823-4988-ac79-0820dcc4c028-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3cd0d23e-9823-4988-ac79-0820dcc4c028-istio-podinfo\") pod \"3cd0d23e-9823-4988-ac79-0820dcc4c028\" (UID: \"3cd0d23e-9823-4988-ac79-0820dcc4c028\") " Nov 03 13:44:48.062019 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:48.061778 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b46130ce-1911-475e-8ec7-5204881eb5ea-workload-socket" pod "b46130ce-1911-475e-8ec7-5204881eb5ea" (UID: "b46130ce-1911-475e-8ec7-5204881eb5ea"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:44:48.062019 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:48.061833 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3cd0d23e-9823-4988-ac79-0820dcc4c028-istio-token\" (UniqueName: \"kubernetes.io/projected/3cd0d23e-9823-4988-ac79-0820dcc4c028-istio-token\") pod \"3cd0d23e-9823-4988-ac79-0820dcc4c028\" (UID: \"3cd0d23e-9823-4988-ac79-0820dcc4c028\") " Nov 03 13:44:48.062019 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:48.061873 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3cd0d23e-9823-4988-ac79-0820dcc4c028-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cd0d23e-9823-4988-ac79-0820dcc4c028-workload-socket\") pod \"3cd0d23e-9823-4988-ac79-0820dcc4c028\" (UID: \"3cd0d23e-9823-4988-ac79-0820dcc4c028\") " Nov 03 13:44:48.062019 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:48.061871 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b46130ce-1911-475e-8ec7-5204881eb5ea-workload-certs" pod "b46130ce-1911-475e-8ec7-5204881eb5ea" (UID: "b46130ce-1911-475e-8ec7-5204881eb5ea"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:44:48.062019 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:48.062011 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b46130ce-1911-475e-8ec7-5204881eb5ea-credential-socket" pod "b46130ce-1911-475e-8ec7-5204881eb5ea" (UID: "b46130ce-1911-475e-8ec7-5204881eb5ea"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:44:48.062540 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:48.062121 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b46130ce-1911-475e-8ec7-5204881eb5ea-istio-data" pod "b46130ce-1911-475e-8ec7-5204881eb5ea" (UID: "b46130ce-1911-475e-8ec7-5204881eb5ea"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:44:48.062540 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:48.062394 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3cd0d23e-9823-4988-ac79-0820dcc4c028-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3cd0d23e-9823-4988-ac79-0820dcc4c028-workload-certs\") pod \"3cd0d23e-9823-4988-ac79-0820dcc4c028\" (UID: \"3cd0d23e-9823-4988-ac79-0820dcc4c028\") " Nov 03 13:44:48.062540 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:48.062503 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b46130ce-1911-475e-8ec7-5204881eb5ea-kube-api-access-q8q4b\" (UniqueName: \"kubernetes.io/projected/b46130ce-1911-475e-8ec7-5204881eb5ea-kube-api-access-q8q4b\") pod \"b46130ce-1911-475e-8ec7-5204881eb5ea\" (UID: \"b46130ce-1911-475e-8ec7-5204881eb5ea\") " Nov 03 13:44:48.062711 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:48.062541 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3cd0d23e-9823-4988-ac79-0820dcc4c028-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3cd0d23e-9823-4988-ac79-0820dcc4c028-istiod-ca-cert\") pod \"3cd0d23e-9823-4988-ac79-0820dcc4c028\" (UID: \"3cd0d23e-9823-4988-ac79-0820dcc4c028\") " Nov 03 13:44:48.062711 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:48.062612 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3cd0d23e-9823-4988-ac79-0820dcc4c028-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cd0d23e-9823-4988-ac79-0820dcc4c028-credential-socket\") pod \"3cd0d23e-9823-4988-ac79-0820dcc4c028\" (UID: \"3cd0d23e-9823-4988-ac79-0820dcc4c028\") " Nov 03 13:44:48.062711 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:48.062666 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b46130ce-1911-475e-8ec7-5204881eb5ea-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b46130ce-1911-475e-8ec7-5204881eb5ea-istiod-ca-cert\") pod \"b46130ce-1911-475e-8ec7-5204881eb5ea\" (UID: \"b46130ce-1911-475e-8ec7-5204881eb5ea\") " Nov 03 13:44:48.063336 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:48.062393 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3cd0d23e-9823-4988-ac79-0820dcc4c028-istio-data" pod "3cd0d23e-9823-4988-ac79-0820dcc4c028" (UID: "3cd0d23e-9823-4988-ac79-0820dcc4c028"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:44:48.063336 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:48.062432 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3cd0d23e-9823-4988-ac79-0820dcc4c028-workload-socket" pod "3cd0d23e-9823-4988-ac79-0820dcc4c028" (UID: "3cd0d23e-9823-4988-ac79-0820dcc4c028"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:44:48.063336 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:48.062644 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3cd0d23e-9823-4988-ac79-0820dcc4c028-workload-certs" pod "3cd0d23e-9823-4988-ac79-0820dcc4c028" (UID: "3cd0d23e-9823-4988-ac79-0820dcc4c028"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:44:48.063336 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:48.063273 5024 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/155ba865-dcfd-4203-82b9-5399ca58d988-credential-socket\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:44:48.063336 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:48.063314 5024 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b46130ce-1911-475e-8ec7-5204881eb5ea-workload-socket\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:44:48.063336 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:48.063312 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3cd0d23e-9823-4988-ac79-0820dcc4c028-istiod-ca-cert" pod "3cd0d23e-9823-4988-ac79-0820dcc4c028" (UID: "3cd0d23e-9823-4988-ac79-0820dcc4c028"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:44:48.063336 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:48.063331 5024 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b46130ce-1911-475e-8ec7-5204881eb5ea-workload-certs\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:44:48.063336 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:48.063316 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3cd0d23e-9823-4988-ac79-0820dcc4c028-credential-socket" pod "3cd0d23e-9823-4988-ac79-0820dcc4c028" (UID: "3cd0d23e-9823-4988-ac79-0820dcc4c028"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:44:48.064379 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:48.063346 5024 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/155ba865-dcfd-4203-82b9-5399ca58d988-istio-envoy\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:44:48.064379 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:48.063360 5024 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b46130ce-1911-475e-8ec7-5204881eb5ea-istio-data\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:44:48.064379 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:48.063374 5024 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/155ba865-dcfd-4203-82b9-5399ca58d988-istio-token\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:44:48.064379 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:48.063388 5024 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3cd0d23e-9823-4988-ac79-0820dcc4c028-istio-data\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:44:48.064379 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:48.063405 5024 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b46130ce-1911-475e-8ec7-5204881eb5ea-credential-socket\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:44:48.064379 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:48.063421 5024 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/155ba865-dcfd-4203-82b9-5399ca58d988-istiod-ca-cert\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:44:48.064379 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:48.063436 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sqbdt\" (UniqueName: \"kubernetes.io/projected/155ba865-dcfd-4203-82b9-5399ca58d988-kube-api-access-sqbdt\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:44:48.064379 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:48.063452 5024 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cd0d23e-9823-4988-ac79-0820dcc4c028-workload-socket\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:44:48.064379 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:48.063466 5024 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3cd0d23e-9823-4988-ac79-0820dcc4c028-workload-certs\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:44:48.064379 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:48.063481 5024 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/155ba865-dcfd-4203-82b9-5399ca58d988-istio-data\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:44:48.064379 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:48.063496 5024 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/155ba865-dcfd-4203-82b9-5399ca58d988-workload-socket\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:44:48.064379 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:48.063511 5024 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/155ba865-dcfd-4203-82b9-5399ca58d988-workload-certs\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:44:48.064379 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:48.063526 5024 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/155ba865-dcfd-4203-82b9-5399ca58d988-istio-podinfo\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:44:48.064379 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:48.063855 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b46130ce-1911-475e-8ec7-5204881eb5ea-istiod-ca-cert" pod "b46130ce-1911-475e-8ec7-5204881eb5ea" (UID: "b46130ce-1911-475e-8ec7-5204881eb5ea"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:44:48.064379 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:48.064246 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b46130ce-1911-475e-8ec7-5204881eb5ea-istio-token" pod "b46130ce-1911-475e-8ec7-5204881eb5ea" (UID: "b46130ce-1911-475e-8ec7-5204881eb5ea"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:44:48.065190 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:48.064678 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b46130ce-1911-475e-8ec7-5204881eb5ea-istio-envoy" pod "b46130ce-1911-475e-8ec7-5204881eb5ea" (UID: "b46130ce-1911-475e-8ec7-5204881eb5ea"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:44:48.065190 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:48.064695 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3cd0d23e-9823-4988-ac79-0820dcc4c028-kube-api-access-rhbfk" pod "3cd0d23e-9823-4988-ac79-0820dcc4c028" (UID: "3cd0d23e-9823-4988-ac79-0820dcc4c028"). InnerVolumeSpecName "kube-api-access-rhbfk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:44:48.065190 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:48.064907 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3cd0d23e-9823-4988-ac79-0820dcc4c028-istio-podinfo" pod "3cd0d23e-9823-4988-ac79-0820dcc4c028" (UID: "3cd0d23e-9823-4988-ac79-0820dcc4c028"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 13:44:48.065190 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:48.064930 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3cd0d23e-9823-4988-ac79-0820dcc4c028-istio-token" pod "3cd0d23e-9823-4988-ac79-0820dcc4c028" (UID: "3cd0d23e-9823-4988-ac79-0820dcc4c028"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:44:48.065527 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:48.065509 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b46130ce-1911-475e-8ec7-5204881eb5ea-istio-podinfo" pod "b46130ce-1911-475e-8ec7-5204881eb5ea" (UID: "b46130ce-1911-475e-8ec7-5204881eb5ea"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 13:44:48.065611 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:48.065596 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3cd0d23e-9823-4988-ac79-0820dcc4c028-istio-envoy" pod "3cd0d23e-9823-4988-ac79-0820dcc4c028" (UID: "3cd0d23e-9823-4988-ac79-0820dcc4c028"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:44:48.065701 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:48.065681 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b46130ce-1911-475e-8ec7-5204881eb5ea-kube-api-access-q8q4b" pod "b46130ce-1911-475e-8ec7-5204881eb5ea" (UID: "b46130ce-1911-475e-8ec7-5204881eb5ea"). InnerVolumeSpecName "kube-api-access-q8q4b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:44:48.164296 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:48.164257 5024 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3cd0d23e-9823-4988-ac79-0820dcc4c028-istio-envoy\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:44:48.164296 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:48.164300 5024 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b46130ce-1911-475e-8ec7-5204881eb5ea-istio-podinfo\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:44:48.164481 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:48.164312 5024 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b46130ce-1911-475e-8ec7-5204881eb5ea-istio-token\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:44:48.164481 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:48.164321 5024 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3cd0d23e-9823-4988-ac79-0820dcc4c028-istio-podinfo\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:44:48.164481 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:48.164330 5024 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3cd0d23e-9823-4988-ac79-0820dcc4c028-istio-token\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:44:48.164481 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:48.164340 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q8q4b\" (UniqueName: \"kubernetes.io/projected/b46130ce-1911-475e-8ec7-5204881eb5ea-kube-api-access-q8q4b\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:44:48.164481 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:48.164354 5024 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3cd0d23e-9823-4988-ac79-0820dcc4c028-istiod-ca-cert\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:44:48.164481 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:48.164363 5024 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cd0d23e-9823-4988-ac79-0820dcc4c028-credential-socket\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:44:48.164481 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:48.164371 5024 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b46130ce-1911-475e-8ec7-5204881eb5ea-istiod-ca-cert\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:44:48.164481 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:48.164380 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rhbfk\" (UniqueName: \"kubernetes.io/projected/3cd0d23e-9823-4988-ac79-0820dcc4c028-kube-api-access-rhbfk\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:44:48.164481 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:48.164389 5024 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b46130ce-1911-475e-8ec7-5204881eb5ea-istio-envoy\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:44:48.184023 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:48.183990 5024 generic.go:356] "Generic (PLEG): container finished" podID="155ba865-dcfd-4203-82b9-5399ca58d988" containerID="b497e622f8a4363b340c1d0fe74937acbcd77a6120d6e2e07dbfd73c7c50b244" exitCode=0 Nov 03 13:44:48.184181 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:48.184065 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-h2ns8" Nov 03 13:44:48.184253 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:48.184065 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-h2ns8" event={"ID":"155ba865-dcfd-4203-82b9-5399ca58d988","Type":"ContainerDied","Data":"b497e622f8a4363b340c1d0fe74937acbcd77a6120d6e2e07dbfd73c7c50b244"} Nov 03 13:44:48.184333 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:48.184311 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-h2ns8" event={"ID":"155ba865-dcfd-4203-82b9-5399ca58d988","Type":"ContainerDied","Data":"d83b13849381eef46d6fec8f5fc1bdf4c3aff737f82ce61a1042a2ef26b51f98"} Nov 03 13:44:48.184379 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:48.184354 5024 scope.go:122] "RemoveContainer" containerID="b497e622f8a4363b340c1d0fe74937acbcd77a6120d6e2e07dbfd73c7c50b244" Nov 03 13:44:48.185661 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:48.185635 5024 generic.go:356] "Generic (PLEG): container finished" podID="3cd0d23e-9823-4988-ac79-0820dcc4c028" containerID="8cef8680dcbf34f6e439e6c1d96440fe7a53d66759de93261895d1190c9769ed" exitCode=0 Nov 03 13:44:48.185879 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:48.185663 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-nr5hn" event={"ID":"3cd0d23e-9823-4988-ac79-0820dcc4c028","Type":"ContainerDied","Data":"8cef8680dcbf34f6e439e6c1d96440fe7a53d66759de93261895d1190c9769ed"} Nov 03 13:44:48.185950 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:48.185920 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-nr5hn" event={"ID":"3cd0d23e-9823-4988-ac79-0820dcc4c028","Type":"ContainerDied","Data":"7d41b7cc92ccd825236e58107052a0126aee2168b939255dcb973cdcdcda12e2"} Nov 03 13:44:48.185950 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:48.185696 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-nr5hn" Nov 03 13:44:48.191581 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:48.191556 5024 generic.go:356] "Generic (PLEG): container finished" podID="b46130ce-1911-475e-8ec7-5204881eb5ea" containerID="e525aba39eee600786bacf1d3950338c7cc4baaa122bc1eb2d7be06c938f0d0d" exitCode=0 Nov 03 13:44:48.191696 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:48.191587 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-xtsb8" event={"ID":"b46130ce-1911-475e-8ec7-5204881eb5ea","Type":"ContainerDied","Data":"e525aba39eee600786bacf1d3950338c7cc4baaa122bc1eb2d7be06c938f0d0d"} Nov 03 13:44:48.191696 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:48.191621 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-xtsb8" Nov 03 13:44:48.191818 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:48.191627 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-xtsb8" event={"ID":"b46130ce-1911-475e-8ec7-5204881eb5ea","Type":"ContainerDied","Data":"4009da59fe60b3d6bcc8fb1673cd92346a5af2a6272d483784d4af75572ef4c5"} Nov 03 13:44:48.205847 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:48.205826 5024 scope.go:122] "RemoveContainer" containerID="b497e622f8a4363b340c1d0fe74937acbcd77a6120d6e2e07dbfd73c7c50b244" Nov 03 13:44:48.206171 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:44:48.206149 5024 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b497e622f8a4363b340c1d0fe74937acbcd77a6120d6e2e07dbfd73c7c50b244\": container with ID starting with b497e622f8a4363b340c1d0fe74937acbcd77a6120d6e2e07dbfd73c7c50b244 not found: ID does not exist" containerID="b497e622f8a4363b340c1d0fe74937acbcd77a6120d6e2e07dbfd73c7c50b244" Nov 03 13:44:48.206252 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:48.206177 5024 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b497e622f8a4363b340c1d0fe74937acbcd77a6120d6e2e07dbfd73c7c50b244"} err="failed to get container status \"b497e622f8a4363b340c1d0fe74937acbcd77a6120d6e2e07dbfd73c7c50b244\": rpc error: code = NotFound desc = could not find container \"b497e622f8a4363b340c1d0fe74937acbcd77a6120d6e2e07dbfd73c7c50b244\": container with ID starting with b497e622f8a4363b340c1d0fe74937acbcd77a6120d6e2e07dbfd73c7c50b244 not found: ID does not exist" Nov 03 13:44:48.206252 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:48.206193 5024 scope.go:122] "RemoveContainer" containerID="8cef8680dcbf34f6e439e6c1d96440fe7a53d66759de93261895d1190c9769ed" Nov 03 13:44:48.218359 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:48.218325 5024 scope.go:122] "RemoveContainer" containerID="8cef8680dcbf34f6e439e6c1d96440fe7a53d66759de93261895d1190c9769ed" Nov 03 13:44:48.218786 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:44:48.218764 5024 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8cef8680dcbf34f6e439e6c1d96440fe7a53d66759de93261895d1190c9769ed\": container with ID starting with 8cef8680dcbf34f6e439e6c1d96440fe7a53d66759de93261895d1190c9769ed not found: ID does not exist" containerID="8cef8680dcbf34f6e439e6c1d96440fe7a53d66759de93261895d1190c9769ed" Nov 03 13:44:48.218875 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:48.218795 5024 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8cef8680dcbf34f6e439e6c1d96440fe7a53d66759de93261895d1190c9769ed"} err="failed to get container status \"8cef8680dcbf34f6e439e6c1d96440fe7a53d66759de93261895d1190c9769ed\": rpc error: code = NotFound desc = could not find container \"8cef8680dcbf34f6e439e6c1d96440fe7a53d66759de93261895d1190c9769ed\": container with ID starting with 8cef8680dcbf34f6e439e6c1d96440fe7a53d66759de93261895d1190c9769ed not found: ID does not exist" Nov 03 13:44:48.218875 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:48.218816 5024 scope.go:122] "RemoveContainer" containerID="e525aba39eee600786bacf1d3950338c7cc4baaa122bc1eb2d7be06c938f0d0d" Nov 03 13:44:48.221423 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:48.221396 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-84587756dc-nr5hn"] Nov 03 13:44:48.229434 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:48.229416 5024 scope.go:122] "RemoveContainer" containerID="e525aba39eee600786bacf1d3950338c7cc4baaa122bc1eb2d7be06c938f0d0d" Nov 03 13:44:48.229752 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:44:48.229731 5024 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e525aba39eee600786bacf1d3950338c7cc4baaa122bc1eb2d7be06c938f0d0d\": container with ID starting with e525aba39eee600786bacf1d3950338c7cc4baaa122bc1eb2d7be06c938f0d0d not found: ID does not exist" containerID="e525aba39eee600786bacf1d3950338c7cc4baaa122bc1eb2d7be06c938f0d0d" Nov 03 13:44:48.229853 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:48.229756 5024 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e525aba39eee600786bacf1d3950338c7cc4baaa122bc1eb2d7be06c938f0d0d"} err="failed to get container status \"e525aba39eee600786bacf1d3950338c7cc4baaa122bc1eb2d7be06c938f0d0d\": rpc error: code = NotFound desc = could not find container \"e525aba39eee600786bacf1d3950338c7cc4baaa122bc1eb2d7be06c938f0d0d\": container with ID starting with e525aba39eee600786bacf1d3950338c7cc4baaa122bc1eb2d7be06c938f0d0d not found: ID does not exist" Nov 03 13:44:48.230559 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:48.230539 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-84587756dc-nr5hn"] Nov 03 13:44:48.250052 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:48.250030 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-84587756dc-h2ns8"] Nov 03 13:44:48.256872 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:48.256850 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-84587756dc-h2ns8"] Nov 03 13:44:48.271276 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:48.271250 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-84587756dc-xtsb8"] Nov 03 13:44:48.278695 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:48.278673 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-84587756dc-xtsb8"] Nov 03 13:44:48.312512 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:48.312457 5024 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w1-08111b-9cc448f95-n6ttb" podUID="c4490dbf-a557-44df-b654-b7a47f5c0604" containerName="payload-processing" containerID="cri-o://33652f2e5fd1ab20dc35880a4a41b3da56699adebf21290475a1c9d3fba35f1b" gracePeriod=25 Nov 03 13:44:48.327002 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:48.326973 5024 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-08111b-744df4c7d7-b5xcs" podUID="459af5d0-4e46-4ad0-b123-5642ceae6e74" containerName="payload-pre-processing" containerID="cri-o://86d550975a2ec9158fd99e9d700a0502ee24107c5ee979b725760987d8f292be" gracePeriod=25 Nov 03 13:44:48.581220 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:48.581189 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-08111b-9cc448f95-n6ttb" Nov 03 13:44:48.666068 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:48.666042 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-08111b-744df4c7d7-b5xcs" Nov 03 13:44:48.670026 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:48.670004 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c4490dbf-a557-44df-b654-b7a47f5c0604-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4490dbf-a557-44df-b654-b7a47f5c0604-plugins-config-volume\") pod \"c4490dbf-a557-44df-b654-b7a47f5c0604\" (UID: \"c4490dbf-a557-44df-b654-b7a47f5c0604\") " Nov 03 13:44:48.670161 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:48.670051 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4490dbf-a557-44df-b654-b7a47f5c0604-kube-api-access-9zclm\" (UniqueName: \"kubernetes.io/projected/c4490dbf-a557-44df-b654-b7a47f5c0604-kube-api-access-9zclm\") pod \"c4490dbf-a557-44df-b654-b7a47f5c0604\" (UID: \"c4490dbf-a557-44df-b654-b7a47f5c0604\") " Nov 03 13:44:48.670392 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:48.670370 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4490dbf-a557-44df-b654-b7a47f5c0604-plugins-config-volume" pod "c4490dbf-a557-44df-b654-b7a47f5c0604" (UID: "c4490dbf-a557-44df-b654-b7a47f5c0604"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:44:48.670534 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:48.670513 5024 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4490dbf-a557-44df-b654-b7a47f5c0604-plugins-config-volume\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:44:48.672280 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:48.672254 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4490dbf-a557-44df-b654-b7a47f5c0604-kube-api-access-9zclm" pod "c4490dbf-a557-44df-b654-b7a47f5c0604" (UID: "c4490dbf-a557-44df-b654-b7a47f5c0604"). InnerVolumeSpecName "kube-api-access-9zclm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:44:48.771419 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:48.771329 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/459af5d0-4e46-4ad0-b123-5642ceae6e74-kube-api-access-b8j7s\" (UniqueName: \"kubernetes.io/projected/459af5d0-4e46-4ad0-b123-5642ceae6e74-kube-api-access-b8j7s\") pod \"459af5d0-4e46-4ad0-b123-5642ceae6e74\" (UID: \"459af5d0-4e46-4ad0-b123-5642ceae6e74\") " Nov 03 13:44:48.771589 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:48.771568 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/459af5d0-4e46-4ad0-b123-5642ceae6e74-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/459af5d0-4e46-4ad0-b123-5642ceae6e74-plugins-config-volume\") pod \"459af5d0-4e46-4ad0-b123-5642ceae6e74\" (UID: \"459af5d0-4e46-4ad0-b123-5642ceae6e74\") " Nov 03 13:44:48.771974 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:48.771956 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9zclm\" (UniqueName: \"kubernetes.io/projected/c4490dbf-a557-44df-b654-b7a47f5c0604-kube-api-access-9zclm\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:44:48.772025 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:48.771997 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/459af5d0-4e46-4ad0-b123-5642ceae6e74-plugins-config-volume" pod "459af5d0-4e46-4ad0-b123-5642ceae6e74" (UID: "459af5d0-4e46-4ad0-b123-5642ceae6e74"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:44:48.773437 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:48.773413 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/459af5d0-4e46-4ad0-b123-5642ceae6e74-kube-api-access-b8j7s" pod "459af5d0-4e46-4ad0-b123-5642ceae6e74" (UID: "459af5d0-4e46-4ad0-b123-5642ceae6e74"). InnerVolumeSpecName "kube-api-access-b8j7s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:44:48.873391 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:48.873364 5024 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/459af5d0-4e46-4ad0-b123-5642ceae6e74-plugins-config-volume\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:44:48.873391 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:48.873390 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b8j7s\" (UniqueName: \"kubernetes.io/projected/459af5d0-4e46-4ad0-b123-5642ceae6e74-kube-api-access-b8j7s\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:44:49.199192 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:49.199153 5024 generic.go:356] "Generic (PLEG): container finished" podID="459af5d0-4e46-4ad0-b123-5642ceae6e74" containerID="86d550975a2ec9158fd99e9d700a0502ee24107c5ee979b725760987d8f292be" exitCode=0 Nov 03 13:44:49.199659 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:49.199207 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-08111b-744df4c7d7-b5xcs" Nov 03 13:44:49.199659 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:49.199237 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-08111b-744df4c7d7-b5xcs" event={"ID":"459af5d0-4e46-4ad0-b123-5642ceae6e74","Type":"ContainerDied","Data":"86d550975a2ec9158fd99e9d700a0502ee24107c5ee979b725760987d8f292be"} Nov 03 13:44:49.199659 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:49.199338 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-08111b-744df4c7d7-b5xcs" event={"ID":"459af5d0-4e46-4ad0-b123-5642ceae6e74","Type":"ContainerDied","Data":"4b92eb884b44004c08712a47feb1b3b2035fef8b1233f7ec0989559f820d691b"} Nov 03 13:44:49.199659 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:49.199374 5024 scope.go:122] "RemoveContainer" containerID="86d550975a2ec9158fd99e9d700a0502ee24107c5ee979b725760987d8f292be" Nov 03 13:44:49.201912 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:49.201886 5024 generic.go:356] "Generic (PLEG): container finished" podID="c4490dbf-a557-44df-b654-b7a47f5c0604" containerID="33652f2e5fd1ab20dc35880a4a41b3da56699adebf21290475a1c9d3fba35f1b" exitCode=0 Nov 03 13:44:49.202056 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:49.201930 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-08111b-9cc448f95-n6ttb" Nov 03 13:44:49.202056 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:49.201956 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-08111b-9cc448f95-n6ttb" event={"ID":"c4490dbf-a557-44df-b654-b7a47f5c0604","Type":"ContainerDied","Data":"33652f2e5fd1ab20dc35880a4a41b3da56699adebf21290475a1c9d3fba35f1b"} Nov 03 13:44:49.202056 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:49.202010 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-08111b-9cc448f95-n6ttb" event={"ID":"c4490dbf-a557-44df-b654-b7a47f5c0604","Type":"ContainerDied","Data":"468e3c8558af17ac063448bf1838df60f8c635637ad1e5abc1924e72bac1f7dd"} Nov 03 13:44:49.210341 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:49.210317 5024 scope.go:122] "RemoveContainer" containerID="86d550975a2ec9158fd99e9d700a0502ee24107c5ee979b725760987d8f292be" Nov 03 13:44:49.210620 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:44:49.210593 5024 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"86d550975a2ec9158fd99e9d700a0502ee24107c5ee979b725760987d8f292be\": container with ID starting with 86d550975a2ec9158fd99e9d700a0502ee24107c5ee979b725760987d8f292be not found: ID does not exist" containerID="86d550975a2ec9158fd99e9d700a0502ee24107c5ee979b725760987d8f292be" Nov 03 13:44:49.210733 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:49.210627 5024 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86d550975a2ec9158fd99e9d700a0502ee24107c5ee979b725760987d8f292be"} err="failed to get container status \"86d550975a2ec9158fd99e9d700a0502ee24107c5ee979b725760987d8f292be\": rpc error: code = NotFound desc = could not find container \"86d550975a2ec9158fd99e9d700a0502ee24107c5ee979b725760987d8f292be\": container with ID starting with 86d550975a2ec9158fd99e9d700a0502ee24107c5ee979b725760987d8f292be not found: ID does not exist" Nov 03 13:44:49.210733 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:49.210651 5024 scope.go:122] "RemoveContainer" containerID="33652f2e5fd1ab20dc35880a4a41b3da56699adebf21290475a1c9d3fba35f1b" Nov 03 13:44:49.220344 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:49.220320 5024 scope.go:122] "RemoveContainer" containerID="33652f2e5fd1ab20dc35880a4a41b3da56699adebf21290475a1c9d3fba35f1b" Nov 03 13:44:49.220582 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:44:49.220561 5024 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"33652f2e5fd1ab20dc35880a4a41b3da56699adebf21290475a1c9d3fba35f1b\": container with ID starting with 33652f2e5fd1ab20dc35880a4a41b3da56699adebf21290475a1c9d3fba35f1b not found: ID does not exist" containerID="33652f2e5fd1ab20dc35880a4a41b3da56699adebf21290475a1c9d3fba35f1b" Nov 03 13:44:49.220659 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:49.220591 5024 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33652f2e5fd1ab20dc35880a4a41b3da56699adebf21290475a1c9d3fba35f1b"} err="failed to get container status \"33652f2e5fd1ab20dc35880a4a41b3da56699adebf21290475a1c9d3fba35f1b\": rpc error: code = NotFound desc = could not find container \"33652f2e5fd1ab20dc35880a4a41b3da56699adebf21290475a1c9d3fba35f1b\": container with ID starting with 33652f2e5fd1ab20dc35880a4a41b3da56699adebf21290475a1c9d3fba35f1b not found: ID does not exist" Nov 03 13:44:49.235134 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:49.235111 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-08111b-9cc448f95-n6ttb"] Nov 03 13:44:49.242519 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:49.242492 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-08111b-9cc448f95-n6ttb"] Nov 03 13:44:49.258641 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:49.258612 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-08111b-744df4c7d7-b5xcs"] Nov 03 13:44:49.266848 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:49.266821 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-08111b-744df4c7d7-b5xcs"] Nov 03 13:44:50.111910 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:50.111871 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="155ba865-dcfd-4203-82b9-5399ca58d988" path="/var/lib/kubelet/pods/155ba865-dcfd-4203-82b9-5399ca58d988/volumes" Nov 03 13:44:50.112615 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:50.112587 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3cd0d23e-9823-4988-ac79-0820dcc4c028" path="/var/lib/kubelet/pods/3cd0d23e-9823-4988-ac79-0820dcc4c028/volumes" Nov 03 13:44:50.113121 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:50.113104 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="459af5d0-4e46-4ad0-b123-5642ceae6e74" path="/var/lib/kubelet/pods/459af5d0-4e46-4ad0-b123-5642ceae6e74/volumes" Nov 03 13:44:50.114183 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:50.114163 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b46130ce-1911-475e-8ec7-5204881eb5ea" path="/var/lib/kubelet/pods/b46130ce-1911-475e-8ec7-5204881eb5ea/volumes" Nov 03 13:44:50.114718 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:50.114700 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c4490dbf-a557-44df-b654-b7a47f5c0604" path="/var/lib/kubelet/pods/c4490dbf-a557-44df-b654-b7a47f5c0604/volumes" Nov 03 13:44:50.266691 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:50.266656 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-9add72-6b8fc4988f-kxlrl"] Nov 03 13:44:50.404830 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:50.404732 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-9add72-f74fc96d-xgkrk"] Nov 03 13:44:50.405008 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:50.404975 5024 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-9add72-f74fc96d-xgkrk" podUID="fa15c0e4-6d08-4a12-9320-687d6f47ed4e" containerName="maas-api" containerID="cri-o://e661833ee83ad6f141e043af449aa4d1477fde8be130663c70aa02cbed593871" gracePeriod=30 Nov 03 13:44:50.643914 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:50.643875 5024 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-08111b/premium-w1-08111b-kserve-57555d598-dx9g5" podUID="e45c569a-243f-49ab-8d74-dd08cbb6127c" containerName="main" containerID="cri-o://f260365e7cd791f7ee4cb0c4526ddb1047ed8821c14214cef7e33c1fae97190f" gracePeriod=45 Nov 03 13:44:50.650925 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:50.650890 5024 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-08111b/simulator-w1-08111b-kserve-7f65c5867-btf5b" podUID="e8aba665-e4cc-4e67-8cdd-d09ef7542e15" containerName="main" containerID="cri-o://448f466488f569306e0803889b69dcea09f90745995e4723f29cb7c4088320cb" gracePeriod=45 Nov 03 13:44:50.771131 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:50.770847 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-9add72-f74fc96d-xgkrk" Nov 03 13:44:50.937401 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:50.921511 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa15c0e4-6d08-4a12-9320-687d6f47ed4e-kube-api-access-phvzn\" (UniqueName: \"kubernetes.io/projected/fa15c0e4-6d08-4a12-9320-687d6f47ed4e-kube-api-access-phvzn\") pod \"fa15c0e4-6d08-4a12-9320-687d6f47ed4e\" (UID: \"fa15c0e4-6d08-4a12-9320-687d6f47ed4e\") " Nov 03 13:44:50.937401 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:50.921615 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fa15c0e4-6d08-4a12-9320-687d6f47ed4e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fa15c0e4-6d08-4a12-9320-687d6f47ed4e-maas-api-tls\") pod \"fa15c0e4-6d08-4a12-9320-687d6f47ed4e\" (UID: \"fa15c0e4-6d08-4a12-9320-687d6f47ed4e\") " Nov 03 13:44:50.937401 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:50.921716 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fa15c0e4-6d08-4a12-9320-687d6f47ed4e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa15c0e4-6d08-4a12-9320-687d6f47ed4e-openshift-service-ca\") pod \"fa15c0e4-6d08-4a12-9320-687d6f47ed4e\" (UID: \"fa15c0e4-6d08-4a12-9320-687d6f47ed4e\") " Nov 03 13:44:50.937401 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:50.921776 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fa15c0e4-6d08-4a12-9320-687d6f47ed4e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/fa15c0e4-6d08-4a12-9320-687d6f47ed4e-metrics-certs\") pod \"fa15c0e4-6d08-4a12-9320-687d6f47ed4e\" (UID: \"fa15c0e4-6d08-4a12-9320-687d6f47ed4e\") " Nov 03 13:44:50.937401 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:50.926578 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa15c0e4-6d08-4a12-9320-687d6f47ed4e-openshift-service-ca" pod "fa15c0e4-6d08-4a12-9320-687d6f47ed4e" (UID: "fa15c0e4-6d08-4a12-9320-687d6f47ed4e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:44:50.950561 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:50.950511 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fa15c0e4-6d08-4a12-9320-687d6f47ed4e-maas-api-tls" pod "fa15c0e4-6d08-4a12-9320-687d6f47ed4e" (UID: "fa15c0e4-6d08-4a12-9320-687d6f47ed4e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 13:44:50.952570 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:50.952522 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fa15c0e4-6d08-4a12-9320-687d6f47ed4e-metrics-certs" pod "fa15c0e4-6d08-4a12-9320-687d6f47ed4e" (UID: "fa15c0e4-6d08-4a12-9320-687d6f47ed4e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 13:44:50.952873 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:50.952848 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa15c0e4-6d08-4a12-9320-687d6f47ed4e-kube-api-access-phvzn" pod "fa15c0e4-6d08-4a12-9320-687d6f47ed4e" (UID: "fa15c0e4-6d08-4a12-9320-687d6f47ed4e"). InnerVolumeSpecName "kube-api-access-phvzn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:44:50.957462 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:50.957441 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-08111b/simulator-w1-08111b-kserve-7f65c5867-btf5b" Nov 03 13:44:50.969436 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:50.969416 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-08111b/premium-w1-08111b-kserve-57555d598-dx9g5" Nov 03 13:44:51.027012 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:51.026969 5024 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa15c0e4-6d08-4a12-9320-687d6f47ed4e-openshift-service-ca\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:44:51.027323 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:51.027032 5024 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fa15c0e4-6d08-4a12-9320-687d6f47ed4e-metrics-certs\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:44:51.027323 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:51.027049 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-phvzn\" (UniqueName: \"kubernetes.io/projected/fa15c0e4-6d08-4a12-9320-687d6f47ed4e-kube-api-access-phvzn\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:44:51.027323 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:51.027066 5024 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fa15c0e4-6d08-4a12-9320-687d6f47ed4e-maas-api-tls\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:44:51.128304 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:51.128247 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8aba665-e4cc-4e67-8cdd-d09ef7542e15-home\" (UniqueName: \"kubernetes.io/empty-dir/e8aba665-e4cc-4e67-8cdd-d09ef7542e15-home\") pod \"e8aba665-e4cc-4e67-8cdd-d09ef7542e15\" (UID: \"e8aba665-e4cc-4e67-8cdd-d09ef7542e15\") " Nov 03 13:44:51.128486 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:51.128331 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8aba665-e4cc-4e67-8cdd-d09ef7542e15-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e8aba665-e4cc-4e67-8cdd-d09ef7542e15-model-cache\") pod \"e8aba665-e4cc-4e67-8cdd-d09ef7542e15\" (UID: \"e8aba665-e4cc-4e67-8cdd-d09ef7542e15\") " Nov 03 13:44:51.128486 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:51.128368 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8aba665-e4cc-4e67-8cdd-d09ef7542e15-kube-api-access-d9fbk\" (UniqueName: \"kubernetes.io/projected/e8aba665-e4cc-4e67-8cdd-d09ef7542e15-kube-api-access-d9fbk\") pod \"e8aba665-e4cc-4e67-8cdd-d09ef7542e15\" (UID: \"e8aba665-e4cc-4e67-8cdd-d09ef7542e15\") " Nov 03 13:44:51.128486 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:51.128399 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e45c569a-243f-49ab-8d74-dd08cbb6127c-home\" (UniqueName: \"kubernetes.io/empty-dir/e45c569a-243f-49ab-8d74-dd08cbb6127c-home\") pod \"e45c569a-243f-49ab-8d74-dd08cbb6127c\" (UID: \"e45c569a-243f-49ab-8d74-dd08cbb6127c\") " Nov 03 13:44:51.128668 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:51.128487 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e8aba665-e4cc-4e67-8cdd-d09ef7542e15-tls-certs\" (UniqueName: \"kubernetes.io/secret/e8aba665-e4cc-4e67-8cdd-d09ef7542e15-tls-certs\") pod \"e8aba665-e4cc-4e67-8cdd-d09ef7542e15\" (UID: \"e8aba665-e4cc-4e67-8cdd-d09ef7542e15\") " Nov 03 13:44:51.128668 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:51.128536 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e45c569a-243f-49ab-8d74-dd08cbb6127c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e45c569a-243f-49ab-8d74-dd08cbb6127c-model-cache\") pod \"e45c569a-243f-49ab-8d74-dd08cbb6127c\" (UID: \"e45c569a-243f-49ab-8d74-dd08cbb6127c\") " Nov 03 13:44:51.128668 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:51.128562 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e45c569a-243f-49ab-8d74-dd08cbb6127c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e45c569a-243f-49ab-8d74-dd08cbb6127c-dshm\") pod \"e45c569a-243f-49ab-8d74-dd08cbb6127c\" (UID: \"e45c569a-243f-49ab-8d74-dd08cbb6127c\") " Nov 03 13:44:51.128668 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:51.128608 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8aba665-e4cc-4e67-8cdd-d09ef7542e15-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e8aba665-e4cc-4e67-8cdd-d09ef7542e15-dshm\") pod \"e8aba665-e4cc-4e67-8cdd-d09ef7542e15\" (UID: \"e8aba665-e4cc-4e67-8cdd-d09ef7542e15\") " Nov 03 13:44:51.128668 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:51.128620 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8aba665-e4cc-4e67-8cdd-d09ef7542e15-home" pod "e8aba665-e4cc-4e67-8cdd-d09ef7542e15" (UID: "e8aba665-e4cc-4e67-8cdd-d09ef7542e15"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:44:51.128944 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:51.128666 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8aba665-e4cc-4e67-8cdd-d09ef7542e15-model-cache" pod "e8aba665-e4cc-4e67-8cdd-d09ef7542e15" (UID: "e8aba665-e4cc-4e67-8cdd-d09ef7542e15"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:44:51.128944 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:51.128679 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e45c569a-243f-49ab-8d74-dd08cbb6127c-kube-api-access-825gk\" (UniqueName: \"kubernetes.io/projected/e45c569a-243f-49ab-8d74-dd08cbb6127c-kube-api-access-825gk\") pod \"e45c569a-243f-49ab-8d74-dd08cbb6127c\" (UID: \"e45c569a-243f-49ab-8d74-dd08cbb6127c\") " Nov 03 13:44:51.128944 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:51.128780 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e45c569a-243f-49ab-8d74-dd08cbb6127c-home" pod "e45c569a-243f-49ab-8d74-dd08cbb6127c" (UID: "e45c569a-243f-49ab-8d74-dd08cbb6127c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:44:51.128944 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:51.128804 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e45c569a-243f-49ab-8d74-dd08cbb6127c-model-cache" pod "e45c569a-243f-49ab-8d74-dd08cbb6127c" (UID: "e45c569a-243f-49ab-8d74-dd08cbb6127c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:44:51.128944 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:51.128807 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e45c569a-243f-49ab-8d74-dd08cbb6127c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e45c569a-243f-49ab-8d74-dd08cbb6127c-tmp-dir\") pod \"e45c569a-243f-49ab-8d74-dd08cbb6127c\" (UID: \"e45c569a-243f-49ab-8d74-dd08cbb6127c\") " Nov 03 13:44:51.128944 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:51.128870 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e45c569a-243f-49ab-8d74-dd08cbb6127c-tls-certs\" (UniqueName: \"kubernetes.io/secret/e45c569a-243f-49ab-8d74-dd08cbb6127c-tls-certs\") pod \"e45c569a-243f-49ab-8d74-dd08cbb6127c\" (UID: \"e45c569a-243f-49ab-8d74-dd08cbb6127c\") " Nov 03 13:44:51.128944 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:51.128900 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8aba665-e4cc-4e67-8cdd-d09ef7542e15-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e8aba665-e4cc-4e67-8cdd-d09ef7542e15-tmp-dir\") pod \"e8aba665-e4cc-4e67-8cdd-d09ef7542e15\" (UID: \"e8aba665-e4cc-4e67-8cdd-d09ef7542e15\") " Nov 03 13:44:51.129725 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:51.129387 5024 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e8aba665-e4cc-4e67-8cdd-d09ef7542e15-home\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:44:51.129725 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:51.129415 5024 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e8aba665-e4cc-4e67-8cdd-d09ef7542e15-model-cache\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:44:51.129725 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:51.129431 5024 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e45c569a-243f-49ab-8d74-dd08cbb6127c-home\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:44:51.129725 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:51.129446 5024 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e45c569a-243f-49ab-8d74-dd08cbb6127c-model-cache\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:44:51.129725 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:51.129687 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8aba665-e4cc-4e67-8cdd-d09ef7542e15-tmp-dir" pod "e8aba665-e4cc-4e67-8cdd-d09ef7542e15" (UID: "e8aba665-e4cc-4e67-8cdd-d09ef7542e15"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:44:51.130267 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:51.130237 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e45c569a-243f-49ab-8d74-dd08cbb6127c-tmp-dir" pod "e45c569a-243f-49ab-8d74-dd08cbb6127c" (UID: "e45c569a-243f-49ab-8d74-dd08cbb6127c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:44:51.131239 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:51.131210 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8aba665-e4cc-4e67-8cdd-d09ef7542e15-kube-api-access-d9fbk" pod "e8aba665-e4cc-4e67-8cdd-d09ef7542e15" (UID: "e8aba665-e4cc-4e67-8cdd-d09ef7542e15"). InnerVolumeSpecName "kube-api-access-d9fbk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:44:51.131393 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:51.131273 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e8aba665-e4cc-4e67-8cdd-d09ef7542e15-tls-certs" pod "e8aba665-e4cc-4e67-8cdd-d09ef7542e15" (UID: "e8aba665-e4cc-4e67-8cdd-d09ef7542e15"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 13:44:51.131393 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:51.131318 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e45c569a-243f-49ab-8d74-dd08cbb6127c-kube-api-access-825gk" pod "e45c569a-243f-49ab-8d74-dd08cbb6127c" (UID: "e45c569a-243f-49ab-8d74-dd08cbb6127c"). InnerVolumeSpecName "kube-api-access-825gk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:44:51.132585 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:51.132546 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e45c569a-243f-49ab-8d74-dd08cbb6127c-tls-certs" pod "e45c569a-243f-49ab-8d74-dd08cbb6127c" (UID: "e45c569a-243f-49ab-8d74-dd08cbb6127c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 13:44:51.132671 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:51.132633 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e45c569a-243f-49ab-8d74-dd08cbb6127c-dshm" pod "e45c569a-243f-49ab-8d74-dd08cbb6127c" (UID: "e45c569a-243f-49ab-8d74-dd08cbb6127c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:44:51.133056 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:51.133030 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8aba665-e4cc-4e67-8cdd-d09ef7542e15-dshm" pod "e8aba665-e4cc-4e67-8cdd-d09ef7542e15" (UID: "e8aba665-e4cc-4e67-8cdd-d09ef7542e15"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:44:51.226133 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:51.226030 5024 generic.go:356] "Generic (PLEG): container finished" podID="e45c569a-243f-49ab-8d74-dd08cbb6127c" containerID="f260365e7cd791f7ee4cb0c4526ddb1047ed8821c14214cef7e33c1fae97190f" exitCode=0 Nov 03 13:44:51.226751 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:51.226635 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-08111b/premium-w1-08111b-kserve-57555d598-dx9g5" event={"ID":"e45c569a-243f-49ab-8d74-dd08cbb6127c","Type":"ContainerDied","Data":"f260365e7cd791f7ee4cb0c4526ddb1047ed8821c14214cef7e33c1fae97190f"} Nov 03 13:44:51.226751 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:51.226696 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-08111b/premium-w1-08111b-kserve-57555d598-dx9g5" event={"ID":"e45c569a-243f-49ab-8d74-dd08cbb6127c","Type":"ContainerDied","Data":"9ef8259972cb6062da22404b19f33369b60a1e1e9e188b5172dfed0946980748"} Nov 03 13:44:51.226751 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:51.226718 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-08111b/premium-w1-08111b-kserve-57555d598-dx9g5" Nov 03 13:44:51.227945 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:51.226730 5024 scope.go:122] "RemoveContainer" containerID="f260365e7cd791f7ee4cb0c4526ddb1047ed8821c14214cef7e33c1fae97190f" Nov 03 13:44:51.230249 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:51.230216 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d9fbk\" (UniqueName: \"kubernetes.io/projected/e8aba665-e4cc-4e67-8cdd-d09ef7542e15-kube-api-access-d9fbk\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:44:51.230384 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:51.230247 5024 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e8aba665-e4cc-4e67-8cdd-d09ef7542e15-tls-certs\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:44:51.230384 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:51.230264 5024 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e45c569a-243f-49ab-8d74-dd08cbb6127c-dshm\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:44:51.230384 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:51.230277 5024 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e8aba665-e4cc-4e67-8cdd-d09ef7542e15-dshm\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:44:51.230384 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:51.230306 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-825gk\" (UniqueName: \"kubernetes.io/projected/e45c569a-243f-49ab-8d74-dd08cbb6127c-kube-api-access-825gk\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:44:51.230384 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:51.230319 5024 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e45c569a-243f-49ab-8d74-dd08cbb6127c-tmp-dir\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:44:51.230384 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:51.230332 5024 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e45c569a-243f-49ab-8d74-dd08cbb6127c-tls-certs\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:44:51.230384 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:51.230352 5024 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e8aba665-e4cc-4e67-8cdd-d09ef7542e15-tmp-dir\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:44:51.230781 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:51.230719 5024 generic.go:356] "Generic (PLEG): container finished" podID="fa15c0e4-6d08-4a12-9320-687d6f47ed4e" containerID="e661833ee83ad6f141e043af449aa4d1477fde8be130663c70aa02cbed593871" exitCode=0 Nov 03 13:44:51.230888 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:51.230866 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-9add72-f74fc96d-xgkrk" event={"ID":"fa15c0e4-6d08-4a12-9320-687d6f47ed4e","Type":"ContainerDied","Data":"e661833ee83ad6f141e043af449aa4d1477fde8be130663c70aa02cbed593871"} Nov 03 13:44:51.230970 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:51.230953 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-9add72-f74fc96d-xgkrk" event={"ID":"fa15c0e4-6d08-4a12-9320-687d6f47ed4e","Type":"ContainerDied","Data":"12d20e8ea37e025eeda488523b7f296ac845f25d12006da2be3f313b5dc1f545"} Nov 03 13:44:51.231066 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:51.231052 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-9add72-f74fc96d-xgkrk" Nov 03 13:44:51.236407 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:51.236334 5024 generic.go:356] "Generic (PLEG): container finished" podID="e8aba665-e4cc-4e67-8cdd-d09ef7542e15" containerID="448f466488f569306e0803889b69dcea09f90745995e4723f29cb7c4088320cb" exitCode=0 Nov 03 13:44:51.236407 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:51.236369 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-08111b/simulator-w1-08111b-kserve-7f65c5867-btf5b" event={"ID":"e8aba665-e4cc-4e67-8cdd-d09ef7542e15","Type":"ContainerDied","Data":"448f466488f569306e0803889b69dcea09f90745995e4723f29cb7c4088320cb"} Nov 03 13:44:51.236407 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:51.236392 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-08111b/simulator-w1-08111b-kserve-7f65c5867-btf5b" Nov 03 13:44:51.236593 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:51.236396 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-08111b/simulator-w1-08111b-kserve-7f65c5867-btf5b" event={"ID":"e8aba665-e4cc-4e67-8cdd-d09ef7542e15","Type":"ContainerDied","Data":"0fa375b76abce0e4f4c39c5b380958c0f5d6b2c8ef7519ae2dc9695bd5c37dd3"} Nov 03 13:44:51.242885 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:51.242851 5024 scope.go:122] "RemoveContainer" containerID="f260365e7cd791f7ee4cb0c4526ddb1047ed8821c14214cef7e33c1fae97190f" Nov 03 13:44:51.243260 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:44:51.243226 5024 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f260365e7cd791f7ee4cb0c4526ddb1047ed8821c14214cef7e33c1fae97190f\": container with ID starting with f260365e7cd791f7ee4cb0c4526ddb1047ed8821c14214cef7e33c1fae97190f not found: ID does not exist" containerID="f260365e7cd791f7ee4cb0c4526ddb1047ed8821c14214cef7e33c1fae97190f" Nov 03 13:44:51.243398 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:51.243276 5024 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f260365e7cd791f7ee4cb0c4526ddb1047ed8821c14214cef7e33c1fae97190f"} err="failed to get container status \"f260365e7cd791f7ee4cb0c4526ddb1047ed8821c14214cef7e33c1fae97190f\": rpc error: code = NotFound desc = could not find container \"f260365e7cd791f7ee4cb0c4526ddb1047ed8821c14214cef7e33c1fae97190f\": container with ID starting with f260365e7cd791f7ee4cb0c4526ddb1047ed8821c14214cef7e33c1fae97190f not found: ID does not exist" Nov 03 13:44:51.243398 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:51.243320 5024 scope.go:122] "RemoveContainer" containerID="e661833ee83ad6f141e043af449aa4d1477fde8be130663c70aa02cbed593871" Nov 03 13:44:51.255119 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:51.255096 5024 scope.go:122] "RemoveContainer" containerID="e661833ee83ad6f141e043af449aa4d1477fde8be130663c70aa02cbed593871" Nov 03 13:44:51.255426 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:44:51.255408 5024 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e661833ee83ad6f141e043af449aa4d1477fde8be130663c70aa02cbed593871\": container with ID starting with e661833ee83ad6f141e043af449aa4d1477fde8be130663c70aa02cbed593871 not found: ID does not exist" containerID="e661833ee83ad6f141e043af449aa4d1477fde8be130663c70aa02cbed593871" Nov 03 13:44:51.255504 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:51.255433 5024 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e661833ee83ad6f141e043af449aa4d1477fde8be130663c70aa02cbed593871"} err="failed to get container status \"e661833ee83ad6f141e043af449aa4d1477fde8be130663c70aa02cbed593871\": rpc error: code = NotFound desc = could not find container \"e661833ee83ad6f141e043af449aa4d1477fde8be130663c70aa02cbed593871\": container with ID starting with e661833ee83ad6f141e043af449aa4d1477fde8be130663c70aa02cbed593871 not found: ID does not exist" Nov 03 13:44:51.255504 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:51.255447 5024 scope.go:122] "RemoveContainer" containerID="448f466488f569306e0803889b69dcea09f90745995e4723f29cb7c4088320cb" Nov 03 13:44:51.267633 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:51.267602 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-08111b/premium-w1-08111b-kserve-57555d598-dx9g5"] Nov 03 13:44:51.268031 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:51.267714 5024 scope.go:122] "RemoveContainer" containerID="448f466488f569306e0803889b69dcea09f90745995e4723f29cb7c4088320cb" Nov 03 13:44:51.268093 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:44:51.268045 5024 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"448f466488f569306e0803889b69dcea09f90745995e4723f29cb7c4088320cb\": container with ID starting with 448f466488f569306e0803889b69dcea09f90745995e4723f29cb7c4088320cb not found: ID does not exist" containerID="448f466488f569306e0803889b69dcea09f90745995e4723f29cb7c4088320cb" Nov 03 13:44:51.268093 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:51.268074 5024 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"448f466488f569306e0803889b69dcea09f90745995e4723f29cb7c4088320cb"} err="failed to get container status \"448f466488f569306e0803889b69dcea09f90745995e4723f29cb7c4088320cb\": rpc error: code = NotFound desc = could not find container \"448f466488f569306e0803889b69dcea09f90745995e4723f29cb7c4088320cb\": container with ID starting with 448f466488f569306e0803889b69dcea09f90745995e4723f29cb7c4088320cb not found: ID does not exist" Nov 03 13:44:51.290338 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:51.281536 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-08111b/premium-w1-08111b-kserve-57555d598-dx9g5"] Nov 03 13:44:51.313944 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:51.313906 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-9add72-f74fc96d-xgkrk"] Nov 03 13:44:51.324897 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:51.324866 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-9add72-f74fc96d-xgkrk"] Nov 03 13:44:51.350542 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:51.350510 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-08111b/simulator-w1-08111b-kserve-7f65c5867-btf5b"] Nov 03 13:44:51.361350 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:51.361324 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-08111b/simulator-w1-08111b-kserve-7f65c5867-btf5b"] Nov 03 13:44:52.108117 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:52.108074 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e45c569a-243f-49ab-8d74-dd08cbb6127c" path="/var/lib/kubelet/pods/e45c569a-243f-49ab-8d74-dd08cbb6127c/volumes" Nov 03 13:44:52.108653 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:52.108628 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e8aba665-e4cc-4e67-8cdd-d09ef7542e15" path="/var/lib/kubelet/pods/e8aba665-e4cc-4e67-8cdd-d09ef7542e15/volumes" Nov 03 13:44:52.109104 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:52.109084 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fa15c0e4-6d08-4a12-9320-687d6f47ed4e" path="/var/lib/kubelet/pods/fa15c0e4-6d08-4a12-9320-687d6f47ed4e/volumes" Nov 03 13:44:55.267527 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:55.267483 5024 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-9add72-6b8fc4988f-kxlrl" podUID="a4f6951d-6317-4b83-b744-723ed7e576a4" containerName="payload-processing" containerID="cri-o://d05fa2e6f498a28aed9b557a946245a32b83b18384d5d8202984791b478ff736" gracePeriod=25 Nov 03 13:44:59.864681 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:59.864639 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-g94p9"] Nov 03 13:44:59.865109 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:59.864875 5024 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-g94p9" podUID="bef13398-28a1-4064-aa3b-ea0f3ebf64fd" containerName="istio-proxy" containerID="cri-o://2c088d3e468590c47b8bbc3776aa33503b0baa5b9f400d89e231f0781cac3e16" gracePeriod=30 Nov 03 13:44:59.880844 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:59.880817 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-4w96b"] Nov 03 13:44:59.881016 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:44:59.880995 5024 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-4w96b" podUID="cabfe6c9-07d8-420b-bb66-143190883f61" containerName="istio-proxy" containerID="cri-o://6662648feedad8d9f19220ad83effd7ebfb71a18ab245fed582074b8f870b602" gracePeriod=30 Nov 03 13:45:02.066675 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:02.066625 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-9add72-6b8fc4988f-kxlrl" podUID="a4f6951d-6317-4b83-b744-723ed7e576a4" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 13:45:05.167796 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:05.167762 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-g94p9" Nov 03 13:45:05.210054 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:05.210021 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-4w96b" Nov 03 13:45:05.294227 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:05.294189 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cabfe6c9-07d8-420b-bb66-143190883f61-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cabfe6c9-07d8-420b-bb66-143190883f61-workload-certs\") pod \"cabfe6c9-07d8-420b-bb66-143190883f61\" (UID: \"cabfe6c9-07d8-420b-bb66-143190883f61\") " Nov 03 13:45:05.294461 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:05.294249 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bef13398-28a1-4064-aa3b-ea0f3ebf64fd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bef13398-28a1-4064-aa3b-ea0f3ebf64fd-istiod-ca-cert\") pod \"bef13398-28a1-4064-aa3b-ea0f3ebf64fd\" (UID: \"bef13398-28a1-4064-aa3b-ea0f3ebf64fd\") " Nov 03 13:45:05.294461 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:05.294294 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bef13398-28a1-4064-aa3b-ea0f3ebf64fd-istio-token\" (UniqueName: \"kubernetes.io/projected/bef13398-28a1-4064-aa3b-ea0f3ebf64fd-istio-token\") pod \"bef13398-28a1-4064-aa3b-ea0f3ebf64fd\" (UID: \"bef13398-28a1-4064-aa3b-ea0f3ebf64fd\") " Nov 03 13:45:05.294461 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:05.294323 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bef13398-28a1-4064-aa3b-ea0f3ebf64fd-kube-api-access-8bbxg\" (UniqueName: \"kubernetes.io/projected/bef13398-28a1-4064-aa3b-ea0f3ebf64fd-kube-api-access-8bbxg\") pod \"bef13398-28a1-4064-aa3b-ea0f3ebf64fd\" (UID: \"bef13398-28a1-4064-aa3b-ea0f3ebf64fd\") " Nov 03 13:45:05.294461 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:05.294364 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cabfe6c9-07d8-420b-bb66-143190883f61-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cabfe6c9-07d8-420b-bb66-143190883f61-workload-socket\") pod \"cabfe6c9-07d8-420b-bb66-143190883f61\" (UID: \"cabfe6c9-07d8-420b-bb66-143190883f61\") " Nov 03 13:45:05.294461 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:05.294388 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cabfe6c9-07d8-420b-bb66-143190883f61-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cabfe6c9-07d8-420b-bb66-143190883f61-istio-envoy\") pod \"cabfe6c9-07d8-420b-bb66-143190883f61\" (UID: \"cabfe6c9-07d8-420b-bb66-143190883f61\") " Nov 03 13:45:05.294735 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:05.294471 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bef13398-28a1-4064-aa3b-ea0f3ebf64fd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bef13398-28a1-4064-aa3b-ea0f3ebf64fd-credential-socket\") pod \"bef13398-28a1-4064-aa3b-ea0f3ebf64fd\" (UID: \"bef13398-28a1-4064-aa3b-ea0f3ebf64fd\") " Nov 03 13:45:05.294735 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:05.294510 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bef13398-28a1-4064-aa3b-ea0f3ebf64fd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bef13398-28a1-4064-aa3b-ea0f3ebf64fd-istio-podinfo\") pod \"bef13398-28a1-4064-aa3b-ea0f3ebf64fd\" (UID: \"bef13398-28a1-4064-aa3b-ea0f3ebf64fd\") " Nov 03 13:45:05.294735 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:05.294563 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cabfe6c9-07d8-420b-bb66-143190883f61-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cabfe6c9-07d8-420b-bb66-143190883f61-credential-socket\") pod \"cabfe6c9-07d8-420b-bb66-143190883f61\" (UID: \"cabfe6c9-07d8-420b-bb66-143190883f61\") " Nov 03 13:45:05.294735 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:05.294632 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bef13398-28a1-4064-aa3b-ea0f3ebf64fd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bef13398-28a1-4064-aa3b-ea0f3ebf64fd-workload-socket\") pod \"bef13398-28a1-4064-aa3b-ea0f3ebf64fd\" (UID: \"bef13398-28a1-4064-aa3b-ea0f3ebf64fd\") " Nov 03 13:45:05.294735 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:05.294666 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bef13398-28a1-4064-aa3b-ea0f3ebf64fd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bef13398-28a1-4064-aa3b-ea0f3ebf64fd-workload-certs\") pod \"bef13398-28a1-4064-aa3b-ea0f3ebf64fd\" (UID: \"bef13398-28a1-4064-aa3b-ea0f3ebf64fd\") " Nov 03 13:45:05.294987 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:05.294752 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cabfe6c9-07d8-420b-bb66-143190883f61-kube-api-access-m55hj\" (UniqueName: \"kubernetes.io/projected/cabfe6c9-07d8-420b-bb66-143190883f61-kube-api-access-m55hj\") pod \"cabfe6c9-07d8-420b-bb66-143190883f61\" (UID: \"cabfe6c9-07d8-420b-bb66-143190883f61\") " Nov 03 13:45:05.295037 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:05.294626 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cabfe6c9-07d8-420b-bb66-143190883f61-workload-certs" pod "cabfe6c9-07d8-420b-bb66-143190883f61" (UID: "cabfe6c9-07d8-420b-bb66-143190883f61"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:45:05.295037 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:05.295020 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bef13398-28a1-4064-aa3b-ea0f3ebf64fd-workload-socket" pod "bef13398-28a1-4064-aa3b-ea0f3ebf64fd" (UID: "bef13398-28a1-4064-aa3b-ea0f3ebf64fd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:45:05.295139 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:05.294820 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cabfe6c9-07d8-420b-bb66-143190883f61-workload-socket" pod "cabfe6c9-07d8-420b-bb66-143190883f61" (UID: "cabfe6c9-07d8-420b-bb66-143190883f61"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:45:05.295139 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:05.295076 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cabfe6c9-07d8-420b-bb66-143190883f61-credential-socket" pod "cabfe6c9-07d8-420b-bb66-143190883f61" (UID: "cabfe6c9-07d8-420b-bb66-143190883f61"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:45:05.295139 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:05.294727 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bef13398-28a1-4064-aa3b-ea0f3ebf64fd-istiod-ca-cert" pod "bef13398-28a1-4064-aa3b-ea0f3ebf64fd" (UID: "bef13398-28a1-4064-aa3b-ea0f3ebf64fd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:45:05.295139 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:05.294835 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bef13398-28a1-4064-aa3b-ea0f3ebf64fd-credential-socket" pod "bef13398-28a1-4064-aa3b-ea0f3ebf64fd" (UID: "bef13398-28a1-4064-aa3b-ea0f3ebf64fd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:45:05.295417 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:05.295180 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bef13398-28a1-4064-aa3b-ea0f3ebf64fd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bef13398-28a1-4064-aa3b-ea0f3ebf64fd-istio-envoy\") pod \"bef13398-28a1-4064-aa3b-ea0f3ebf64fd\" (UID: \"bef13398-28a1-4064-aa3b-ea0f3ebf64fd\") " Nov 03 13:45:05.295417 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:05.295226 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bef13398-28a1-4064-aa3b-ea0f3ebf64fd-workload-certs" pod "bef13398-28a1-4064-aa3b-ea0f3ebf64fd" (UID: "bef13398-28a1-4064-aa3b-ea0f3ebf64fd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:45:05.295417 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:05.295235 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bef13398-28a1-4064-aa3b-ea0f3ebf64fd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bef13398-28a1-4064-aa3b-ea0f3ebf64fd-istio-data\") pod \"bef13398-28a1-4064-aa3b-ea0f3ebf64fd\" (UID: \"bef13398-28a1-4064-aa3b-ea0f3ebf64fd\") " Nov 03 13:45:05.295417 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:05.295307 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cabfe6c9-07d8-420b-bb66-143190883f61-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cabfe6c9-07d8-420b-bb66-143190883f61-istio-data\") pod \"cabfe6c9-07d8-420b-bb66-143190883f61\" (UID: \"cabfe6c9-07d8-420b-bb66-143190883f61\") " Nov 03 13:45:05.295624 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:05.295450 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bef13398-28a1-4064-aa3b-ea0f3ebf64fd-istio-data" pod "bef13398-28a1-4064-aa3b-ea0f3ebf64fd" (UID: "bef13398-28a1-4064-aa3b-ea0f3ebf64fd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:45:05.295624 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:05.295450 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cabfe6c9-07d8-420b-bb66-143190883f61-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cabfe6c9-07d8-420b-bb66-143190883f61-istio-podinfo\") pod \"cabfe6c9-07d8-420b-bb66-143190883f61\" (UID: \"cabfe6c9-07d8-420b-bb66-143190883f61\") " Nov 03 13:45:05.295624 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:05.295525 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cabfe6c9-07d8-420b-bb66-143190883f61-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cabfe6c9-07d8-420b-bb66-143190883f61-istiod-ca-cert\") pod \"cabfe6c9-07d8-420b-bb66-143190883f61\" (UID: \"cabfe6c9-07d8-420b-bb66-143190883f61\") " Nov 03 13:45:05.295624 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:05.295561 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cabfe6c9-07d8-420b-bb66-143190883f61-istio-token\" (UniqueName: \"kubernetes.io/projected/cabfe6c9-07d8-420b-bb66-143190883f61-istio-token\") pod \"cabfe6c9-07d8-420b-bb66-143190883f61\" (UID: \"cabfe6c9-07d8-420b-bb66-143190883f61\") " Nov 03 13:45:05.296035 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:05.296015 5024 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cabfe6c9-07d8-420b-bb66-143190883f61-credential-socket\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:45:05.296096 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:05.296042 5024 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bef13398-28a1-4064-aa3b-ea0f3ebf64fd-workload-socket\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:45:05.296096 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:05.296057 5024 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bef13398-28a1-4064-aa3b-ea0f3ebf64fd-workload-certs\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:45:05.296096 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:05.296072 5024 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bef13398-28a1-4064-aa3b-ea0f3ebf64fd-istio-data\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:45:05.296096 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:05.296085 5024 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cabfe6c9-07d8-420b-bb66-143190883f61-workload-certs\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:45:05.296337 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:05.296099 5024 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bef13398-28a1-4064-aa3b-ea0f3ebf64fd-istiod-ca-cert\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:45:05.296337 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:05.296115 5024 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cabfe6c9-07d8-420b-bb66-143190883f61-workload-socket\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:45:05.296337 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:05.296129 5024 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bef13398-28a1-4064-aa3b-ea0f3ebf64fd-credential-socket\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:45:05.297742 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:05.297468 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cabfe6c9-07d8-420b-bb66-143190883f61-istio-envoy" pod "cabfe6c9-07d8-420b-bb66-143190883f61" (UID: "cabfe6c9-07d8-420b-bb66-143190883f61"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:45:05.297742 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:05.297495 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cabfe6c9-07d8-420b-bb66-143190883f61-istiod-ca-cert" pod "cabfe6c9-07d8-420b-bb66-143190883f61" (UID: "cabfe6c9-07d8-420b-bb66-143190883f61"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:45:05.297742 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:05.297551 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bef13398-28a1-4064-aa3b-ea0f3ebf64fd-istio-podinfo" pod "bef13398-28a1-4064-aa3b-ea0f3ebf64fd" (UID: "bef13398-28a1-4064-aa3b-ea0f3ebf64fd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 13:45:05.297742 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:05.297709 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bef13398-28a1-4064-aa3b-ea0f3ebf64fd-kube-api-access-8bbxg" pod "bef13398-28a1-4064-aa3b-ea0f3ebf64fd" (UID: "bef13398-28a1-4064-aa3b-ea0f3ebf64fd"). InnerVolumeSpecName "kube-api-access-8bbxg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:45:05.298026 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:05.297747 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cabfe6c9-07d8-420b-bb66-143190883f61-istio-data" pod "cabfe6c9-07d8-420b-bb66-143190883f61" (UID: "cabfe6c9-07d8-420b-bb66-143190883f61"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:45:05.298026 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:05.297752 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cabfe6c9-07d8-420b-bb66-143190883f61-istio-podinfo" pod "cabfe6c9-07d8-420b-bb66-143190883f61" (UID: "cabfe6c9-07d8-420b-bb66-143190883f61"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 13:45:05.298670 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:05.298629 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bef13398-28a1-4064-aa3b-ea0f3ebf64fd-istio-token" pod "bef13398-28a1-4064-aa3b-ea0f3ebf64fd" (UID: "bef13398-28a1-4064-aa3b-ea0f3ebf64fd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:45:05.298786 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:05.298748 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cabfe6c9-07d8-420b-bb66-143190883f61-kube-api-access-m55hj" pod "cabfe6c9-07d8-420b-bb66-143190883f61" (UID: "cabfe6c9-07d8-420b-bb66-143190883f61"). InnerVolumeSpecName "kube-api-access-m55hj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:45:05.299058 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:05.299035 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bef13398-28a1-4064-aa3b-ea0f3ebf64fd-istio-envoy" pod "bef13398-28a1-4064-aa3b-ea0f3ebf64fd" (UID: "bef13398-28a1-4064-aa3b-ea0f3ebf64fd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:45:05.299099 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:05.299038 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cabfe6c9-07d8-420b-bb66-143190883f61-istio-token" pod "cabfe6c9-07d8-420b-bb66-143190883f61" (UID: "cabfe6c9-07d8-420b-bb66-143190883f61"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:45:05.320691 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:05.320658 5024 generic.go:356] "Generic (PLEG): container finished" podID="cabfe6c9-07d8-420b-bb66-143190883f61" containerID="6662648feedad8d9f19220ad83effd7ebfb71a18ab245fed582074b8f870b602" exitCode=0 Nov 03 13:45:05.320851 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:05.320704 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-4w96b" Nov 03 13:45:05.320851 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:05.320738 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-4w96b" event={"ID":"cabfe6c9-07d8-420b-bb66-143190883f61","Type":"ContainerDied","Data":"6662648feedad8d9f19220ad83effd7ebfb71a18ab245fed582074b8f870b602"} Nov 03 13:45:05.320851 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:05.320793 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-4w96b" event={"ID":"cabfe6c9-07d8-420b-bb66-143190883f61","Type":"ContainerDied","Data":"1d053ded65a95f1497d6165338d1797b8d3d4bff13cccac6f0577faa9794a631"} Nov 03 13:45:05.320851 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:05.320825 5024 scope.go:122] "RemoveContainer" containerID="6662648feedad8d9f19220ad83effd7ebfb71a18ab245fed582074b8f870b602" Nov 03 13:45:05.322325 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:05.322279 5024 generic.go:356] "Generic (PLEG): container finished" podID="bef13398-28a1-4064-aa3b-ea0f3ebf64fd" containerID="2c088d3e468590c47b8bbc3776aa33503b0baa5b9f400d89e231f0781cac3e16" exitCode=0 Nov 03 13:45:05.322456 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:05.322340 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-g94p9" Nov 03 13:45:05.322456 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:05.322328 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-g94p9" event={"ID":"bef13398-28a1-4064-aa3b-ea0f3ebf64fd","Type":"ContainerDied","Data":"2c088d3e468590c47b8bbc3776aa33503b0baa5b9f400d89e231f0781cac3e16"} Nov 03 13:45:05.322456 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:05.322403 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-g94p9" event={"ID":"bef13398-28a1-4064-aa3b-ea0f3ebf64fd","Type":"ContainerDied","Data":"dcf091dd73cb6ec98ee2ad274d97a9464cf67046727af464ad6413f53ce9bf78"} Nov 03 13:45:05.330755 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:05.330736 5024 scope.go:122] "RemoveContainer" containerID="6662648feedad8d9f19220ad83effd7ebfb71a18ab245fed582074b8f870b602" Nov 03 13:45:05.331060 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:45:05.331033 5024 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6662648feedad8d9f19220ad83effd7ebfb71a18ab245fed582074b8f870b602\": container with ID starting with 6662648feedad8d9f19220ad83effd7ebfb71a18ab245fed582074b8f870b602 not found: ID does not exist" containerID="6662648feedad8d9f19220ad83effd7ebfb71a18ab245fed582074b8f870b602" Nov 03 13:45:05.331159 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:05.331068 5024 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6662648feedad8d9f19220ad83effd7ebfb71a18ab245fed582074b8f870b602"} err="failed to get container status \"6662648feedad8d9f19220ad83effd7ebfb71a18ab245fed582074b8f870b602\": rpc error: code = NotFound desc = could not find container \"6662648feedad8d9f19220ad83effd7ebfb71a18ab245fed582074b8f870b602\": container with ID starting with 6662648feedad8d9f19220ad83effd7ebfb71a18ab245fed582074b8f870b602 not found: ID does not exist" Nov 03 13:45:05.331159 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:05.331091 5024 scope.go:122] "RemoveContainer" containerID="2c088d3e468590c47b8bbc3776aa33503b0baa5b9f400d89e231f0781cac3e16" Nov 03 13:45:05.339913 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:05.339896 5024 scope.go:122] "RemoveContainer" containerID="2c088d3e468590c47b8bbc3776aa33503b0baa5b9f400d89e231f0781cac3e16" Nov 03 13:45:05.340213 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:45:05.340192 5024 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c088d3e468590c47b8bbc3776aa33503b0baa5b9f400d89e231f0781cac3e16\": container with ID starting with 2c088d3e468590c47b8bbc3776aa33503b0baa5b9f400d89e231f0781cac3e16 not found: ID does not exist" containerID="2c088d3e468590c47b8bbc3776aa33503b0baa5b9f400d89e231f0781cac3e16" Nov 03 13:45:05.340276 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:05.340221 5024 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c088d3e468590c47b8bbc3776aa33503b0baa5b9f400d89e231f0781cac3e16"} err="failed to get container status \"2c088d3e468590c47b8bbc3776aa33503b0baa5b9f400d89e231f0781cac3e16\": rpc error: code = NotFound desc = could not find container \"2c088d3e468590c47b8bbc3776aa33503b0baa5b9f400d89e231f0781cac3e16\": container with ID starting with 2c088d3e468590c47b8bbc3776aa33503b0baa5b9f400d89e231f0781cac3e16 not found: ID does not exist" Nov 03 13:45:05.350991 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:05.350966 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-g94p9"] Nov 03 13:45:05.355945 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:05.355921 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-g94p9"] Nov 03 13:45:05.376962 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:05.376933 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-4w96b"] Nov 03 13:45:05.386028 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:05.386000 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-9add72-openshift-default-c7769bc6c-4w96b"] Nov 03 13:45:05.396877 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:05.396813 5024 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cabfe6c9-07d8-420b-bb66-143190883f61-istio-podinfo\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:45:05.396877 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:05.396838 5024 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cabfe6c9-07d8-420b-bb66-143190883f61-istiod-ca-cert\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:45:05.396877 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:05.396850 5024 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cabfe6c9-07d8-420b-bb66-143190883f61-istio-token\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:45:05.396877 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:05.396859 5024 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bef13398-28a1-4064-aa3b-ea0f3ebf64fd-istio-token\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:45:05.396877 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:05.396868 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8bbxg\" (UniqueName: \"kubernetes.io/projected/bef13398-28a1-4064-aa3b-ea0f3ebf64fd-kube-api-access-8bbxg\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:45:05.396877 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:05.396878 5024 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cabfe6c9-07d8-420b-bb66-143190883f61-istio-envoy\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:45:05.397154 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:05.396886 5024 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bef13398-28a1-4064-aa3b-ea0f3ebf64fd-istio-podinfo\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:45:05.397154 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:05.396895 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m55hj\" (UniqueName: \"kubernetes.io/projected/cabfe6c9-07d8-420b-bb66-143190883f61-kube-api-access-m55hj\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:45:05.397154 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:05.396904 5024 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bef13398-28a1-4064-aa3b-ea0f3ebf64fd-istio-envoy\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:45:05.397154 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:05.396912 5024 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cabfe6c9-07d8-420b-bb66-143190883f61-istio-data\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:45:06.109307 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:06.109250 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bef13398-28a1-4064-aa3b-ea0f3ebf64fd" path="/var/lib/kubelet/pods/bef13398-28a1-4064-aa3b-ea0f3ebf64fd/volumes" Nov 03 13:45:06.109941 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:06.109920 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cabfe6c9-07d8-420b-bb66-143190883f61" path="/var/lib/kubelet/pods/cabfe6c9-07d8-420b-bb66-143190883f61/volumes" Nov 03 13:45:09.508748 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:45:09.508648 5024 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d7mwz" containerName="extractor" Nov 03 13:45:12.066690 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:12.066653 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-9add72-6b8fc4988f-kxlrl" podUID="a4f6951d-6317-4b83-b744-723ed7e576a4" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 13:45:14.386498 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:14.386448 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-08111b-openshift-default-5b6dd8787c-v2hw4"] Nov 03 13:45:14.386949 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:14.386734 5024 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-08111b-openshift-default-5b6dd8787c-v2hw4" podUID="ca014f3a-ff92-4bb9-9a7a-8c8d3dabe486" containerName="istio-proxy" containerID="cri-o://1c031b5792bffc146b78656c47e3d1e7873d61a5ce2371294eaf19c77a57a974" gracePeriod=30 Nov 03 13:45:15.401215 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:15.401179 5024 generic.go:356] "Generic (PLEG): container finished" podID="a4f6951d-6317-4b83-b744-723ed7e576a4" containerID="d05fa2e6f498a28aed9b557a946245a32b83b18384d5d8202984791b478ff736" exitCode=0 Nov 03 13:45:15.401626 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:15.401257 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-9add72-6b8fc4988f-kxlrl" event={"ID":"a4f6951d-6317-4b83-b744-723ed7e576a4","Type":"ContainerDied","Data":"d05fa2e6f498a28aed9b557a946245a32b83b18384d5d8202984791b478ff736"} Nov 03 13:45:15.543732 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:15.543702 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-9add72-6b8fc4988f-kxlrl" Nov 03 13:45:15.724614 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:15.724577 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4f6951d-6317-4b83-b744-723ed7e576a4-kube-api-access-jrf25\" (UniqueName: \"kubernetes.io/projected/a4f6951d-6317-4b83-b744-723ed7e576a4-kube-api-access-jrf25\") pod \"a4f6951d-6317-4b83-b744-723ed7e576a4\" (UID: \"a4f6951d-6317-4b83-b744-723ed7e576a4\") " Nov 03 13:45:15.724775 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:15.724752 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a4f6951d-6317-4b83-b744-723ed7e576a4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4f6951d-6317-4b83-b744-723ed7e576a4-plugins-config-volume\") pod \"a4f6951d-6317-4b83-b744-723ed7e576a4\" (UID: \"a4f6951d-6317-4b83-b744-723ed7e576a4\") " Nov 03 13:45:15.725186 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:15.725158 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4f6951d-6317-4b83-b744-723ed7e576a4-plugins-config-volume" pod "a4f6951d-6317-4b83-b744-723ed7e576a4" (UID: "a4f6951d-6317-4b83-b744-723ed7e576a4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:45:15.726805 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:15.726774 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4f6951d-6317-4b83-b744-723ed7e576a4-kube-api-access-jrf25" pod "a4f6951d-6317-4b83-b744-723ed7e576a4" (UID: "a4f6951d-6317-4b83-b744-723ed7e576a4"). InnerVolumeSpecName "kube-api-access-jrf25". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:45:15.826691 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:15.826603 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jrf25\" (UniqueName: \"kubernetes.io/projected/a4f6951d-6317-4b83-b744-723ed7e576a4-kube-api-access-jrf25\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:45:15.826691 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:15.826638 5024 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4f6951d-6317-4b83-b744-723ed7e576a4-plugins-config-volume\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:45:16.408886 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:16.408787 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-9add72-6b8fc4988f-kxlrl" Nov 03 13:45:16.408886 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:16.408810 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-9add72-6b8fc4988f-kxlrl" event={"ID":"a4f6951d-6317-4b83-b744-723ed7e576a4","Type":"ContainerDied","Data":"340a0ae6096a46948e378716e173176790aff81d3a68f364023e2daea1f65e77"} Nov 03 13:45:16.408886 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:16.408874 5024 scope.go:122] "RemoveContainer" containerID="d05fa2e6f498a28aed9b557a946245a32b83b18384d5d8202984791b478ff736" Nov 03 13:45:16.434143 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:16.434104 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-9add72-6b8fc4988f-kxlrl"] Nov 03 13:45:16.436214 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:16.436188 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-9add72-6b8fc4988f-kxlrl"] Nov 03 13:45:18.111545 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:18.111501 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a4f6951d-6317-4b83-b744-723ed7e576a4" path="/var/lib/kubelet/pods/a4f6951d-6317-4b83-b744-723ed7e576a4/volumes" Nov 03 13:45:19.660930 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:19.660863 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-08111b-openshift-default-5b6dd8787c-v2hw4" Nov 03 13:45:19.777084 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:19.777031 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca014f3a-ff92-4bb9-9a7a-8c8d3dabe486-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca014f3a-ff92-4bb9-9a7a-8c8d3dabe486-istio-data\") pod \"ca014f3a-ff92-4bb9-9a7a-8c8d3dabe486\" (UID: \"ca014f3a-ff92-4bb9-9a7a-8c8d3dabe486\") " Nov 03 13:45:19.777414 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:19.777388 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca014f3a-ff92-4bb9-9a7a-8c8d3dabe486-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca014f3a-ff92-4bb9-9a7a-8c8d3dabe486-istio-envoy\") pod \"ca014f3a-ff92-4bb9-9a7a-8c8d3dabe486\" (UID: \"ca014f3a-ff92-4bb9-9a7a-8c8d3dabe486\") " Nov 03 13:45:19.777947 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:19.777924 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca014f3a-ff92-4bb9-9a7a-8c8d3dabe486-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca014f3a-ff92-4bb9-9a7a-8c8d3dabe486-workload-socket\") pod \"ca014f3a-ff92-4bb9-9a7a-8c8d3dabe486\" (UID: \"ca014f3a-ff92-4bb9-9a7a-8c8d3dabe486\") " Nov 03 13:45:19.778156 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:19.778138 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca014f3a-ff92-4bb9-9a7a-8c8d3dabe486-kube-api-access-q8kdt\" (UniqueName: \"kubernetes.io/projected/ca014f3a-ff92-4bb9-9a7a-8c8d3dabe486-kube-api-access-q8kdt\") pod \"ca014f3a-ff92-4bb9-9a7a-8c8d3dabe486\" (UID: \"ca014f3a-ff92-4bb9-9a7a-8c8d3dabe486\") " Nov 03 13:45:19.778384 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:19.777557 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca014f3a-ff92-4bb9-9a7a-8c8d3dabe486-istio-data" pod "ca014f3a-ff92-4bb9-9a7a-8c8d3dabe486" (UID: "ca014f3a-ff92-4bb9-9a7a-8c8d3dabe486"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:45:19.778476 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:19.778357 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca014f3a-ff92-4bb9-9a7a-8c8d3dabe486-workload-socket" pod "ca014f3a-ff92-4bb9-9a7a-8c8d3dabe486" (UID: "ca014f3a-ff92-4bb9-9a7a-8c8d3dabe486"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:45:19.778476 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:19.778364 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca014f3a-ff92-4bb9-9a7a-8c8d3dabe486-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca014f3a-ff92-4bb9-9a7a-8c8d3dabe486-credential-socket\") pod \"ca014f3a-ff92-4bb9-9a7a-8c8d3dabe486\" (UID: \"ca014f3a-ff92-4bb9-9a7a-8c8d3dabe486\") " Nov 03 13:45:19.778626 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:19.778473 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ca014f3a-ff92-4bb9-9a7a-8c8d3dabe486-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca014f3a-ff92-4bb9-9a7a-8c8d3dabe486-istiod-ca-cert\") pod \"ca014f3a-ff92-4bb9-9a7a-8c8d3dabe486\" (UID: \"ca014f3a-ff92-4bb9-9a7a-8c8d3dabe486\") " Nov 03 13:45:19.778626 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:19.778558 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca014f3a-ff92-4bb9-9a7a-8c8d3dabe486-istio-token\" (UniqueName: \"kubernetes.io/projected/ca014f3a-ff92-4bb9-9a7a-8c8d3dabe486-istio-token\") pod \"ca014f3a-ff92-4bb9-9a7a-8c8d3dabe486\" (UID: \"ca014f3a-ff92-4bb9-9a7a-8c8d3dabe486\") " Nov 03 13:45:19.778626 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:19.778588 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ca014f3a-ff92-4bb9-9a7a-8c8d3dabe486-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca014f3a-ff92-4bb9-9a7a-8c8d3dabe486-istio-podinfo\") pod \"ca014f3a-ff92-4bb9-9a7a-8c8d3dabe486\" (UID: \"ca014f3a-ff92-4bb9-9a7a-8c8d3dabe486\") " Nov 03 13:45:19.778794 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:19.778646 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca014f3a-ff92-4bb9-9a7a-8c8d3dabe486-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca014f3a-ff92-4bb9-9a7a-8c8d3dabe486-workload-certs\") pod \"ca014f3a-ff92-4bb9-9a7a-8c8d3dabe486\" (UID: \"ca014f3a-ff92-4bb9-9a7a-8c8d3dabe486\") " Nov 03 13:45:19.779073 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:19.779051 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca014f3a-ff92-4bb9-9a7a-8c8d3dabe486-credential-socket" pod "ca014f3a-ff92-4bb9-9a7a-8c8d3dabe486" (UID: "ca014f3a-ff92-4bb9-9a7a-8c8d3dabe486"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:45:19.779360 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:19.779337 5024 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca014f3a-ff92-4bb9-9a7a-8c8d3dabe486-istio-data\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:45:19.779465 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:19.779364 5024 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca014f3a-ff92-4bb9-9a7a-8c8d3dabe486-workload-socket\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:45:19.779601 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:19.779578 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca014f3a-ff92-4bb9-9a7a-8c8d3dabe486-workload-certs" pod "ca014f3a-ff92-4bb9-9a7a-8c8d3dabe486" (UID: "ca014f3a-ff92-4bb9-9a7a-8c8d3dabe486"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:45:19.779907 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:19.779887 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca014f3a-ff92-4bb9-9a7a-8c8d3dabe486-istiod-ca-cert" pod "ca014f3a-ff92-4bb9-9a7a-8c8d3dabe486" (UID: "ca014f3a-ff92-4bb9-9a7a-8c8d3dabe486"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:45:19.782961 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:19.782705 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca014f3a-ff92-4bb9-9a7a-8c8d3dabe486-istio-envoy" pod "ca014f3a-ff92-4bb9-9a7a-8c8d3dabe486" (UID: "ca014f3a-ff92-4bb9-9a7a-8c8d3dabe486"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:45:19.783277 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:19.783224 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca014f3a-ff92-4bb9-9a7a-8c8d3dabe486-kube-api-access-q8kdt" pod "ca014f3a-ff92-4bb9-9a7a-8c8d3dabe486" (UID: "ca014f3a-ff92-4bb9-9a7a-8c8d3dabe486"). InnerVolumeSpecName "kube-api-access-q8kdt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:45:19.788151 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:19.788126 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ca014f3a-ff92-4bb9-9a7a-8c8d3dabe486-istio-podinfo" pod "ca014f3a-ff92-4bb9-9a7a-8c8d3dabe486" (UID: "ca014f3a-ff92-4bb9-9a7a-8c8d3dabe486"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 13:45:19.790331 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:19.790253 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca014f3a-ff92-4bb9-9a7a-8c8d3dabe486-istio-token" pod "ca014f3a-ff92-4bb9-9a7a-8c8d3dabe486" (UID: "ca014f3a-ff92-4bb9-9a7a-8c8d3dabe486"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:45:19.880427 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:19.880389 5024 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca014f3a-ff92-4bb9-9a7a-8c8d3dabe486-credential-socket\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:45:19.880427 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:19.880424 5024 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca014f3a-ff92-4bb9-9a7a-8c8d3dabe486-istiod-ca-cert\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:45:19.880630 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:19.880440 5024 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca014f3a-ff92-4bb9-9a7a-8c8d3dabe486-istio-token\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:45:19.880630 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:19.880452 5024 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca014f3a-ff92-4bb9-9a7a-8c8d3dabe486-istio-podinfo\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:45:19.880630 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:19.880465 5024 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca014f3a-ff92-4bb9-9a7a-8c8d3dabe486-workload-certs\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:45:19.880630 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:19.880476 5024 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca014f3a-ff92-4bb9-9a7a-8c8d3dabe486-istio-envoy\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:45:19.880630 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:19.880488 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q8kdt\" (UniqueName: \"kubernetes.io/projected/ca014f3a-ff92-4bb9-9a7a-8c8d3dabe486-kube-api-access-q8kdt\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:45:20.442063 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:20.441978 5024 generic.go:356] "Generic (PLEG): container finished" podID="ca014f3a-ff92-4bb9-9a7a-8c8d3dabe486" containerID="1c031b5792bffc146b78656c47e3d1e7873d61a5ce2371294eaf19c77a57a974" exitCode=0 Nov 03 13:45:20.442208 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:20.442055 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-08111b-openshift-default-5b6dd8787c-v2hw4" event={"ID":"ca014f3a-ff92-4bb9-9a7a-8c8d3dabe486","Type":"ContainerDied","Data":"1c031b5792bffc146b78656c47e3d1e7873d61a5ce2371294eaf19c77a57a974"} Nov 03 13:45:20.442208 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:20.442069 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-08111b-openshift-default-5b6dd8787c-v2hw4" Nov 03 13:45:20.442208 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:20.442145 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-08111b-openshift-default-5b6dd8787c-v2hw4" event={"ID":"ca014f3a-ff92-4bb9-9a7a-8c8d3dabe486","Type":"ContainerDied","Data":"975c0ba90f891e0704532a6fc22c9f2065279f7e614128867cbdf569902563d1"} Nov 03 13:45:20.442208 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:20.442168 5024 scope.go:122] "RemoveContainer" containerID="1c031b5792bffc146b78656c47e3d1e7873d61a5ce2371294eaf19c77a57a974" Nov 03 13:45:20.451807 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:20.451789 5024 scope.go:122] "RemoveContainer" containerID="5a0d74c33d3dea8b8445327a18b596aa922754e42b80e3915e17f7f73bbb73b0" Nov 03 13:45:20.463684 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:20.463666 5024 scope.go:122] "RemoveContainer" containerID="1c031b5792bffc146b78656c47e3d1e7873d61a5ce2371294eaf19c77a57a974" Nov 03 13:45:20.464032 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:45:20.464002 5024 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c031b5792bffc146b78656c47e3d1e7873d61a5ce2371294eaf19c77a57a974\": container with ID starting with 1c031b5792bffc146b78656c47e3d1e7873d61a5ce2371294eaf19c77a57a974 not found: ID does not exist" containerID="1c031b5792bffc146b78656c47e3d1e7873d61a5ce2371294eaf19c77a57a974" Nov 03 13:45:20.464109 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:20.464041 5024 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c031b5792bffc146b78656c47e3d1e7873d61a5ce2371294eaf19c77a57a974"} err="failed to get container status \"1c031b5792bffc146b78656c47e3d1e7873d61a5ce2371294eaf19c77a57a974\": rpc error: code = NotFound desc = could not find container \"1c031b5792bffc146b78656c47e3d1e7873d61a5ce2371294eaf19c77a57a974\": container with ID starting with 1c031b5792bffc146b78656c47e3d1e7873d61a5ce2371294eaf19c77a57a974 not found: ID does not exist" Nov 03 13:45:20.464109 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:20.464058 5024 scope.go:122] "RemoveContainer" containerID="5a0d74c33d3dea8b8445327a18b596aa922754e42b80e3915e17f7f73bbb73b0" Nov 03 13:45:20.464352 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:45:20.464330 5024 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a0d74c33d3dea8b8445327a18b596aa922754e42b80e3915e17f7f73bbb73b0\": container with ID starting with 5a0d74c33d3dea8b8445327a18b596aa922754e42b80e3915e17f7f73bbb73b0 not found: ID does not exist" containerID="5a0d74c33d3dea8b8445327a18b596aa922754e42b80e3915e17f7f73bbb73b0" Nov 03 13:45:20.464434 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:20.464363 5024 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a0d74c33d3dea8b8445327a18b596aa922754e42b80e3915e17f7f73bbb73b0"} err="failed to get container status \"5a0d74c33d3dea8b8445327a18b596aa922754e42b80e3915e17f7f73bbb73b0\": rpc error: code = NotFound desc = could not find container \"5a0d74c33d3dea8b8445327a18b596aa922754e42b80e3915e17f7f73bbb73b0\": container with ID starting with 5a0d74c33d3dea8b8445327a18b596aa922754e42b80e3915e17f7f73bbb73b0 not found: ID does not exist" Nov 03 13:45:20.471316 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:20.471271 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-08111b-openshift-default-5b6dd8787c-v2hw4"] Nov 03 13:45:20.475492 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:20.475470 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-08111b-openshift-default-5b6dd8787c-v2hw4"] Nov 03 13:45:22.108667 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:22.108635 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ca014f3a-ff92-4bb9-9a7a-8c8d3dabe486" path="/var/lib/kubelet/pods/ca014f3a-ff92-4bb9-9a7a-8c8d3dabe486/volumes" Nov 03 13:45:23.168646 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:23.168609 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-a4ab8e-68655794c-7c8x7"] Nov 03 13:45:28.169644 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:28.169607 5024 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-a4ab8e-68655794c-7c8x7" podUID="7b571fd7-340c-455e-be5a-479ed17d7517" containerName="payload-pre-processing" containerID="cri-o://bd7484986d1d81caa8d9b7003da562b709053386e05f9d2a97fd6b84b8832e76" gracePeriod=25 Nov 03 13:45:30.697772 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:30.697727 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-ded926-5f5bffd8fd-kn5lm"] Nov 03 13:45:30.698327 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:30.698070 5024 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ded926-5f5bffd8fd-kn5lm" podUID="5e1668ec-1b0f-4600-8935-64733905b0c6" containerName="maas-api" containerID="cri-o://ad0fc84950464f28185e109ed9e9ced1c4c95401ec78ed523a1c04b4ca92163c" gracePeriod=30 Nov 03 13:45:30.974448 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:30.974415 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ded926-5f5bffd8fd-kn5lm" Nov 03 13:45:31.104600 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:31.104562 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e1668ec-1b0f-4600-8935-64733905b0c6-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e1668ec-1b0f-4600-8935-64733905b0c6-openshift-service-ca\") pod \"5e1668ec-1b0f-4600-8935-64733905b0c6\" (UID: \"5e1668ec-1b0f-4600-8935-64733905b0c6\") " Nov 03 13:45:31.104771 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:31.104616 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5e1668ec-1b0f-4600-8935-64733905b0c6-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5e1668ec-1b0f-4600-8935-64733905b0c6-metrics-certs\") pod \"5e1668ec-1b0f-4600-8935-64733905b0c6\" (UID: \"5e1668ec-1b0f-4600-8935-64733905b0c6\") " Nov 03 13:45:31.104771 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:31.104704 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5e1668ec-1b0f-4600-8935-64733905b0c6-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5e1668ec-1b0f-4600-8935-64733905b0c6-maas-api-tls\") pod \"5e1668ec-1b0f-4600-8935-64733905b0c6\" (UID: \"5e1668ec-1b0f-4600-8935-64733905b0c6\") " Nov 03 13:45:31.104845 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:31.104778 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e1668ec-1b0f-4600-8935-64733905b0c6-kube-api-access-rmcqt\" (UniqueName: \"kubernetes.io/projected/5e1668ec-1b0f-4600-8935-64733905b0c6-kube-api-access-rmcqt\") pod \"5e1668ec-1b0f-4600-8935-64733905b0c6\" (UID: \"5e1668ec-1b0f-4600-8935-64733905b0c6\") " Nov 03 13:45:31.105003 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:31.104975 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e1668ec-1b0f-4600-8935-64733905b0c6-openshift-service-ca" pod "5e1668ec-1b0f-4600-8935-64733905b0c6" (UID: "5e1668ec-1b0f-4600-8935-64733905b0c6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:45:31.105185 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:31.105166 5024 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e1668ec-1b0f-4600-8935-64733905b0c6-openshift-service-ca\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:45:31.106819 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:31.106792 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e1668ec-1b0f-4600-8935-64733905b0c6-maas-api-tls" pod "5e1668ec-1b0f-4600-8935-64733905b0c6" (UID: "5e1668ec-1b0f-4600-8935-64733905b0c6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 13:45:31.106928 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:31.106915 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e1668ec-1b0f-4600-8935-64733905b0c6-kube-api-access-rmcqt" pod "5e1668ec-1b0f-4600-8935-64733905b0c6" (UID: "5e1668ec-1b0f-4600-8935-64733905b0c6"). InnerVolumeSpecName "kube-api-access-rmcqt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:45:31.107012 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:31.106989 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e1668ec-1b0f-4600-8935-64733905b0c6-metrics-certs" pod "5e1668ec-1b0f-4600-8935-64733905b0c6" (UID: "5e1668ec-1b0f-4600-8935-64733905b0c6"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 13:45:31.206192 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:31.206108 5024 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5e1668ec-1b0f-4600-8935-64733905b0c6-maas-api-tls\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:45:31.206192 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:31.206141 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rmcqt\" (UniqueName: \"kubernetes.io/projected/5e1668ec-1b0f-4600-8935-64733905b0c6-kube-api-access-rmcqt\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:45:31.206192 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:31.206151 5024 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5e1668ec-1b0f-4600-8935-64733905b0c6-metrics-certs\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:45:31.521669 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:31.521577 5024 generic.go:356] "Generic (PLEG): container finished" podID="5e1668ec-1b0f-4600-8935-64733905b0c6" containerID="ad0fc84950464f28185e109ed9e9ced1c4c95401ec78ed523a1c04b4ca92163c" exitCode=0 Nov 03 13:45:31.521669 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:31.521646 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ded926-5f5bffd8fd-kn5lm" Nov 03 13:45:31.521908 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:31.521663 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ded926-5f5bffd8fd-kn5lm" event={"ID":"5e1668ec-1b0f-4600-8935-64733905b0c6","Type":"ContainerDied","Data":"ad0fc84950464f28185e109ed9e9ced1c4c95401ec78ed523a1c04b4ca92163c"} Nov 03 13:45:31.521908 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:31.521723 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ded926-5f5bffd8fd-kn5lm" event={"ID":"5e1668ec-1b0f-4600-8935-64733905b0c6","Type":"ContainerDied","Data":"1eb9a4ee7648e7b2fae5f47bd1dc6f831b42aeba9929d3a8703371c2459c5a08"} Nov 03 13:45:31.521908 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:31.521738 5024 scope.go:122] "RemoveContainer" containerID="ad0fc84950464f28185e109ed9e9ced1c4c95401ec78ed523a1c04b4ca92163c" Nov 03 13:45:31.531436 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:31.531416 5024 scope.go:122] "RemoveContainer" containerID="ad0fc84950464f28185e109ed9e9ced1c4c95401ec78ed523a1c04b4ca92163c" Nov 03 13:45:31.531759 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:45:31.531716 5024 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad0fc84950464f28185e109ed9e9ced1c4c95401ec78ed523a1c04b4ca92163c\": container with ID starting with ad0fc84950464f28185e109ed9e9ced1c4c95401ec78ed523a1c04b4ca92163c not found: ID does not exist" containerID="ad0fc84950464f28185e109ed9e9ced1c4c95401ec78ed523a1c04b4ca92163c" Nov 03 13:45:31.531866 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:31.531768 5024 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad0fc84950464f28185e109ed9e9ced1c4c95401ec78ed523a1c04b4ca92163c"} err="failed to get container status \"ad0fc84950464f28185e109ed9e9ced1c4c95401ec78ed523a1c04b4ca92163c\": rpc error: code = NotFound desc = could not find container \"ad0fc84950464f28185e109ed9e9ced1c4c95401ec78ed523a1c04b4ca92163c\": container with ID starting with ad0fc84950464f28185e109ed9e9ced1c4c95401ec78ed523a1c04b4ca92163c not found: ID does not exist" Nov 03 13:45:31.551771 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:31.551742 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-ded926-5f5bffd8fd-kn5lm"] Nov 03 13:45:31.561931 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:31.561897 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-ded926-5f5bffd8fd-kn5lm"] Nov 03 13:45:32.109481 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:32.109444 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e1668ec-1b0f-4600-8935-64733905b0c6" path="/var/lib/kubelet/pods/5e1668ec-1b0f-4600-8935-64733905b0c6/volumes" Nov 03 13:45:33.938800 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:33.938746 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ded926-5f5bffd8fd-kn5lm" podUID="5e1668ec-1b0f-4600-8935-64733905b0c6" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.66:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 13:45:35.824106 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:35.824066 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-a4ab8e-68655794c-7c8x7" podUID="7b571fd7-340c-455e-be5a-479ed17d7517" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 13:45:39.011926 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:39.011874 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-a4ab8e-b94b5469d-kwm6h"] Nov 03 13:45:39.012435 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:39.012094 5024 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a4ab8e-b94b5469d-kwm6h" podUID="bb5f109b-305d-4eec-a3c9-b917a0bf9981" containerName="maas-api" containerID="cri-o://90f475e3677a120a451ad1e577fe36e8f65a53b684a3d3e344a1dc3e747236cf" gracePeriod=30 Nov 03 13:45:39.303035 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:39.303004 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a4ab8e-b94b5469d-kwm6h" Nov 03 13:45:39.396019 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:39.395980 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb5f109b-305d-4eec-a3c9-b917a0bf9981-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb5f109b-305d-4eec-a3c9-b917a0bf9981-openshift-service-ca\") pod \"bb5f109b-305d-4eec-a3c9-b917a0bf9981\" (UID: \"bb5f109b-305d-4eec-a3c9-b917a0bf9981\") " Nov 03 13:45:39.396216 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:39.396145 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb5f109b-305d-4eec-a3c9-b917a0bf9981-kube-api-access-8pvpq\" (UniqueName: \"kubernetes.io/projected/bb5f109b-305d-4eec-a3c9-b917a0bf9981-kube-api-access-8pvpq\") pod \"bb5f109b-305d-4eec-a3c9-b917a0bf9981\" (UID: \"bb5f109b-305d-4eec-a3c9-b917a0bf9981\") " Nov 03 13:45:39.396326 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:39.396224 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bb5f109b-305d-4eec-a3c9-b917a0bf9981-metrics-certs\" (UniqueName: \"kubernetes.io/secret/bb5f109b-305d-4eec-a3c9-b917a0bf9981-metrics-certs\") pod \"bb5f109b-305d-4eec-a3c9-b917a0bf9981\" (UID: \"bb5f109b-305d-4eec-a3c9-b917a0bf9981\") " Nov 03 13:45:39.396326 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:39.396306 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bb5f109b-305d-4eec-a3c9-b917a0bf9981-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bb5f109b-305d-4eec-a3c9-b917a0bf9981-maas-api-tls\") pod \"bb5f109b-305d-4eec-a3c9-b917a0bf9981\" (UID: \"bb5f109b-305d-4eec-a3c9-b917a0bf9981\") " Nov 03 13:45:39.396431 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:39.396383 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb5f109b-305d-4eec-a3c9-b917a0bf9981-openshift-service-ca" pod "bb5f109b-305d-4eec-a3c9-b917a0bf9981" (UID: "bb5f109b-305d-4eec-a3c9-b917a0bf9981"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:45:39.396716 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:39.396690 5024 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb5f109b-305d-4eec-a3c9-b917a0bf9981-openshift-service-ca\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:45:39.398246 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:39.398224 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb5f109b-305d-4eec-a3c9-b917a0bf9981-kube-api-access-8pvpq" pod "bb5f109b-305d-4eec-a3c9-b917a0bf9981" (UID: "bb5f109b-305d-4eec-a3c9-b917a0bf9981"). InnerVolumeSpecName "kube-api-access-8pvpq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:45:39.398459 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:39.398437 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb5f109b-305d-4eec-a3c9-b917a0bf9981-metrics-certs" pod "bb5f109b-305d-4eec-a3c9-b917a0bf9981" (UID: "bb5f109b-305d-4eec-a3c9-b917a0bf9981"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 13:45:39.398764 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:39.398741 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb5f109b-305d-4eec-a3c9-b917a0bf9981-maas-api-tls" pod "bb5f109b-305d-4eec-a3c9-b917a0bf9981" (UID: "bb5f109b-305d-4eec-a3c9-b917a0bf9981"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 13:45:39.497799 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:39.497764 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8pvpq\" (UniqueName: \"kubernetes.io/projected/bb5f109b-305d-4eec-a3c9-b917a0bf9981-kube-api-access-8pvpq\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:45:39.497799 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:39.497799 5024 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bb5f109b-305d-4eec-a3c9-b917a0bf9981-metrics-certs\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:45:39.497993 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:39.497812 5024 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bb5f109b-305d-4eec-a3c9-b917a0bf9981-maas-api-tls\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:45:39.507851 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:45:39.507811 5024 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d7mwz" containerName="extractor" Nov 03 13:45:39.563161 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:39.563087 5024 generic.go:356] "Generic (PLEG): container finished" podID="bb5f109b-305d-4eec-a3c9-b917a0bf9981" containerID="90f475e3677a120a451ad1e577fe36e8f65a53b684a3d3e344a1dc3e747236cf" exitCode=0 Nov 03 13:45:39.563161 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:39.563136 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a4ab8e-b94b5469d-kwm6h" Nov 03 13:45:39.563161 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:39.563145 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a4ab8e-b94b5469d-kwm6h" event={"ID":"bb5f109b-305d-4eec-a3c9-b917a0bf9981","Type":"ContainerDied","Data":"90f475e3677a120a451ad1e577fe36e8f65a53b684a3d3e344a1dc3e747236cf"} Nov 03 13:45:39.563411 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:39.563188 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a4ab8e-b94b5469d-kwm6h" event={"ID":"bb5f109b-305d-4eec-a3c9-b917a0bf9981","Type":"ContainerDied","Data":"34dc125c3728b57262418771231f4264d2b6804b8cacaebfa05a38acb2f0d953"} Nov 03 13:45:39.563411 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:39.563209 5024 scope.go:122] "RemoveContainer" containerID="90f475e3677a120a451ad1e577fe36e8f65a53b684a3d3e344a1dc3e747236cf" Nov 03 13:45:39.574135 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:39.574109 5024 scope.go:122] "RemoveContainer" containerID="90f475e3677a120a451ad1e577fe36e8f65a53b684a3d3e344a1dc3e747236cf" Nov 03 13:45:39.574517 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:45:39.574482 5024 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"90f475e3677a120a451ad1e577fe36e8f65a53b684a3d3e344a1dc3e747236cf\": container with ID starting with 90f475e3677a120a451ad1e577fe36e8f65a53b684a3d3e344a1dc3e747236cf not found: ID does not exist" containerID="90f475e3677a120a451ad1e577fe36e8f65a53b684a3d3e344a1dc3e747236cf" Nov 03 13:45:39.574625 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:39.574520 5024 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90f475e3677a120a451ad1e577fe36e8f65a53b684a3d3e344a1dc3e747236cf"} err="failed to get container status \"90f475e3677a120a451ad1e577fe36e8f65a53b684a3d3e344a1dc3e747236cf\": rpc error: code = NotFound desc = could not find container \"90f475e3677a120a451ad1e577fe36e8f65a53b684a3d3e344a1dc3e747236cf\": container with ID starting with 90f475e3677a120a451ad1e577fe36e8f65a53b684a3d3e344a1dc3e747236cf not found: ID does not exist" Nov 03 13:45:39.601682 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:39.601659 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-a4ab8e-b94b5469d-kwm6h"] Nov 03 13:45:39.608217 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:39.608198 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-a4ab8e-b94b5469d-kwm6h"] Nov 03 13:45:40.110316 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:40.110254 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb5f109b-305d-4eec-a3c9-b917a0bf9981" path="/var/lib/kubelet/pods/bb5f109b-305d-4eec-a3c9-b917a0bf9981/volumes" Nov 03 13:45:41.047722 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:41.047683 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-ded926-openshift-default-6bdf65b545-pjqfb"] Nov 03 13:45:41.048050 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:41.048015 5024 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-ded926-openshift-default-6bdf65b545-pjqfb" podUID="736b02c5-9833-4f1b-9f9a-e4c22ab23ac9" containerName="istio-proxy" containerID="cri-o://d801f9bdd21d9a710d7f86b21b1822b6c6daf6b25709307c6052bfc6d103b5dc" gracePeriod=30 Nov 03 13:45:45.823856 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:45.823819 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-a4ab8e-68655794c-7c8x7" podUID="7b571fd7-340c-455e-be5a-479ed17d7517" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 13:45:45.910775 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:45.910694 5024 patch_prober.go:29] interesting pod/e2e-extmod-a-ded926-openshift-default-6bdf65b545-pjqfb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 13:45:45.910917 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:45.910768 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-ded926-openshift-default-6bdf65b545-pjqfb" podUID="736b02c5-9833-4f1b-9f9a-e4c22ab23ac9" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 13:45:46.303399 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:46.303365 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-ded926-openshift-default-6bdf65b545-pjqfb" Nov 03 13:45:46.465210 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:46.465130 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/736b02c5-9833-4f1b-9f9a-e4c22ab23ac9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/736b02c5-9833-4f1b-9f9a-e4c22ab23ac9-istiod-ca-cert\") pod \"736b02c5-9833-4f1b-9f9a-e4c22ab23ac9\" (UID: \"736b02c5-9833-4f1b-9f9a-e4c22ab23ac9\") " Nov 03 13:45:46.465210 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:46.465176 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/736b02c5-9833-4f1b-9f9a-e4c22ab23ac9-kube-api-access-djf4m\" (UniqueName: \"kubernetes.io/projected/736b02c5-9833-4f1b-9f9a-e4c22ab23ac9-kube-api-access-djf4m\") pod \"736b02c5-9833-4f1b-9f9a-e4c22ab23ac9\" (UID: \"736b02c5-9833-4f1b-9f9a-e4c22ab23ac9\") " Nov 03 13:45:46.465448 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:46.465307 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/736b02c5-9833-4f1b-9f9a-e4c22ab23ac9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/736b02c5-9833-4f1b-9f9a-e4c22ab23ac9-istio-envoy\") pod \"736b02c5-9833-4f1b-9f9a-e4c22ab23ac9\" (UID: \"736b02c5-9833-4f1b-9f9a-e4c22ab23ac9\") " Nov 03 13:45:46.465448 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:46.465343 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/736b02c5-9833-4f1b-9f9a-e4c22ab23ac9-istio-token\" (UniqueName: \"kubernetes.io/projected/736b02c5-9833-4f1b-9f9a-e4c22ab23ac9-istio-token\") pod \"736b02c5-9833-4f1b-9f9a-e4c22ab23ac9\" (UID: \"736b02c5-9833-4f1b-9f9a-e4c22ab23ac9\") " Nov 03 13:45:46.465448 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:46.465388 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/736b02c5-9833-4f1b-9f9a-e4c22ab23ac9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/736b02c5-9833-4f1b-9f9a-e4c22ab23ac9-workload-socket\") pod \"736b02c5-9833-4f1b-9f9a-e4c22ab23ac9\" (UID: \"736b02c5-9833-4f1b-9f9a-e4c22ab23ac9\") " Nov 03 13:45:46.465611 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:46.465505 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/736b02c5-9833-4f1b-9f9a-e4c22ab23ac9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/736b02c5-9833-4f1b-9f9a-e4c22ab23ac9-istio-podinfo\") pod \"736b02c5-9833-4f1b-9f9a-e4c22ab23ac9\" (UID: \"736b02c5-9833-4f1b-9f9a-e4c22ab23ac9\") " Nov 03 13:45:46.465611 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:46.465539 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/736b02c5-9833-4f1b-9f9a-e4c22ab23ac9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/736b02c5-9833-4f1b-9f9a-e4c22ab23ac9-istio-data\") pod \"736b02c5-9833-4f1b-9f9a-e4c22ab23ac9\" (UID: \"736b02c5-9833-4f1b-9f9a-e4c22ab23ac9\") " Nov 03 13:45:46.465611 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:46.465553 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/736b02c5-9833-4f1b-9f9a-e4c22ab23ac9-istiod-ca-cert" pod "736b02c5-9833-4f1b-9f9a-e4c22ab23ac9" (UID: "736b02c5-9833-4f1b-9f9a-e4c22ab23ac9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:45:46.465769 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:46.465693 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/736b02c5-9833-4f1b-9f9a-e4c22ab23ac9-workload-socket" pod "736b02c5-9833-4f1b-9f9a-e4c22ab23ac9" (UID: "736b02c5-9833-4f1b-9f9a-e4c22ab23ac9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:45:46.465769 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:46.465737 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/736b02c5-9833-4f1b-9f9a-e4c22ab23ac9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/736b02c5-9833-4f1b-9f9a-e4c22ab23ac9-workload-certs\") pod \"736b02c5-9833-4f1b-9f9a-e4c22ab23ac9\" (UID: \"736b02c5-9833-4f1b-9f9a-e4c22ab23ac9\") " Nov 03 13:45:46.465873 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:46.465780 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/736b02c5-9833-4f1b-9f9a-e4c22ab23ac9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/736b02c5-9833-4f1b-9f9a-e4c22ab23ac9-credential-socket\") pod \"736b02c5-9833-4f1b-9f9a-e4c22ab23ac9\" (UID: \"736b02c5-9833-4f1b-9f9a-e4c22ab23ac9\") " Nov 03 13:45:46.465873 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:46.465789 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/736b02c5-9833-4f1b-9f9a-e4c22ab23ac9-istio-data" pod "736b02c5-9833-4f1b-9f9a-e4c22ab23ac9" (UID: "736b02c5-9833-4f1b-9f9a-e4c22ab23ac9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:45:46.465989 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:46.465906 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/736b02c5-9833-4f1b-9f9a-e4c22ab23ac9-workload-certs" pod "736b02c5-9833-4f1b-9f9a-e4c22ab23ac9" (UID: "736b02c5-9833-4f1b-9f9a-e4c22ab23ac9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:45:46.466053 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:46.466013 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/736b02c5-9833-4f1b-9f9a-e4c22ab23ac9-credential-socket" pod "736b02c5-9833-4f1b-9f9a-e4c22ab23ac9" (UID: "736b02c5-9833-4f1b-9f9a-e4c22ab23ac9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:45:46.466244 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:46.466223 5024 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/736b02c5-9833-4f1b-9f9a-e4c22ab23ac9-workload-certs\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:45:46.466349 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:46.466248 5024 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/736b02c5-9833-4f1b-9f9a-e4c22ab23ac9-credential-socket\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:45:46.466349 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:46.466264 5024 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/736b02c5-9833-4f1b-9f9a-e4c22ab23ac9-istiod-ca-cert\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:45:46.466349 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:46.466279 5024 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/736b02c5-9833-4f1b-9f9a-e4c22ab23ac9-workload-socket\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:45:46.466349 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:46.466322 5024 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/736b02c5-9833-4f1b-9f9a-e4c22ab23ac9-istio-data\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:45:46.467950 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:46.467928 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/736b02c5-9833-4f1b-9f9a-e4c22ab23ac9-istio-envoy" pod "736b02c5-9833-4f1b-9f9a-e4c22ab23ac9" (UID: "736b02c5-9833-4f1b-9f9a-e4c22ab23ac9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:45:46.468086 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:46.468059 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/736b02c5-9833-4f1b-9f9a-e4c22ab23ac9-istio-podinfo" pod "736b02c5-9833-4f1b-9f9a-e4c22ab23ac9" (UID: "736b02c5-9833-4f1b-9f9a-e4c22ab23ac9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 13:45:46.468183 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:46.468099 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/736b02c5-9833-4f1b-9f9a-e4c22ab23ac9-istio-token" pod "736b02c5-9833-4f1b-9f9a-e4c22ab23ac9" (UID: "736b02c5-9833-4f1b-9f9a-e4c22ab23ac9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:45:46.468183 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:46.468162 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/736b02c5-9833-4f1b-9f9a-e4c22ab23ac9-kube-api-access-djf4m" pod "736b02c5-9833-4f1b-9f9a-e4c22ab23ac9" (UID: "736b02c5-9833-4f1b-9f9a-e4c22ab23ac9"). InnerVolumeSpecName "kube-api-access-djf4m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:45:46.567896 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:46.567863 5024 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/736b02c5-9833-4f1b-9f9a-e4c22ab23ac9-istio-envoy\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:45:46.567896 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:46.567893 5024 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/736b02c5-9833-4f1b-9f9a-e4c22ab23ac9-istio-token\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:45:46.567896 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:46.567905 5024 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/736b02c5-9833-4f1b-9f9a-e4c22ab23ac9-istio-podinfo\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:45:46.568141 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:46.567916 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-djf4m\" (UniqueName: \"kubernetes.io/projected/736b02c5-9833-4f1b-9f9a-e4c22ab23ac9-kube-api-access-djf4m\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:45:46.601729 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:46.601694 5024 generic.go:356] "Generic (PLEG): container finished" podID="736b02c5-9833-4f1b-9f9a-e4c22ab23ac9" containerID="d801f9bdd21d9a710d7f86b21b1822b6c6daf6b25709307c6052bfc6d103b5dc" exitCode=0 Nov 03 13:45:46.601881 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:46.601739 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-ded926-openshift-default-6bdf65b545-pjqfb" Nov 03 13:45:46.601881 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:46.601777 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-ded926-openshift-default-6bdf65b545-pjqfb" event={"ID":"736b02c5-9833-4f1b-9f9a-e4c22ab23ac9","Type":"ContainerDied","Data":"d801f9bdd21d9a710d7f86b21b1822b6c6daf6b25709307c6052bfc6d103b5dc"} Nov 03 13:45:46.601881 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:46.601838 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-ded926-openshift-default-6bdf65b545-pjqfb" event={"ID":"736b02c5-9833-4f1b-9f9a-e4c22ab23ac9","Type":"ContainerDied","Data":"c029b30dcf6dc8d768eb4fd79e6b3fbd528db5137c7c1f64a0c6b279a790ae8c"} Nov 03 13:45:46.601881 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:46.601874 5024 scope.go:122] "RemoveContainer" containerID="d801f9bdd21d9a710d7f86b21b1822b6c6daf6b25709307c6052bfc6d103b5dc" Nov 03 13:45:46.611406 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:46.611384 5024 scope.go:122] "RemoveContainer" containerID="80f9d6d535afd0f850b398d01f7f1d9c71d9147c90bff43a2c95173a88c39ca9" Nov 03 13:45:46.619936 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:46.619916 5024 scope.go:122] "RemoveContainer" containerID="d801f9bdd21d9a710d7f86b21b1822b6c6daf6b25709307c6052bfc6d103b5dc" Nov 03 13:45:46.620200 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:45:46.620178 5024 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d801f9bdd21d9a710d7f86b21b1822b6c6daf6b25709307c6052bfc6d103b5dc\": container with ID starting with d801f9bdd21d9a710d7f86b21b1822b6c6daf6b25709307c6052bfc6d103b5dc not found: ID does not exist" containerID="d801f9bdd21d9a710d7f86b21b1822b6c6daf6b25709307c6052bfc6d103b5dc" Nov 03 13:45:46.620272 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:46.620208 5024 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d801f9bdd21d9a710d7f86b21b1822b6c6daf6b25709307c6052bfc6d103b5dc"} err="failed to get container status \"d801f9bdd21d9a710d7f86b21b1822b6c6daf6b25709307c6052bfc6d103b5dc\": rpc error: code = NotFound desc = could not find container \"d801f9bdd21d9a710d7f86b21b1822b6c6daf6b25709307c6052bfc6d103b5dc\": container with ID starting with d801f9bdd21d9a710d7f86b21b1822b6c6daf6b25709307c6052bfc6d103b5dc not found: ID does not exist" Nov 03 13:45:46.620272 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:46.620225 5024 scope.go:122] "RemoveContainer" containerID="80f9d6d535afd0f850b398d01f7f1d9c71d9147c90bff43a2c95173a88c39ca9" Nov 03 13:45:46.620527 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:45:46.620509 5024 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"80f9d6d535afd0f850b398d01f7f1d9c71d9147c90bff43a2c95173a88c39ca9\": container with ID starting with 80f9d6d535afd0f850b398d01f7f1d9c71d9147c90bff43a2c95173a88c39ca9 not found: ID does not exist" containerID="80f9d6d535afd0f850b398d01f7f1d9c71d9147c90bff43a2c95173a88c39ca9" Nov 03 13:45:46.620574 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:46.620536 5024 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80f9d6d535afd0f850b398d01f7f1d9c71d9147c90bff43a2c95173a88c39ca9"} err="failed to get container status \"80f9d6d535afd0f850b398d01f7f1d9c71d9147c90bff43a2c95173a88c39ca9\": rpc error: code = NotFound desc = could not find container \"80f9d6d535afd0f850b398d01f7f1d9c71d9147c90bff43a2c95173a88c39ca9\": container with ID starting with 80f9d6d535afd0f850b398d01f7f1d9c71d9147c90bff43a2c95173a88c39ca9 not found: ID does not exist" Nov 03 13:45:46.636923 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:46.636892 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-ded926-openshift-default-6bdf65b545-pjqfb"] Nov 03 13:45:46.643988 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:46.643955 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-ded926-openshift-default-6bdf65b545-pjqfb"] Nov 03 13:45:48.108637 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:48.108598 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="736b02c5-9833-4f1b-9f9a-e4c22ab23ac9" path="/var/lib/kubelet/pods/736b02c5-9833-4f1b-9f9a-e4c22ab23ac9/volumes" Nov 03 13:45:48.444219 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:48.444184 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-a4ab8e-68655794c-7c8x7" Nov 03 13:45:48.595451 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:48.595396 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b571fd7-340c-455e-be5a-479ed17d7517-kube-api-access-gzr6r\" (UniqueName: \"kubernetes.io/projected/7b571fd7-340c-455e-be5a-479ed17d7517-kube-api-access-gzr6r\") pod \"7b571fd7-340c-455e-be5a-479ed17d7517\" (UID: \"7b571fd7-340c-455e-be5a-479ed17d7517\") " Nov 03 13:45:48.595451 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:48.595453 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7b571fd7-340c-455e-be5a-479ed17d7517-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b571fd7-340c-455e-be5a-479ed17d7517-plugins-config-volume\") pod \"7b571fd7-340c-455e-be5a-479ed17d7517\" (UID: \"7b571fd7-340c-455e-be5a-479ed17d7517\") " Nov 03 13:45:48.595894 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:48.595873 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b571fd7-340c-455e-be5a-479ed17d7517-plugins-config-volume" pod "7b571fd7-340c-455e-be5a-479ed17d7517" (UID: "7b571fd7-340c-455e-be5a-479ed17d7517"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:45:48.597741 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:48.597715 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b571fd7-340c-455e-be5a-479ed17d7517-kube-api-access-gzr6r" pod "7b571fd7-340c-455e-be5a-479ed17d7517" (UID: "7b571fd7-340c-455e-be5a-479ed17d7517"). InnerVolumeSpecName "kube-api-access-gzr6r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:45:48.614429 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:48.614402 5024 generic.go:356] "Generic (PLEG): container finished" podID="7b571fd7-340c-455e-be5a-479ed17d7517" containerID="bd7484986d1d81caa8d9b7003da562b709053386e05f9d2a97fd6b84b8832e76" exitCode=0 Nov 03 13:45:48.614567 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:48.614469 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-a4ab8e-68655794c-7c8x7" Nov 03 13:45:48.614567 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:48.614483 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-a4ab8e-68655794c-7c8x7" event={"ID":"7b571fd7-340c-455e-be5a-479ed17d7517","Type":"ContainerDied","Data":"bd7484986d1d81caa8d9b7003da562b709053386e05f9d2a97fd6b84b8832e76"} Nov 03 13:45:48.614567 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:48.614536 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-a4ab8e-68655794c-7c8x7" event={"ID":"7b571fd7-340c-455e-be5a-479ed17d7517","Type":"ContainerDied","Data":"3552fbc31a0750fc583482247a9c390fb682d2cff898bbfed5c2985134446f38"} Nov 03 13:45:48.614567 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:48.614566 5024 scope.go:122] "RemoveContainer" containerID="bd7484986d1d81caa8d9b7003da562b709053386e05f9d2a97fd6b84b8832e76" Nov 03 13:45:48.623499 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:48.623482 5024 scope.go:122] "RemoveContainer" containerID="bd7484986d1d81caa8d9b7003da562b709053386e05f9d2a97fd6b84b8832e76" Nov 03 13:45:48.623805 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:45:48.623785 5024 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd7484986d1d81caa8d9b7003da562b709053386e05f9d2a97fd6b84b8832e76\": container with ID starting with bd7484986d1d81caa8d9b7003da562b709053386e05f9d2a97fd6b84b8832e76 not found: ID does not exist" containerID="bd7484986d1d81caa8d9b7003da562b709053386e05f9d2a97fd6b84b8832e76" Nov 03 13:45:48.623871 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:48.623812 5024 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd7484986d1d81caa8d9b7003da562b709053386e05f9d2a97fd6b84b8832e76"} err="failed to get container status \"bd7484986d1d81caa8d9b7003da562b709053386e05f9d2a97fd6b84b8832e76\": rpc error: code = NotFound desc = could not find container \"bd7484986d1d81caa8d9b7003da562b709053386e05f9d2a97fd6b84b8832e76\": container with ID starting with bd7484986d1d81caa8d9b7003da562b709053386e05f9d2a97fd6b84b8832e76 not found: ID does not exist" Nov 03 13:45:48.642266 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:48.642238 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-a4ab8e-68655794c-7c8x7"] Nov 03 13:45:48.646170 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:48.646144 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-a4ab8e-68655794c-7c8x7"] Nov 03 13:45:48.697354 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:48.697269 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gzr6r\" (UniqueName: \"kubernetes.io/projected/7b571fd7-340c-455e-be5a-479ed17d7517-kube-api-access-gzr6r\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:45:48.697354 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:48.697310 5024 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b571fd7-340c-455e-be5a-479ed17d7517-plugins-config-volume\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:45:50.108629 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:50.108593 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7b571fd7-340c-455e-be5a-479ed17d7517" path="/var/lib/kubelet/pods/7b571fd7-340c-455e-be5a-479ed17d7517/volumes" Nov 03 13:45:50.627381 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:50.627345 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-wwvqr"] Nov 03 13:45:50.627627 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:50.627599 5024 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-wwvqr" podUID="fffd24a4-d55d-4661-a37d-c344d3343698" containerName="istio-proxy" containerID="cri-o://32d699d9bfb32447f36dcd5aa244107d932d046cdacc4d81479e50a884ded653" gracePeriod=30 Nov 03 13:45:50.745364 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:50.743594 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-7c192b-65886b4cf7-n24wm"] Nov 03 13:45:50.745364 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:50.743842 5024 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-7c192b-65886b4cf7-n24wm" podUID="11a9f177-f213-4929-9a64-19065b49b3f5" containerName="maas-api" containerID="cri-o://1b2fbb0821772430b070fdaef7983496a80f01d2782f460cf53575a44ba4fda7" gracePeriod=30 Nov 03 13:45:51.028011 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:51.027985 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-7c192b-65886b4cf7-n24wm" Nov 03 13:45:51.127619 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:51.127588 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/11a9f177-f213-4929-9a64-19065b49b3f5-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/11a9f177-f213-4929-9a64-19065b49b3f5-maas-api-tls\") pod \"11a9f177-f213-4929-9a64-19065b49b3f5\" (UID: \"11a9f177-f213-4929-9a64-19065b49b3f5\") " Nov 03 13:45:51.127619 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:51.127624 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/11a9f177-f213-4929-9a64-19065b49b3f5-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/11a9f177-f213-4929-9a64-19065b49b3f5-openshift-service-ca\") pod \"11a9f177-f213-4929-9a64-19065b49b3f5\" (UID: \"11a9f177-f213-4929-9a64-19065b49b3f5\") " Nov 03 13:45:51.128159 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:51.127709 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11a9f177-f213-4929-9a64-19065b49b3f5-kube-api-access-rdnwq\" (UniqueName: \"kubernetes.io/projected/11a9f177-f213-4929-9a64-19065b49b3f5-kube-api-access-rdnwq\") pod \"11a9f177-f213-4929-9a64-19065b49b3f5\" (UID: \"11a9f177-f213-4929-9a64-19065b49b3f5\") " Nov 03 13:45:51.128159 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:51.127771 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/11a9f177-f213-4929-9a64-19065b49b3f5-metrics-certs\" (UniqueName: \"kubernetes.io/secret/11a9f177-f213-4929-9a64-19065b49b3f5-metrics-certs\") pod \"11a9f177-f213-4929-9a64-19065b49b3f5\" (UID: \"11a9f177-f213-4929-9a64-19065b49b3f5\") " Nov 03 13:45:51.128159 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:51.128088 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11a9f177-f213-4929-9a64-19065b49b3f5-openshift-service-ca" pod "11a9f177-f213-4929-9a64-19065b49b3f5" (UID: "11a9f177-f213-4929-9a64-19065b49b3f5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:45:51.129720 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:51.129688 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/11a9f177-f213-4929-9a64-19065b49b3f5-maas-api-tls" pod "11a9f177-f213-4929-9a64-19065b49b3f5" (UID: "11a9f177-f213-4929-9a64-19065b49b3f5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 13:45:51.129852 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:51.129824 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11a9f177-f213-4929-9a64-19065b49b3f5-kube-api-access-rdnwq" pod "11a9f177-f213-4929-9a64-19065b49b3f5" (UID: "11a9f177-f213-4929-9a64-19065b49b3f5"). InnerVolumeSpecName "kube-api-access-rdnwq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:45:51.129921 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:51.129903 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/11a9f177-f213-4929-9a64-19065b49b3f5-metrics-certs" pod "11a9f177-f213-4929-9a64-19065b49b3f5" (UID: "11a9f177-f213-4929-9a64-19065b49b3f5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 13:45:51.228934 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:51.228835 5024 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/11a9f177-f213-4929-9a64-19065b49b3f5-maas-api-tls\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:45:51.228934 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:51.228876 5024 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/11a9f177-f213-4929-9a64-19065b49b3f5-openshift-service-ca\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:45:51.228934 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:51.228890 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rdnwq\" (UniqueName: \"kubernetes.io/projected/11a9f177-f213-4929-9a64-19065b49b3f5-kube-api-access-rdnwq\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:45:51.228934 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:51.228905 5024 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/11a9f177-f213-4929-9a64-19065b49b3f5-metrics-certs\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:45:51.636135 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:51.636096 5024 generic.go:356] "Generic (PLEG): container finished" podID="11a9f177-f213-4929-9a64-19065b49b3f5" containerID="1b2fbb0821772430b070fdaef7983496a80f01d2782f460cf53575a44ba4fda7" exitCode=0 Nov 03 13:45:51.636359 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:51.636153 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-7c192b-65886b4cf7-n24wm" Nov 03 13:45:51.636359 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:51.636171 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-7c192b-65886b4cf7-n24wm" event={"ID":"11a9f177-f213-4929-9a64-19065b49b3f5","Type":"ContainerDied","Data":"1b2fbb0821772430b070fdaef7983496a80f01d2782f460cf53575a44ba4fda7"} Nov 03 13:45:51.636359 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:51.636226 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-7c192b-65886b4cf7-n24wm" event={"ID":"11a9f177-f213-4929-9a64-19065b49b3f5","Type":"ContainerDied","Data":"827c29fd3592cc5c7ab2b7d41cea228398586aac1eaf7d63b0402915abad59f0"} Nov 03 13:45:51.636359 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:51.636249 5024 scope.go:122] "RemoveContainer" containerID="1b2fbb0821772430b070fdaef7983496a80f01d2782f460cf53575a44ba4fda7" Nov 03 13:45:51.646309 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:51.646276 5024 scope.go:122] "RemoveContainer" containerID="1b2fbb0821772430b070fdaef7983496a80f01d2782f460cf53575a44ba4fda7" Nov 03 13:45:51.646563 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:45:51.646537 5024 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b2fbb0821772430b070fdaef7983496a80f01d2782f460cf53575a44ba4fda7\": container with ID starting with 1b2fbb0821772430b070fdaef7983496a80f01d2782f460cf53575a44ba4fda7 not found: ID does not exist" containerID="1b2fbb0821772430b070fdaef7983496a80f01d2782f460cf53575a44ba4fda7" Nov 03 13:45:51.646618 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:51.646574 5024 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b2fbb0821772430b070fdaef7983496a80f01d2782f460cf53575a44ba4fda7"} err="failed to get container status \"1b2fbb0821772430b070fdaef7983496a80f01d2782f460cf53575a44ba4fda7\": rpc error: code = NotFound desc = could not find container \"1b2fbb0821772430b070fdaef7983496a80f01d2782f460cf53575a44ba4fda7\": container with ID starting with 1b2fbb0821772430b070fdaef7983496a80f01d2782f460cf53575a44ba4fda7 not found: ID does not exist" Nov 03 13:45:51.669569 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:51.669542 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-7c192b-65886b4cf7-n24wm"] Nov 03 13:45:51.673925 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:51.673898 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-7c192b-65886b4cf7-n24wm"] Nov 03 13:45:52.118120 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:52.118081 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="11a9f177-f213-4929-9a64-19065b49b3f5" path="/var/lib/kubelet/pods/11a9f177-f213-4929-9a64-19065b49b3f5/volumes" Nov 03 13:45:55.892299 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:55.892262 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-wwvqr" Nov 03 13:45:55.973165 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:55.973067 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fffd24a4-d55d-4661-a37d-c344d3343698-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fffd24a4-d55d-4661-a37d-c344d3343698-credential-socket\") pod \"fffd24a4-d55d-4661-a37d-c344d3343698\" (UID: \"fffd24a4-d55d-4661-a37d-c344d3343698\") " Nov 03 13:45:55.973165 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:55.973121 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fffd24a4-d55d-4661-a37d-c344d3343698-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fffd24a4-d55d-4661-a37d-c344d3343698-workload-socket\") pod \"fffd24a4-d55d-4661-a37d-c344d3343698\" (UID: \"fffd24a4-d55d-4661-a37d-c344d3343698\") " Nov 03 13:45:55.973165 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:55.973148 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fffd24a4-d55d-4661-a37d-c344d3343698-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fffd24a4-d55d-4661-a37d-c344d3343698-istio-data\") pod \"fffd24a4-d55d-4661-a37d-c344d3343698\" (UID: \"fffd24a4-d55d-4661-a37d-c344d3343698\") " Nov 03 13:45:55.973510 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:55.973173 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fffd24a4-d55d-4661-a37d-c344d3343698-kube-api-access-74rcf\" (UniqueName: \"kubernetes.io/projected/fffd24a4-d55d-4661-a37d-c344d3343698-kube-api-access-74rcf\") pod \"fffd24a4-d55d-4661-a37d-c344d3343698\" (UID: \"fffd24a4-d55d-4661-a37d-c344d3343698\") " Nov 03 13:45:55.973510 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:55.973209 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fffd24a4-d55d-4661-a37d-c344d3343698-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fffd24a4-d55d-4661-a37d-c344d3343698-istio-envoy\") pod \"fffd24a4-d55d-4661-a37d-c344d3343698\" (UID: \"fffd24a4-d55d-4661-a37d-c344d3343698\") " Nov 03 13:45:55.973510 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:55.973250 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fffd24a4-d55d-4661-a37d-c344d3343698-istio-token\" (UniqueName: \"kubernetes.io/projected/fffd24a4-d55d-4661-a37d-c344d3343698-istio-token\") pod \"fffd24a4-d55d-4661-a37d-c344d3343698\" (UID: \"fffd24a4-d55d-4661-a37d-c344d3343698\") " Nov 03 13:45:55.973510 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:55.973344 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fffd24a4-d55d-4661-a37d-c344d3343698-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fffd24a4-d55d-4661-a37d-c344d3343698-istio-podinfo\") pod \"fffd24a4-d55d-4661-a37d-c344d3343698\" (UID: \"fffd24a4-d55d-4661-a37d-c344d3343698\") " Nov 03 13:45:55.973510 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:55.973410 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fffd24a4-d55d-4661-a37d-c344d3343698-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fffd24a4-d55d-4661-a37d-c344d3343698-workload-certs\") pod \"fffd24a4-d55d-4661-a37d-c344d3343698\" (UID: \"fffd24a4-d55d-4661-a37d-c344d3343698\") " Nov 03 13:45:55.973510 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:55.973434 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fffd24a4-d55d-4661-a37d-c344d3343698-credential-socket" pod "fffd24a4-d55d-4661-a37d-c344d3343698" (UID: "fffd24a4-d55d-4661-a37d-c344d3343698"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:45:55.973510 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:55.973445 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fffd24a4-d55d-4661-a37d-c344d3343698-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fffd24a4-d55d-4661-a37d-c344d3343698-istiod-ca-cert\") pod \"fffd24a4-d55d-4661-a37d-c344d3343698\" (UID: \"fffd24a4-d55d-4661-a37d-c344d3343698\") " Nov 03 13:45:55.973872 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:55.973547 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fffd24a4-d55d-4661-a37d-c344d3343698-workload-socket" pod "fffd24a4-d55d-4661-a37d-c344d3343698" (UID: "fffd24a4-d55d-4661-a37d-c344d3343698"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:45:55.973872 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:55.973561 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fffd24a4-d55d-4661-a37d-c344d3343698-istio-data" pod "fffd24a4-d55d-4661-a37d-c344d3343698" (UID: "fffd24a4-d55d-4661-a37d-c344d3343698"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:45:55.973872 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:55.973750 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fffd24a4-d55d-4661-a37d-c344d3343698-workload-certs" pod "fffd24a4-d55d-4661-a37d-c344d3343698" (UID: "fffd24a4-d55d-4661-a37d-c344d3343698"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:45:55.973872 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:55.973781 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fffd24a4-d55d-4661-a37d-c344d3343698-istiod-ca-cert" pod "fffd24a4-d55d-4661-a37d-c344d3343698" (UID: "fffd24a4-d55d-4661-a37d-c344d3343698"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:45:55.974091 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:55.973937 5024 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fffd24a4-d55d-4661-a37d-c344d3343698-workload-certs\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:45:55.974091 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:55.973955 5024 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fffd24a4-d55d-4661-a37d-c344d3343698-istiod-ca-cert\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:45:55.974091 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:55.973970 5024 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fffd24a4-d55d-4661-a37d-c344d3343698-credential-socket\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:45:55.974091 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:55.973984 5024 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fffd24a4-d55d-4661-a37d-c344d3343698-workload-socket\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:45:55.974091 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:55.973997 5024 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fffd24a4-d55d-4661-a37d-c344d3343698-istio-data\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:45:55.975773 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:55.975745 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fffd24a4-d55d-4661-a37d-c344d3343698-istio-envoy" pod "fffd24a4-d55d-4661-a37d-c344d3343698" (UID: "fffd24a4-d55d-4661-a37d-c344d3343698"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:45:55.975945 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:55.975787 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fffd24a4-d55d-4661-a37d-c344d3343698-istio-podinfo" pod "fffd24a4-d55d-4661-a37d-c344d3343698" (UID: "fffd24a4-d55d-4661-a37d-c344d3343698"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 13:45:55.975945 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:55.975874 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fffd24a4-d55d-4661-a37d-c344d3343698-istio-token" pod "fffd24a4-d55d-4661-a37d-c344d3343698" (UID: "fffd24a4-d55d-4661-a37d-c344d3343698"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:45:55.975945 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:55.975895 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fffd24a4-d55d-4661-a37d-c344d3343698-kube-api-access-74rcf" pod "fffd24a4-d55d-4661-a37d-c344d3343698" (UID: "fffd24a4-d55d-4661-a37d-c344d3343698"). InnerVolumeSpecName "kube-api-access-74rcf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:45:56.075300 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:56.075252 5024 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fffd24a4-d55d-4661-a37d-c344d3343698-istio-podinfo\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:45:56.075449 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:56.075312 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-74rcf\" (UniqueName: \"kubernetes.io/projected/fffd24a4-d55d-4661-a37d-c344d3343698-kube-api-access-74rcf\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:45:56.075449 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:56.075330 5024 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fffd24a4-d55d-4661-a37d-c344d3343698-istio-envoy\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:45:56.075449 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:56.075342 5024 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fffd24a4-d55d-4661-a37d-c344d3343698-istio-token\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:45:56.664469 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:56.664428 5024 generic.go:356] "Generic (PLEG): container finished" podID="fffd24a4-d55d-4661-a37d-c344d3343698" containerID="32d699d9bfb32447f36dcd5aa244107d932d046cdacc4d81479e50a884ded653" exitCode=0 Nov 03 13:45:56.664634 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:56.664485 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-wwvqr" Nov 03 13:45:56.664634 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:56.664522 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-wwvqr" event={"ID":"fffd24a4-d55d-4661-a37d-c344d3343698","Type":"ContainerDied","Data":"32d699d9bfb32447f36dcd5aa244107d932d046cdacc4d81479e50a884ded653"} Nov 03 13:45:56.664634 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:56.664583 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-wwvqr" event={"ID":"fffd24a4-d55d-4661-a37d-c344d3343698","Type":"ContainerDied","Data":"01027176032f1ba6e445c9a9520865f3c993d7e7b2ce2c170cec6e7845375509"} Nov 03 13:45:56.664634 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:56.664612 5024 scope.go:122] "RemoveContainer" containerID="32d699d9bfb32447f36dcd5aa244107d932d046cdacc4d81479e50a884ded653" Nov 03 13:45:56.673560 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:56.673538 5024 scope.go:122] "RemoveContainer" containerID="32d699d9bfb32447f36dcd5aa244107d932d046cdacc4d81479e50a884ded653" Nov 03 13:45:56.673816 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:45:56.673795 5024 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32d699d9bfb32447f36dcd5aa244107d932d046cdacc4d81479e50a884ded653\": container with ID starting with 32d699d9bfb32447f36dcd5aa244107d932d046cdacc4d81479e50a884ded653 not found: ID does not exist" containerID="32d699d9bfb32447f36dcd5aa244107d932d046cdacc4d81479e50a884ded653" Nov 03 13:45:56.673875 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:56.673824 5024 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32d699d9bfb32447f36dcd5aa244107d932d046cdacc4d81479e50a884ded653"} err="failed to get container status \"32d699d9bfb32447f36dcd5aa244107d932d046cdacc4d81479e50a884ded653\": rpc error: code = NotFound desc = could not find container \"32d699d9bfb32447f36dcd5aa244107d932d046cdacc4d81479e50a884ded653\": container with ID starting with 32d699d9bfb32447f36dcd5aa244107d932d046cdacc4d81479e50a884ded653 not found: ID does not exist" Nov 03 13:45:56.686434 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:56.686404 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-wwvqr"] Nov 03 13:45:56.693753 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:56.693723 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-a4ab8e-openshift-default-8645f4b6c5-wwvqr"] Nov 03 13:45:58.109448 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:58.109405 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fffd24a4-d55d-4661-a37d-c344d3343698" path="/var/lib/kubelet/pods/fffd24a4-d55d-4661-a37d-c344d3343698/volumes" Nov 03 13:45:58.347407 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:58.347374 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-k2fbp"] Nov 03 13:45:58.347625 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:58.347596 5024 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-k2fbp" podUID="fd2442f9-1e68-4d0d-b363-42f2f6a03f07" containerName="istio-proxy" containerID="cri-o://a80377780a05717214347004f3699ecbbae085d484755ef7e8087c9a859e5b2a" gracePeriod=30 Nov 03 13:45:59.382451 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:45:59.382421 5024 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-796d65f566-65br2_795bfdb5-d443-4daf-a0d1-e0bb7567416c/authorino/0.log" Nov 03 13:46:00.216710 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:00.216676 5024 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-bsvp9_8f111884-464c-4369-88ce-d3aed661014d/manager/0.log" Nov 03 13:46:01.556533 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:01.556494 5024 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-6f5b4db54-wxw9w_9af794f7-e3d1-4049-9828-499e3a49a290/manager/0.log" Nov 03 13:46:02.801824 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:02.801787 5024 patch_prober.go:29] interesting pod/e2e-extmod-b-7c192b-openshift-default-5c496b978f-k2fbp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 13:46:02.802249 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:02.801865 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-k2fbp" podUID="fd2442f9-1e68-4d0d-b363-42f2f6a03f07" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 13:46:03.599218 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:03.599192 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-k2fbp" Nov 03 13:46:03.643713 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:03.643633 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fd2442f9-1e68-4d0d-b363-42f2f6a03f07-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd2442f9-1e68-4d0d-b363-42f2f6a03f07-istio-podinfo\") pod \"fd2442f9-1e68-4d0d-b363-42f2f6a03f07\" (UID: \"fd2442f9-1e68-4d0d-b363-42f2f6a03f07\") " Nov 03 13:46:03.643713 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:03.643693 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd2442f9-1e68-4d0d-b363-42f2f6a03f07-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd2442f9-1e68-4d0d-b363-42f2f6a03f07-istio-data\") pod \"fd2442f9-1e68-4d0d-b363-42f2f6a03f07\" (UID: \"fd2442f9-1e68-4d0d-b363-42f2f6a03f07\") " Nov 03 13:46:03.643929 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:03.643787 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd2442f9-1e68-4d0d-b363-42f2f6a03f07-kube-api-access-p9mgw\" (UniqueName: \"kubernetes.io/projected/fd2442f9-1e68-4d0d-b363-42f2f6a03f07-kube-api-access-p9mgw\") pod \"fd2442f9-1e68-4d0d-b363-42f2f6a03f07\" (UID: \"fd2442f9-1e68-4d0d-b363-42f2f6a03f07\") " Nov 03 13:46:03.643929 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:03.643829 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd2442f9-1e68-4d0d-b363-42f2f6a03f07-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd2442f9-1e68-4d0d-b363-42f2f6a03f07-workload-socket\") pod \"fd2442f9-1e68-4d0d-b363-42f2f6a03f07\" (UID: \"fd2442f9-1e68-4d0d-b363-42f2f6a03f07\") " Nov 03 13:46:03.643929 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:03.643862 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd2442f9-1e68-4d0d-b363-42f2f6a03f07-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd2442f9-1e68-4d0d-b363-42f2f6a03f07-workload-certs\") pod \"fd2442f9-1e68-4d0d-b363-42f2f6a03f07\" (UID: \"fd2442f9-1e68-4d0d-b363-42f2f6a03f07\") " Nov 03 13:46:03.643929 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:03.643902 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd2442f9-1e68-4d0d-b363-42f2f6a03f07-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd2442f9-1e68-4d0d-b363-42f2f6a03f07-istiod-ca-cert\") pod \"fd2442f9-1e68-4d0d-b363-42f2f6a03f07\" (UID: \"fd2442f9-1e68-4d0d-b363-42f2f6a03f07\") " Nov 03 13:46:03.644143 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:03.643935 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd2442f9-1e68-4d0d-b363-42f2f6a03f07-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd2442f9-1e68-4d0d-b363-42f2f6a03f07-credential-socket\") pod \"fd2442f9-1e68-4d0d-b363-42f2f6a03f07\" (UID: \"fd2442f9-1e68-4d0d-b363-42f2f6a03f07\") " Nov 03 13:46:03.644143 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:03.643981 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd2442f9-1e68-4d0d-b363-42f2f6a03f07-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd2442f9-1e68-4d0d-b363-42f2f6a03f07-istio-envoy\") pod \"fd2442f9-1e68-4d0d-b363-42f2f6a03f07\" (UID: \"fd2442f9-1e68-4d0d-b363-42f2f6a03f07\") " Nov 03 13:46:03.644143 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:03.644013 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd2442f9-1e68-4d0d-b363-42f2f6a03f07-istio-token\" (UniqueName: \"kubernetes.io/projected/fd2442f9-1e68-4d0d-b363-42f2f6a03f07-istio-token\") pod \"fd2442f9-1e68-4d0d-b363-42f2f6a03f07\" (UID: \"fd2442f9-1e68-4d0d-b363-42f2f6a03f07\") " Nov 03 13:46:03.644143 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:03.644071 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd2442f9-1e68-4d0d-b363-42f2f6a03f07-istio-data" pod "fd2442f9-1e68-4d0d-b363-42f2f6a03f07" (UID: "fd2442f9-1e68-4d0d-b363-42f2f6a03f07"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:46:03.644385 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:03.644302 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd2442f9-1e68-4d0d-b363-42f2f6a03f07-workload-socket" pod "fd2442f9-1e68-4d0d-b363-42f2f6a03f07" (UID: "fd2442f9-1e68-4d0d-b363-42f2f6a03f07"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:46:03.644444 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:03.644397 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd2442f9-1e68-4d0d-b363-42f2f6a03f07-credential-socket" pod "fd2442f9-1e68-4d0d-b363-42f2f6a03f07" (UID: "fd2442f9-1e68-4d0d-b363-42f2f6a03f07"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:46:03.644515 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:03.644496 5024 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd2442f9-1e68-4d0d-b363-42f2f6a03f07-workload-socket\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:46:03.644572 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:03.644523 5024 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd2442f9-1e68-4d0d-b363-42f2f6a03f07-istio-data\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:46:03.644698 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:03.644671 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd2442f9-1e68-4d0d-b363-42f2f6a03f07-istiod-ca-cert" pod "fd2442f9-1e68-4d0d-b363-42f2f6a03f07" (UID: "fd2442f9-1e68-4d0d-b363-42f2f6a03f07"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:46:03.644823 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:03.644801 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd2442f9-1e68-4d0d-b363-42f2f6a03f07-workload-certs" pod "fd2442f9-1e68-4d0d-b363-42f2f6a03f07" (UID: "fd2442f9-1e68-4d0d-b363-42f2f6a03f07"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:46:03.646332 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:03.646277 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd2442f9-1e68-4d0d-b363-42f2f6a03f07-kube-api-access-p9mgw" pod "fd2442f9-1e68-4d0d-b363-42f2f6a03f07" (UID: "fd2442f9-1e68-4d0d-b363-42f2f6a03f07"). InnerVolumeSpecName "kube-api-access-p9mgw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:46:03.646585 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:03.646555 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd2442f9-1e68-4d0d-b363-42f2f6a03f07-istio-token" pod "fd2442f9-1e68-4d0d-b363-42f2f6a03f07" (UID: "fd2442f9-1e68-4d0d-b363-42f2f6a03f07"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:46:03.646886 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:03.646862 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd2442f9-1e68-4d0d-b363-42f2f6a03f07-istio-envoy" pod "fd2442f9-1e68-4d0d-b363-42f2f6a03f07" (UID: "fd2442f9-1e68-4d0d-b363-42f2f6a03f07"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:46:03.646948 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:03.646871 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fd2442f9-1e68-4d0d-b363-42f2f6a03f07-istio-podinfo" pod "fd2442f9-1e68-4d0d-b363-42f2f6a03f07" (UID: "fd2442f9-1e68-4d0d-b363-42f2f6a03f07"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 13:46:03.698182 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:03.698149 5024 generic.go:356] "Generic (PLEG): container finished" podID="fd2442f9-1e68-4d0d-b363-42f2f6a03f07" containerID="a80377780a05717214347004f3699ecbbae085d484755ef7e8087c9a859e5b2a" exitCode=0 Nov 03 13:46:03.698360 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:03.698195 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-k2fbp" Nov 03 13:46:03.698360 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:03.698233 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-k2fbp" event={"ID":"fd2442f9-1e68-4d0d-b363-42f2f6a03f07","Type":"ContainerDied","Data":"a80377780a05717214347004f3699ecbbae085d484755ef7e8087c9a859e5b2a"} Nov 03 13:46:03.698360 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:03.698308 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-k2fbp" event={"ID":"fd2442f9-1e68-4d0d-b363-42f2f6a03f07","Type":"ContainerDied","Data":"17bbcc3dce28a3572b44a5bd2c82f97ac7e3ec60c09279432b1379e6ecce76aa"} Nov 03 13:46:03.698360 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:03.698332 5024 scope.go:122] "RemoveContainer" containerID="a80377780a05717214347004f3699ecbbae085d484755ef7e8087c9a859e5b2a" Nov 03 13:46:03.708420 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:03.708401 5024 scope.go:122] "RemoveContainer" containerID="a80377780a05717214347004f3699ecbbae085d484755ef7e8087c9a859e5b2a" Nov 03 13:46:03.708649 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:46:03.708628 5024 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a80377780a05717214347004f3699ecbbae085d484755ef7e8087c9a859e5b2a\": container with ID starting with a80377780a05717214347004f3699ecbbae085d484755ef7e8087c9a859e5b2a not found: ID does not exist" containerID="a80377780a05717214347004f3699ecbbae085d484755ef7e8087c9a859e5b2a" Nov 03 13:46:03.708698 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:03.708657 5024 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a80377780a05717214347004f3699ecbbae085d484755ef7e8087c9a859e5b2a"} err="failed to get container status \"a80377780a05717214347004f3699ecbbae085d484755ef7e8087c9a859e5b2a\": rpc error: code = NotFound desc = could not find container \"a80377780a05717214347004f3699ecbbae085d484755ef7e8087c9a859e5b2a\": container with ID starting with a80377780a05717214347004f3699ecbbae085d484755ef7e8087c9a859e5b2a not found: ID does not exist" Nov 03 13:46:03.723947 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:03.723921 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-k2fbp"] Nov 03 13:46:03.727884 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:03.727862 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-7c192b-openshift-default-5c496b978f-k2fbp"] Nov 03 13:46:03.745915 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:03.745894 5024 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd2442f9-1e68-4d0d-b363-42f2f6a03f07-workload-certs\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:46:03.746024 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:03.745918 5024 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd2442f9-1e68-4d0d-b363-42f2f6a03f07-istiod-ca-cert\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:46:03.746024 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:03.745933 5024 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd2442f9-1e68-4d0d-b363-42f2f6a03f07-credential-socket\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:46:03.746024 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:03.745948 5024 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd2442f9-1e68-4d0d-b363-42f2f6a03f07-istio-envoy\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:46:03.746024 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:03.745961 5024 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd2442f9-1e68-4d0d-b363-42f2f6a03f07-istio-token\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:46:03.746024 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:03.745974 5024 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd2442f9-1e68-4d0d-b363-42f2f6a03f07-istio-podinfo\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:46:03.746024 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:03.745988 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p9mgw\" (UniqueName: \"kubernetes.io/projected/fd2442f9-1e68-4d0d-b363-42f2f6a03f07-kube-api-access-p9mgw\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:46:04.106688 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:04.106654 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd2442f9-1e68-4d0d-b363-42f2f6a03f07" path="/var/lib/kubelet/pods/fd2442f9-1e68-4d0d-b363-42f2f6a03f07/volumes" Nov 03 13:46:09.465192 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:09.465150 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-bc865c477-mhwxn"] Nov 03 13:46:09.465777 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:09.465753 5024 cpu_manager.go:414] "containerMap: removing container" podUID="155ba865-dcfd-4203-82b9-5399ca58d988" containerName="istio-proxy" Nov 03 13:46:09.465870 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:09.465781 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="155ba865-dcfd-4203-82b9-5399ca58d988" containerName="istio-proxy" Nov 03 13:46:09.465870 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:09.465800 5024 cpu_manager.go:414] "containerMap: removing container" podUID="11a9f177-f213-4929-9a64-19065b49b3f5" containerName="maas-api" Nov 03 13:46:09.465870 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:09.465808 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11a9f177-f213-4929-9a64-19065b49b3f5" containerName="maas-api" Nov 03 13:46:09.465870 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:09.465820 5024 cpu_manager.go:414] "containerMap: removing container" podUID="cabfe6c9-07d8-420b-bb66-143190883f61" containerName="istio-proxy" Nov 03 13:46:09.465870 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:09.465827 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cabfe6c9-07d8-420b-bb66-143190883f61" containerName="istio-proxy" Nov 03 13:46:09.465870 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:09.465837 5024 cpu_manager.go:414] "containerMap: removing container" podUID="c4490dbf-a557-44df-b654-b7a47f5c0604" containerName="payload-processing" Nov 03 13:46:09.465870 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:09.465845 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4490dbf-a557-44df-b654-b7a47f5c0604" containerName="payload-processing" Nov 03 13:46:09.465870 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:09.465860 5024 cpu_manager.go:414] "containerMap: removing container" podUID="168564ee-9452-4aba-8391-d4f0bdaafa7b" containerName="istio-proxy" Nov 03 13:46:09.465870 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:09.465868 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="168564ee-9452-4aba-8391-d4f0bdaafa7b" containerName="istio-proxy" Nov 03 13:46:09.466191 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:09.465878 5024 cpu_manager.go:414] "containerMap: removing container" podUID="736b02c5-9833-4f1b-9f9a-e4c22ab23ac9" containerName="istio-proxy" Nov 03 13:46:09.466191 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:09.465883 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="736b02c5-9833-4f1b-9f9a-e4c22ab23ac9" containerName="istio-proxy" Nov 03 13:46:09.466191 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:09.465892 5024 cpu_manager.go:414] "containerMap: removing container" podUID="ca014f3a-ff92-4bb9-9a7a-8c8d3dabe486" containerName="istio-proxy" Nov 03 13:46:09.466191 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:09.465896 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca014f3a-ff92-4bb9-9a7a-8c8d3dabe486" containerName="istio-proxy" Nov 03 13:46:09.466191 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:09.465909 5024 cpu_manager.go:414] "containerMap: removing container" podUID="ca014f3a-ff92-4bb9-9a7a-8c8d3dabe486" containerName="istio-proxy" Nov 03 13:46:09.466191 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:09.465914 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca014f3a-ff92-4bb9-9a7a-8c8d3dabe486" containerName="istio-proxy" Nov 03 13:46:09.466191 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:09.465925 5024 cpu_manager.go:414] "containerMap: removing container" podUID="a4f6951d-6317-4b83-b744-723ed7e576a4" containerName="payload-processing" Nov 03 13:46:09.466191 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:09.465930 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4f6951d-6317-4b83-b744-723ed7e576a4" containerName="payload-processing" Nov 03 13:46:09.466191 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:09.465936 5024 cpu_manager.go:414] "containerMap: removing container" podUID="99af7f3f-6179-495a-a526-4cce51075588" containerName="istio-proxy" Nov 03 13:46:09.466191 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:09.465940 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99af7f3f-6179-495a-a526-4cce51075588" containerName="istio-proxy" Nov 03 13:46:09.466191 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:09.465948 5024 cpu_manager.go:414] "containerMap: removing container" podUID="f47e8fd5-9a67-45ca-a817-05b9e20654cc" containerName="payload-processing" Nov 03 13:46:09.466191 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:09.465952 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f47e8fd5-9a67-45ca-a817-05b9e20654cc" containerName="payload-processing" Nov 03 13:46:09.466191 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:09.465961 5024 cpu_manager.go:414] "containerMap: removing container" podUID="e45c569a-243f-49ab-8d74-dd08cbb6127c" containerName="main" Nov 03 13:46:09.466191 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:09.465966 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e45c569a-243f-49ab-8d74-dd08cbb6127c" containerName="main" Nov 03 13:46:09.466191 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:09.465973 5024 cpu_manager.go:414] "containerMap: removing container" podUID="736b02c5-9833-4f1b-9f9a-e4c22ab23ac9" containerName="istio-proxy" Nov 03 13:46:09.466191 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:09.465977 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="736b02c5-9833-4f1b-9f9a-e4c22ab23ac9" containerName="istio-proxy" Nov 03 13:46:09.466191 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:09.465984 5024 cpu_manager.go:414] "containerMap: removing container" podUID="7b571fd7-340c-455e-be5a-479ed17d7517" containerName="payload-pre-processing" Nov 03 13:46:09.466191 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:09.465988 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b571fd7-340c-455e-be5a-479ed17d7517" containerName="payload-pre-processing" Nov 03 13:46:09.466191 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:09.465997 5024 cpu_manager.go:414] "containerMap: removing container" podUID="afc7261b-bfde-425a-8c19-f2f1d521f50a" containerName="maas-api" Nov 03 13:46:09.466191 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:09.466001 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="afc7261b-bfde-425a-8c19-f2f1d521f50a" containerName="maas-api" Nov 03 13:46:09.466191 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:09.466006 5024 cpu_manager.go:414] "containerMap: removing container" podUID="3cd0d23e-9823-4988-ac79-0820dcc4c028" containerName="istio-proxy" Nov 03 13:46:09.466191 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:09.466010 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3cd0d23e-9823-4988-ac79-0820dcc4c028" containerName="istio-proxy" Nov 03 13:46:09.466191 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:09.466017 5024 cpu_manager.go:414] "containerMap: removing container" podUID="b46130ce-1911-475e-8ec7-5204881eb5ea" containerName="istio-proxy" Nov 03 13:46:09.466191 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:09.466021 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b46130ce-1911-475e-8ec7-5204881eb5ea" containerName="istio-proxy" Nov 03 13:46:09.466191 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:09.466028 5024 cpu_manager.go:414] "containerMap: removing container" podUID="fffd24a4-d55d-4661-a37d-c344d3343698" containerName="istio-proxy" Nov 03 13:46:09.466191 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:09.466032 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fffd24a4-d55d-4661-a37d-c344d3343698" containerName="istio-proxy" Nov 03 13:46:09.466191 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:09.466039 5024 cpu_manager.go:414] "containerMap: removing container" podUID="160d4b07-2558-4e4d-afb7-a7d509cc68ca" containerName="istio-proxy" Nov 03 13:46:09.466191 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:09.466043 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="160d4b07-2558-4e4d-afb7-a7d509cc68ca" containerName="istio-proxy" Nov 03 13:46:09.466191 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:09.466051 5024 cpu_manager.go:414] "containerMap: removing container" podUID="95aa021f-6ec9-4f53-810a-58eb1d059081" containerName="istio-proxy" Nov 03 13:46:09.466191 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:09.466055 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95aa021f-6ec9-4f53-810a-58eb1d059081" containerName="istio-proxy" Nov 03 13:46:09.466191 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:09.466062 5024 cpu_manager.go:414] "containerMap: removing container" podUID="7a30a13a-965d-4291-9acd-f109a6ead710" containerName="payload-processing" Nov 03 13:46:09.466191 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:09.466066 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a30a13a-965d-4291-9acd-f109a6ead710" containerName="payload-processing" Nov 03 13:46:09.466191 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:09.466072 5024 cpu_manager.go:414] "containerMap: removing container" podUID="937216cd-5ccd-43e1-8b64-a4d31af284d6" containerName="maas-api" Nov 03 13:46:09.466191 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:09.466076 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="937216cd-5ccd-43e1-8b64-a4d31af284d6" containerName="maas-api" Nov 03 13:46:09.466191 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:09.466082 5024 cpu_manager.go:414] "containerMap: removing container" podUID="d34e5887-ae31-43ce-8d24-9973d8ba447c" containerName="payload-processing" Nov 03 13:46:09.466191 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:09.466087 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d34e5887-ae31-43ce-8d24-9973d8ba447c" containerName="payload-processing" Nov 03 13:46:09.466191 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:09.466095 5024 cpu_manager.go:414] "containerMap: removing container" podUID="168564ee-9452-4aba-8391-d4f0bdaafa7b" containerName="istio-proxy" Nov 03 13:46:09.466191 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:09.466100 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="168564ee-9452-4aba-8391-d4f0bdaafa7b" containerName="istio-proxy" Nov 03 13:46:09.466191 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:09.466107 5024 cpu_manager.go:414] "containerMap: removing container" podUID="5e1668ec-1b0f-4600-8935-64733905b0c6" containerName="maas-api" Nov 03 13:46:09.466191 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:09.466111 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e1668ec-1b0f-4600-8935-64733905b0c6" containerName="maas-api" Nov 03 13:46:09.466191 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:09.466118 5024 cpu_manager.go:414] "containerMap: removing container" podUID="fa15c0e4-6d08-4a12-9320-687d6f47ed4e" containerName="maas-api" Nov 03 13:46:09.466191 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:09.466122 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa15c0e4-6d08-4a12-9320-687d6f47ed4e" containerName="maas-api" Nov 03 13:46:09.466191 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:09.466128 5024 cpu_manager.go:414] "containerMap: removing container" podUID="459af5d0-4e46-4ad0-b123-5642ceae6e74" containerName="payload-pre-processing" Nov 03 13:46:09.466191 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:09.466134 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="459af5d0-4e46-4ad0-b123-5642ceae6e74" containerName="payload-pre-processing" Nov 03 13:46:09.466191 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:09.466144 5024 cpu_manager.go:414] "containerMap: removing container" podUID="e8aba665-e4cc-4e67-8cdd-d09ef7542e15" containerName="main" Nov 03 13:46:09.466191 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:09.466149 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8aba665-e4cc-4e67-8cdd-d09ef7542e15" containerName="main" Nov 03 13:46:09.466191 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:09.466161 5024 cpu_manager.go:414] "containerMap: removing container" podUID="bef13398-28a1-4064-aa3b-ea0f3ebf64fd" containerName="istio-proxy" Nov 03 13:46:09.466191 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:09.466166 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bef13398-28a1-4064-aa3b-ea0f3ebf64fd" containerName="istio-proxy" Nov 03 13:46:09.466191 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:09.466172 5024 cpu_manager.go:414] "containerMap: removing container" podUID="fd2442f9-1e68-4d0d-b363-42f2f6a03f07" containerName="istio-proxy" Nov 03 13:46:09.466191 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:09.466176 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd2442f9-1e68-4d0d-b363-42f2f6a03f07" containerName="istio-proxy" Nov 03 13:46:09.466191 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:09.466184 5024 cpu_manager.go:414] "containerMap: removing container" podUID="160d4b07-2558-4e4d-afb7-a7d509cc68ca" containerName="istio-proxy" Nov 03 13:46:09.466191 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:09.466189 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="160d4b07-2558-4e4d-afb7-a7d509cc68ca" containerName="istio-proxy" Nov 03 13:46:09.466191 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:09.466195 5024 cpu_manager.go:414] "containerMap: removing container" podUID="5fabbac7-721a-4f35-bd22-e86df0cecc62" containerName="main" Nov 03 13:46:09.466191 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:09.466199 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5fabbac7-721a-4f35-bd22-e86df0cecc62" containerName="main" Nov 03 13:46:09.466191 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:09.466206 5024 cpu_manager.go:414] "containerMap: removing container" podUID="bb5f109b-305d-4eec-a3c9-b917a0bf9981" containerName="maas-api" Nov 03 13:46:09.466191 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:09.466210 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb5f109b-305d-4eec-a3c9-b917a0bf9981" containerName="maas-api" Nov 03 13:46:09.466191 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:09.466216 5024 cpu_manager.go:414] "containerMap: removing container" podUID="af43a1c8-fad2-4b80-a0e9-65ab23c3612e" containerName="istio-proxy" Nov 03 13:46:09.468226 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:09.466221 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af43a1c8-fad2-4b80-a0e9-65ab23c3612e" containerName="istio-proxy" Nov 03 13:46:09.468226 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:09.466298 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="cabfe6c9-07d8-420b-bb66-143190883f61" containerName="istio-proxy" Nov 03 13:46:09.468226 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:09.466311 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4490dbf-a557-44df-b654-b7a47f5c0604" containerName="payload-processing" Nov 03 13:46:09.468226 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:09.466317 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="168564ee-9452-4aba-8391-d4f0bdaafa7b" containerName="istio-proxy" Nov 03 13:46:09.468226 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:09.466323 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="736b02c5-9833-4f1b-9f9a-e4c22ab23ac9" containerName="istio-proxy" Nov 03 13:46:09.468226 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:09.466328 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca014f3a-ff92-4bb9-9a7a-8c8d3dabe486" containerName="istio-proxy" Nov 03 13:46:09.468226 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:09.466335 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca014f3a-ff92-4bb9-9a7a-8c8d3dabe486" containerName="istio-proxy" Nov 03 13:46:09.468226 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:09.466341 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4f6951d-6317-4b83-b744-723ed7e576a4" containerName="payload-processing" Nov 03 13:46:09.468226 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:09.466345 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="99af7f3f-6179-495a-a526-4cce51075588" containerName="istio-proxy" Nov 03 13:46:09.468226 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:09.466350 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="f47e8fd5-9a67-45ca-a817-05b9e20654cc" containerName="payload-processing" Nov 03 13:46:09.468226 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:09.466357 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="e45c569a-243f-49ab-8d74-dd08cbb6127c" containerName="main" Nov 03 13:46:09.468226 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:09.466362 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="736b02c5-9833-4f1b-9f9a-e4c22ab23ac9" containerName="istio-proxy" Nov 03 13:46:09.468226 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:09.466367 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b571fd7-340c-455e-be5a-479ed17d7517" containerName="payload-pre-processing" Nov 03 13:46:09.468226 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:09.466372 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="afc7261b-bfde-425a-8c19-f2f1d521f50a" containerName="maas-api" Nov 03 13:46:09.468226 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:09.466376 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="3cd0d23e-9823-4988-ac79-0820dcc4c028" containerName="istio-proxy" Nov 03 13:46:09.468226 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:09.466381 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="b46130ce-1911-475e-8ec7-5204881eb5ea" containerName="istio-proxy" Nov 03 13:46:09.468226 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:09.466387 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="fffd24a4-d55d-4661-a37d-c344d3343698" containerName="istio-proxy" Nov 03 13:46:09.468226 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:09.466392 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="160d4b07-2558-4e4d-afb7-a7d509cc68ca" containerName="istio-proxy" Nov 03 13:46:09.468226 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:09.466397 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="95aa021f-6ec9-4f53-810a-58eb1d059081" containerName="istio-proxy" Nov 03 13:46:09.468226 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:09.466402 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a30a13a-965d-4291-9acd-f109a6ead710" containerName="payload-processing" Nov 03 13:46:09.468226 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:09.466407 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="937216cd-5ccd-43e1-8b64-a4d31af284d6" containerName="maas-api" Nov 03 13:46:09.468226 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:09.466413 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="d34e5887-ae31-43ce-8d24-9973d8ba447c" containerName="payload-processing" Nov 03 13:46:09.468226 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:09.466418 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="168564ee-9452-4aba-8391-d4f0bdaafa7b" containerName="istio-proxy" Nov 03 13:46:09.468226 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:09.466424 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e1668ec-1b0f-4600-8935-64733905b0c6" containerName="maas-api" Nov 03 13:46:09.468226 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:09.466430 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa15c0e4-6d08-4a12-9320-687d6f47ed4e" containerName="maas-api" Nov 03 13:46:09.468226 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:09.466435 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="459af5d0-4e46-4ad0-b123-5642ceae6e74" containerName="payload-pre-processing" Nov 03 13:46:09.468226 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:09.466442 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8aba665-e4cc-4e67-8cdd-d09ef7542e15" containerName="main" Nov 03 13:46:09.468226 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:09.466448 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="bef13398-28a1-4064-aa3b-ea0f3ebf64fd" containerName="istio-proxy" Nov 03 13:46:09.468226 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:09.466453 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd2442f9-1e68-4d0d-b363-42f2f6a03f07" containerName="istio-proxy" Nov 03 13:46:09.468226 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:09.466459 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="160d4b07-2558-4e4d-afb7-a7d509cc68ca" containerName="istio-proxy" Nov 03 13:46:09.468226 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:09.466465 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="5fabbac7-721a-4f35-bd22-e86df0cecc62" containerName="main" Nov 03 13:46:09.468226 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:09.466470 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb5f109b-305d-4eec-a3c9-b917a0bf9981" containerName="maas-api" Nov 03 13:46:09.468226 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:09.466475 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="af43a1c8-fad2-4b80-a0e9-65ab23c3612e" containerName="istio-proxy" Nov 03 13:46:09.468226 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:09.466481 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="155ba865-dcfd-4203-82b9-5399ca58d988" containerName="istio-proxy" Nov 03 13:46:09.468226 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:09.466487 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="11a9f177-f213-4929-9a64-19065b49b3f5" containerName="maas-api" Nov 03 13:46:09.470338 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:09.470319 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-bc865c477-mhwxn" Nov 03 13:46:09.483394 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:09.483363 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-bc865c477-mhwxn"] Nov 03 13:46:09.483541 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:09.483529 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Nov 03 13:46:09.483625 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:09.483611 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Nov 03 13:46:09.483723 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:09.483708 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Nov 03 13:46:09.483843 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:09.483789 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-qn5lb\"" Nov 03 13:46:09.511137 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:46:09.511093 5024 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d7mwz" containerName="extractor" Nov 03 13:46:09.599006 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:09.598958 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/95d50d1a-1756-4b07-a016-a4c2cbd23a4d-webhook-cert\") pod \"maas-controller-bc865c477-mhwxn\" (UID: \"95d50d1a-1756-4b07-a016-a4c2cbd23a4d\") " pod="opendatahub/maas-controller-bc865c477-mhwxn" Nov 03 13:46:09.599222 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:09.599014 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/95d50d1a-1756-4b07-a016-a4c2cbd23a4d-metrics-certs\") pod \"maas-controller-bc865c477-mhwxn\" (UID: \"95d50d1a-1756-4b07-a016-a4c2cbd23a4d\") " pod="opendatahub/maas-controller-bc865c477-mhwxn" Nov 03 13:46:09.599222 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:09.599168 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gv656\" (UniqueName: \"kubernetes.io/projected/95d50d1a-1756-4b07-a016-a4c2cbd23a4d-kube-api-access-gv656\") pod \"maas-controller-bc865c477-mhwxn\" (UID: \"95d50d1a-1756-4b07-a016-a4c2cbd23a4d\") " pod="opendatahub/maas-controller-bc865c477-mhwxn" Nov 03 13:46:09.700741 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:09.700700 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gv656\" (UniqueName: \"kubernetes.io/projected/95d50d1a-1756-4b07-a016-a4c2cbd23a4d-kube-api-access-gv656\") pod \"maas-controller-bc865c477-mhwxn\" (UID: \"95d50d1a-1756-4b07-a016-a4c2cbd23a4d\") " pod="opendatahub/maas-controller-bc865c477-mhwxn" Nov 03 13:46:09.700918 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:09.700778 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/95d50d1a-1756-4b07-a016-a4c2cbd23a4d-webhook-cert\") pod \"maas-controller-bc865c477-mhwxn\" (UID: \"95d50d1a-1756-4b07-a016-a4c2cbd23a4d\") " pod="opendatahub/maas-controller-bc865c477-mhwxn" Nov 03 13:46:09.700918 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:09.700808 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/95d50d1a-1756-4b07-a016-a4c2cbd23a4d-metrics-certs\") pod \"maas-controller-bc865c477-mhwxn\" (UID: \"95d50d1a-1756-4b07-a016-a4c2cbd23a4d\") " pod="opendatahub/maas-controller-bc865c477-mhwxn" Nov 03 13:46:09.703400 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:09.703362 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/95d50d1a-1756-4b07-a016-a4c2cbd23a4d-metrics-certs\") pod \"maas-controller-bc865c477-mhwxn\" (UID: \"95d50d1a-1756-4b07-a016-a4c2cbd23a4d\") " pod="opendatahub/maas-controller-bc865c477-mhwxn" Nov 03 13:46:09.703560 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:09.703480 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/95d50d1a-1756-4b07-a016-a4c2cbd23a4d-webhook-cert\") pod \"maas-controller-bc865c477-mhwxn\" (UID: \"95d50d1a-1756-4b07-a016-a4c2cbd23a4d\") " pod="opendatahub/maas-controller-bc865c477-mhwxn" Nov 03 13:46:09.711757 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:09.711726 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gv656\" (UniqueName: \"kubernetes.io/projected/95d50d1a-1756-4b07-a016-a4c2cbd23a4d-kube-api-access-gv656\") pod \"maas-controller-bc865c477-mhwxn\" (UID: \"95d50d1a-1756-4b07-a016-a4c2cbd23a4d\") " pod="opendatahub/maas-controller-bc865c477-mhwxn" Nov 03 13:46:09.787430 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:09.787343 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-bc865c477-mhwxn" Nov 03 13:46:10.071657 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:10.071628 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-bc865c477-mhwxn"] Nov 03 13:46:10.073718 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:46:10.073676 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95d50d1a_1756_4b07_a016_a4c2cbd23a4d.slice/crio-b6cc476696064f201ecc1e5b2b60a4ebbd03e295d30e596a72e5a092bd80a969 WatchSource:0}: Error finding container b6cc476696064f201ecc1e5b2b60a4ebbd03e295d30e596a72e5a092bd80a969: Status 404 returned error can't find the container with id b6cc476696064f201ecc1e5b2b60a4ebbd03e295d30e596a72e5a092bd80a969 Nov 03 13:46:10.075229 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:10.075208 5024 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 13:46:10.738795 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:10.738744 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-bc865c477-mhwxn" event={"ID":"95d50d1a-1756-4b07-a016-a4c2cbd23a4d","Type":"ContainerStarted","Data":"b6cc476696064f201ecc1e5b2b60a4ebbd03e295d30e596a72e5a092bd80a969"} Nov 03 13:46:12.750756 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:12.750713 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-bc865c477-mhwxn" event={"ID":"95d50d1a-1756-4b07-a016-a4c2cbd23a4d","Type":"ContainerStarted","Data":"6b7774c1aeb8429d5e974a29950f699c15451427d0f6c8523f5ee999411401fd"} Nov 03 13:46:12.751153 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:12.750837 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-bc865c477-mhwxn" Nov 03 13:46:12.774043 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:12.773991 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-bc865c477-mhwxn" podStartSLOduration=2.007854351 podStartE2EDuration="3.773977755s" totalImagesPullingTime="1.766123404s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:46:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:46:12.772440971 +0000 UTC m=+2252.780632249" watchObservedRunningTime="2026-11-03 13:46:12.773977755 +0000 UTC m=+2252.782169057" Nov 03 13:46:20.108870 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:20.108831 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b9tb7"] Nov 03 13:46:20.109442 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:20.108997 5024 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b9tb7" podUID="22e78255-1b4a-430e-ae67-617d02d15b51" containerName="discovery" containerID="cri-o://50b79ccfe3e11fdd16b41b0b2bfcd27c750965754deb3e9370af345870d8720e" gracePeriod=30 Nov 03 13:46:20.363211 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:20.363148 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b9tb7" Nov 03 13:46:20.517745 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:20.517707 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/22e78255-1b4a-430e-ae67-617d02d15b51-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/22e78255-1b4a-430e-ae67-617d02d15b51-istio-csr-dns-cert\") pod \"22e78255-1b4a-430e-ae67-617d02d15b51\" (UID: \"22e78255-1b4a-430e-ae67-617d02d15b51\") " Nov 03 13:46:20.517745 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:20.517749 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/22e78255-1b4a-430e-ae67-617d02d15b51-cacerts\" (UniqueName: \"kubernetes.io/secret/22e78255-1b4a-430e-ae67-617d02d15b51-cacerts\") pod \"22e78255-1b4a-430e-ae67-617d02d15b51\" (UID: \"22e78255-1b4a-430e-ae67-617d02d15b51\") " Nov 03 13:46:20.517964 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:20.517791 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22e78255-1b4a-430e-ae67-617d02d15b51-istio-token\" (UniqueName: \"kubernetes.io/projected/22e78255-1b4a-430e-ae67-617d02d15b51-istio-token\") pod \"22e78255-1b4a-430e-ae67-617d02d15b51\" (UID: \"22e78255-1b4a-430e-ae67-617d02d15b51\") " Nov 03 13:46:20.517964 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:20.517829 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22e78255-1b4a-430e-ae67-617d02d15b51-kube-api-access-zfsn9\" (UniqueName: \"kubernetes.io/projected/22e78255-1b4a-430e-ae67-617d02d15b51-kube-api-access-zfsn9\") pod \"22e78255-1b4a-430e-ae67-617d02d15b51\" (UID: \"22e78255-1b4a-430e-ae67-617d02d15b51\") " Nov 03 13:46:20.517964 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:20.517869 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/22e78255-1b4a-430e-ae67-617d02d15b51-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/22e78255-1b4a-430e-ae67-617d02d15b51-istio-kubeconfig\") pod \"22e78255-1b4a-430e-ae67-617d02d15b51\" (UID: \"22e78255-1b4a-430e-ae67-617d02d15b51\") " Nov 03 13:46:20.517964 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:20.517956 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22e78255-1b4a-430e-ae67-617d02d15b51-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/22e78255-1b4a-430e-ae67-617d02d15b51-local-certs\") pod \"22e78255-1b4a-430e-ae67-617d02d15b51\" (UID: \"22e78255-1b4a-430e-ae67-617d02d15b51\") " Nov 03 13:46:20.518193 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:20.518005 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/22e78255-1b4a-430e-ae67-617d02d15b51-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/22e78255-1b4a-430e-ae67-617d02d15b51-istio-csr-ca-configmap\") pod \"22e78255-1b4a-430e-ae67-617d02d15b51\" (UID: \"22e78255-1b4a-430e-ae67-617d02d15b51\") " Nov 03 13:46:20.518735 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:20.518688 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22e78255-1b4a-430e-ae67-617d02d15b51-istio-csr-ca-configmap" pod "22e78255-1b4a-430e-ae67-617d02d15b51" (UID: "22e78255-1b4a-430e-ae67-617d02d15b51"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:46:20.520346 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:20.520279 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/22e78255-1b4a-430e-ae67-617d02d15b51-istio-kubeconfig" pod "22e78255-1b4a-430e-ae67-617d02d15b51" (UID: "22e78255-1b4a-430e-ae67-617d02d15b51"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 13:46:20.520462 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:20.520431 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22e78255-1b4a-430e-ae67-617d02d15b51-istio-token" pod "22e78255-1b4a-430e-ae67-617d02d15b51" (UID: "22e78255-1b4a-430e-ae67-617d02d15b51"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:46:20.520658 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:20.520624 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22e78255-1b4a-430e-ae67-617d02d15b51-local-certs" pod "22e78255-1b4a-430e-ae67-617d02d15b51" (UID: "22e78255-1b4a-430e-ae67-617d02d15b51"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:46:20.520766 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:20.520665 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22e78255-1b4a-430e-ae67-617d02d15b51-kube-api-access-zfsn9" pod "22e78255-1b4a-430e-ae67-617d02d15b51" (UID: "22e78255-1b4a-430e-ae67-617d02d15b51"). InnerVolumeSpecName "kube-api-access-zfsn9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:46:20.520766 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:20.520703 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/22e78255-1b4a-430e-ae67-617d02d15b51-cacerts" pod "22e78255-1b4a-430e-ae67-617d02d15b51" (UID: "22e78255-1b4a-430e-ae67-617d02d15b51"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 13:46:20.520766 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:20.520716 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/22e78255-1b4a-430e-ae67-617d02d15b51-istio-csr-dns-cert" pod "22e78255-1b4a-430e-ae67-617d02d15b51" (UID: "22e78255-1b4a-430e-ae67-617d02d15b51"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 13:46:20.619353 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:20.619244 5024 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/22e78255-1b4a-430e-ae67-617d02d15b51-istio-kubeconfig\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:46:20.619353 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:20.619298 5024 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/22e78255-1b4a-430e-ae67-617d02d15b51-local-certs\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:46:20.619353 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:20.619311 5024 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/22e78255-1b4a-430e-ae67-617d02d15b51-istio-csr-ca-configmap\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:46:20.619353 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:20.619328 5024 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/22e78255-1b4a-430e-ae67-617d02d15b51-istio-csr-dns-cert\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:46:20.619353 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:20.619341 5024 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/22e78255-1b4a-430e-ae67-617d02d15b51-cacerts\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:46:20.619353 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:20.619353 5024 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22e78255-1b4a-430e-ae67-617d02d15b51-istio-token\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:46:20.619663 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:20.619363 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zfsn9\" (UniqueName: \"kubernetes.io/projected/22e78255-1b4a-430e-ae67-617d02d15b51-kube-api-access-zfsn9\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:46:20.796030 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:20.795995 5024 generic.go:356] "Generic (PLEG): container finished" podID="22e78255-1b4a-430e-ae67-617d02d15b51" containerID="50b79ccfe3e11fdd16b41b0b2bfcd27c750965754deb3e9370af345870d8720e" exitCode=0 Nov 03 13:46:20.796241 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:20.796051 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b9tb7" Nov 03 13:46:20.796241 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:20.796082 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b9tb7" event={"ID":"22e78255-1b4a-430e-ae67-617d02d15b51","Type":"ContainerDied","Data":"50b79ccfe3e11fdd16b41b0b2bfcd27c750965754deb3e9370af345870d8720e"} Nov 03 13:46:20.796241 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:20.796134 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b9tb7" event={"ID":"22e78255-1b4a-430e-ae67-617d02d15b51","Type":"ContainerDied","Data":"8eaa933fd5d93058e1c4e406761d096c8771ef17c2c0fc5e84943032b0f303e4"} Nov 03 13:46:20.796241 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:20.796154 5024 scope.go:122] "RemoveContainer" containerID="50b79ccfe3e11fdd16b41b0b2bfcd27c750965754deb3e9370af345870d8720e" Nov 03 13:46:20.813059 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:20.813038 5024 scope.go:122] "RemoveContainer" containerID="50b79ccfe3e11fdd16b41b0b2bfcd27c750965754deb3e9370af345870d8720e" Nov 03 13:46:20.813406 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:46:20.813384 5024 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"50b79ccfe3e11fdd16b41b0b2bfcd27c750965754deb3e9370af345870d8720e\": container with ID starting with 50b79ccfe3e11fdd16b41b0b2bfcd27c750965754deb3e9370af345870d8720e not found: ID does not exist" containerID="50b79ccfe3e11fdd16b41b0b2bfcd27c750965754deb3e9370af345870d8720e" Nov 03 13:46:20.813470 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:20.813416 5024 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50b79ccfe3e11fdd16b41b0b2bfcd27c750965754deb3e9370af345870d8720e"} err="failed to get container status \"50b79ccfe3e11fdd16b41b0b2bfcd27c750965754deb3e9370af345870d8720e\": rpc error: code = NotFound desc = could not find container \"50b79ccfe3e11fdd16b41b0b2bfcd27c750965754deb3e9370af345870d8720e\": container with ID starting with 50b79ccfe3e11fdd16b41b0b2bfcd27c750965754deb3e9370af345870d8720e not found: ID does not exist" Nov 03 13:46:20.820520 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:20.820491 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b9tb7"] Nov 03 13:46:20.828056 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:20.828030 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b9tb7"] Nov 03 13:46:22.107355 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:22.107316 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="22e78255-1b4a-430e-ae67-617d02d15b51" path="/var/lib/kubelet/pods/22e78255-1b4a-430e-ae67-617d02d15b51/volumes" Nov 03 13:46:23.758171 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:23.758138 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-bc865c477-mhwxn" Nov 03 13:46:27.735792 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:27.735748 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-84587756dc-zndxf"] Nov 03 13:46:27.736211 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:27.736181 5024 cpu_manager.go:414] "containerMap: removing container" podUID="22e78255-1b4a-430e-ae67-617d02d15b51" containerName="discovery" Nov 03 13:46:27.736211 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:27.736195 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22e78255-1b4a-430e-ae67-617d02d15b51" containerName="discovery" Nov 03 13:46:27.736321 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:27.736310 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="22e78255-1b4a-430e-ae67-617d02d15b51" containerName="discovery" Nov 03 13:46:27.739516 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:27.739497 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-zndxf" Nov 03 13:46:27.750865 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:27.750840 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-84587756dc-92vh9"] Nov 03 13:46:27.754408 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:27.754379 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-84587756dc-k4c4j"] Nov 03 13:46:27.754542 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:27.754517 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-92vh9" Nov 03 13:46:27.757728 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:27.757712 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-84587756dc-zndxf"] Nov 03 13:46:27.757811 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:27.757784 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-k4c4j" Nov 03 13:46:27.768664 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:27.768640 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-84587756dc-92vh9"] Nov 03 13:46:27.770767 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:27.770743 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-84587756dc-k4c4j"] Nov 03 13:46:27.892247 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:27.892209 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73f8f24f-b955-433b-95ed-5f441f7fceb2-credential-socket\") pod \"maas-default-gateway-openshift-default-84587756dc-zndxf\" (UID: \"73f8f24f-b955-433b-95ed-5f441f7fceb2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-zndxf" Nov 03 13:46:27.892247 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:27.892246 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/825941b6-3ed3-47f2-a859-c891c4f29434-credential-socket\") pod \"maas-default-gateway-openshift-default-84587756dc-92vh9\" (UID: \"825941b6-3ed3-47f2-a859-c891c4f29434\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-92vh9" Nov 03 13:46:27.892492 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:27.892265 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/825941b6-3ed3-47f2-a859-c891c4f29434-workload-certs\") pod \"maas-default-gateway-openshift-default-84587756dc-92vh9\" (UID: \"825941b6-3ed3-47f2-a859-c891c4f29434\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-92vh9" Nov 03 13:46:27.892492 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:27.892306 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b70e366-6fba-4bfa-ac3e-0f0c7b0b64d3-istio-podinfo\") pod \"maas-default-gateway-openshift-default-84587756dc-k4c4j\" (UID: \"4b70e366-6fba-4bfa-ac3e-0f0c7b0b64d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-k4c4j" Nov 03 13:46:27.892492 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:27.892357 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b70e366-6fba-4bfa-ac3e-0f0c7b0b64d3-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-84587756dc-k4c4j\" (UID: \"4b70e366-6fba-4bfa-ac3e-0f0c7b0b64d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-k4c4j" Nov 03 13:46:27.892492 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:27.892440 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b70e366-6fba-4bfa-ac3e-0f0c7b0b64d3-workload-socket\") pod \"maas-default-gateway-openshift-default-84587756dc-k4c4j\" (UID: \"4b70e366-6fba-4bfa-ac3e-0f0c7b0b64d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-k4c4j" Nov 03 13:46:27.892492 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:27.892475 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/825941b6-3ed3-47f2-a859-c891c4f29434-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-84587756dc-92vh9\" (UID: \"825941b6-3ed3-47f2-a859-c891c4f29434\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-92vh9" Nov 03 13:46:27.892492 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:27.892494 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73f8f24f-b955-433b-95ed-5f441f7fceb2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-84587756dc-zndxf\" (UID: \"73f8f24f-b955-433b-95ed-5f441f7fceb2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-zndxf" Nov 03 13:46:27.892702 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:27.892547 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/825941b6-3ed3-47f2-a859-c891c4f29434-istio-envoy\") pod \"maas-default-gateway-openshift-default-84587756dc-92vh9\" (UID: \"825941b6-3ed3-47f2-a859-c891c4f29434\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-92vh9" Nov 03 13:46:27.892702 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:27.892570 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73f8f24f-b955-433b-95ed-5f441f7fceb2-istio-envoy\") pod \"maas-default-gateway-openshift-default-84587756dc-zndxf\" (UID: \"73f8f24f-b955-433b-95ed-5f441f7fceb2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-zndxf" Nov 03 13:46:27.892702 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:27.892588 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/825941b6-3ed3-47f2-a859-c891c4f29434-istio-token\") pod \"maas-default-gateway-openshift-default-84587756dc-92vh9\" (UID: \"825941b6-3ed3-47f2-a859-c891c4f29434\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-92vh9" Nov 03 13:46:27.892702 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:27.892607 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b70e366-6fba-4bfa-ac3e-0f0c7b0b64d3-istio-envoy\") pod \"maas-default-gateway-openshift-default-84587756dc-k4c4j\" (UID: \"4b70e366-6fba-4bfa-ac3e-0f0c7b0b64d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-k4c4j" Nov 03 13:46:27.892702 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:27.892630 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73f8f24f-b955-433b-95ed-5f441f7fceb2-istio-data\") pod \"maas-default-gateway-openshift-default-84587756dc-zndxf\" (UID: \"73f8f24f-b955-433b-95ed-5f441f7fceb2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-zndxf" Nov 03 13:46:27.892702 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:27.892656 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hhmxm\" (UniqueName: \"kubernetes.io/projected/825941b6-3ed3-47f2-a859-c891c4f29434-kube-api-access-hhmxm\") pod \"maas-default-gateway-openshift-default-84587756dc-92vh9\" (UID: \"825941b6-3ed3-47f2-a859-c891c4f29434\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-92vh9" Nov 03 13:46:27.892898 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:27.892705 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/825941b6-3ed3-47f2-a859-c891c4f29434-istio-podinfo\") pod \"maas-default-gateway-openshift-default-84587756dc-92vh9\" (UID: \"825941b6-3ed3-47f2-a859-c891c4f29434\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-92vh9" Nov 03 13:46:27.892898 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:27.892766 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73f8f24f-b955-433b-95ed-5f441f7fceb2-workload-socket\") pod \"maas-default-gateway-openshift-default-84587756dc-zndxf\" (UID: \"73f8f24f-b955-433b-95ed-5f441f7fceb2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-zndxf" Nov 03 13:46:27.892898 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:27.892795 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b70e366-6fba-4bfa-ac3e-0f0c7b0b64d3-istio-token\") pod \"maas-default-gateway-openshift-default-84587756dc-k4c4j\" (UID: \"4b70e366-6fba-4bfa-ac3e-0f0c7b0b64d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-k4c4j" Nov 03 13:46:27.892898 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:27.892811 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73f8f24f-b955-433b-95ed-5f441f7fceb2-workload-certs\") pod \"maas-default-gateway-openshift-default-84587756dc-zndxf\" (UID: \"73f8f24f-b955-433b-95ed-5f441f7fceb2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-zndxf" Nov 03 13:46:27.892898 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:27.892828 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kb5h2\" (UniqueName: \"kubernetes.io/projected/73f8f24f-b955-433b-95ed-5f441f7fceb2-kube-api-access-kb5h2\") pod \"maas-default-gateway-openshift-default-84587756dc-zndxf\" (UID: \"73f8f24f-b955-433b-95ed-5f441f7fceb2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-zndxf" Nov 03 13:46:27.892898 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:27.892858 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/825941b6-3ed3-47f2-a859-c891c4f29434-workload-socket\") pod \"maas-default-gateway-openshift-default-84587756dc-92vh9\" (UID: \"825941b6-3ed3-47f2-a859-c891c4f29434\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-92vh9" Nov 03 13:46:27.893085 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:27.892901 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b70e366-6fba-4bfa-ac3e-0f0c7b0b64d3-credential-socket\") pod \"maas-default-gateway-openshift-default-84587756dc-k4c4j\" (UID: \"4b70e366-6fba-4bfa-ac3e-0f0c7b0b64d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-k4c4j" Nov 03 13:46:27.893085 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:27.892966 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/825941b6-3ed3-47f2-a859-c891c4f29434-istio-data\") pod \"maas-default-gateway-openshift-default-84587756dc-92vh9\" (UID: \"825941b6-3ed3-47f2-a859-c891c4f29434\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-92vh9" Nov 03 13:46:27.893085 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:27.892985 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b70e366-6fba-4bfa-ac3e-0f0c7b0b64d3-workload-certs\") pod \"maas-default-gateway-openshift-default-84587756dc-k4c4j\" (UID: \"4b70e366-6fba-4bfa-ac3e-0f0c7b0b64d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-k4c4j" Nov 03 13:46:27.893085 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:27.893009 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73f8f24f-b955-433b-95ed-5f441f7fceb2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-84587756dc-zndxf\" (UID: \"73f8f24f-b955-433b-95ed-5f441f7fceb2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-zndxf" Nov 03 13:46:27.893085 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:27.893030 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/73f8f24f-b955-433b-95ed-5f441f7fceb2-istio-token\") pod \"maas-default-gateway-openshift-default-84587756dc-zndxf\" (UID: \"73f8f24f-b955-433b-95ed-5f441f7fceb2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-zndxf" Nov 03 13:46:27.893085 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:27.893082 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvqlg\" (UniqueName: \"kubernetes.io/projected/4b70e366-6fba-4bfa-ac3e-0f0c7b0b64d3-kube-api-access-zvqlg\") pod \"maas-default-gateway-openshift-default-84587756dc-k4c4j\" (UID: \"4b70e366-6fba-4bfa-ac3e-0f0c7b0b64d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-k4c4j" Nov 03 13:46:27.893271 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:27.893136 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b70e366-6fba-4bfa-ac3e-0f0c7b0b64d3-istio-data\") pod \"maas-default-gateway-openshift-default-84587756dc-k4c4j\" (UID: \"4b70e366-6fba-4bfa-ac3e-0f0c7b0b64d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-k4c4j" Nov 03 13:46:27.994137 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:27.994060 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73f8f24f-b955-433b-95ed-5f441f7fceb2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-84587756dc-zndxf\" (UID: \"73f8f24f-b955-433b-95ed-5f441f7fceb2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-zndxf" Nov 03 13:46:27.994137 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:27.994107 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/825941b6-3ed3-47f2-a859-c891c4f29434-istio-envoy\") pod \"maas-default-gateway-openshift-default-84587756dc-92vh9\" (UID: \"825941b6-3ed3-47f2-a859-c891c4f29434\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-92vh9" Nov 03 13:46:27.994137 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:27.994136 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73f8f24f-b955-433b-95ed-5f441f7fceb2-istio-envoy\") pod \"maas-default-gateway-openshift-default-84587756dc-zndxf\" (UID: \"73f8f24f-b955-433b-95ed-5f441f7fceb2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-zndxf" Nov 03 13:46:27.994496 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:27.994161 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/825941b6-3ed3-47f2-a859-c891c4f29434-istio-token\") pod \"maas-default-gateway-openshift-default-84587756dc-92vh9\" (UID: \"825941b6-3ed3-47f2-a859-c891c4f29434\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-92vh9" Nov 03 13:46:27.994496 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:27.994189 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b70e366-6fba-4bfa-ac3e-0f0c7b0b64d3-istio-envoy\") pod \"maas-default-gateway-openshift-default-84587756dc-k4c4j\" (UID: \"4b70e366-6fba-4bfa-ac3e-0f0c7b0b64d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-k4c4j" Nov 03 13:46:27.994496 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:27.994212 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73f8f24f-b955-433b-95ed-5f441f7fceb2-istio-data\") pod \"maas-default-gateway-openshift-default-84587756dc-zndxf\" (UID: \"73f8f24f-b955-433b-95ed-5f441f7fceb2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-zndxf" Nov 03 13:46:27.994496 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:27.994234 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhmxm\" (UniqueName: \"kubernetes.io/projected/825941b6-3ed3-47f2-a859-c891c4f29434-kube-api-access-hhmxm\") pod \"maas-default-gateway-openshift-default-84587756dc-92vh9\" (UID: \"825941b6-3ed3-47f2-a859-c891c4f29434\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-92vh9" Nov 03 13:46:27.994496 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:27.994435 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/825941b6-3ed3-47f2-a859-c891c4f29434-istio-podinfo\") pod \"maas-default-gateway-openshift-default-84587756dc-92vh9\" (UID: \"825941b6-3ed3-47f2-a859-c891c4f29434\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-92vh9" Nov 03 13:46:27.994496 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:27.994486 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73f8f24f-b955-433b-95ed-5f441f7fceb2-workload-socket\") pod \"maas-default-gateway-openshift-default-84587756dc-zndxf\" (UID: \"73f8f24f-b955-433b-95ed-5f441f7fceb2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-zndxf" Nov 03 13:46:27.994840 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:27.994528 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b70e366-6fba-4bfa-ac3e-0f0c7b0b64d3-istio-token\") pod \"maas-default-gateway-openshift-default-84587756dc-k4c4j\" (UID: \"4b70e366-6fba-4bfa-ac3e-0f0c7b0b64d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-k4c4j" Nov 03 13:46:27.994840 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:27.994565 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73f8f24f-b955-433b-95ed-5f441f7fceb2-workload-certs\") pod \"maas-default-gateway-openshift-default-84587756dc-zndxf\" (UID: \"73f8f24f-b955-433b-95ed-5f441f7fceb2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-zndxf" Nov 03 13:46:27.994840 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:27.994594 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kb5h2\" (UniqueName: \"kubernetes.io/projected/73f8f24f-b955-433b-95ed-5f441f7fceb2-kube-api-access-kb5h2\") pod \"maas-default-gateway-openshift-default-84587756dc-zndxf\" (UID: \"73f8f24f-b955-433b-95ed-5f441f7fceb2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-zndxf" Nov 03 13:46:27.994840 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:27.994620 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/825941b6-3ed3-47f2-a859-c891c4f29434-workload-socket\") pod \"maas-default-gateway-openshift-default-84587756dc-92vh9\" (UID: \"825941b6-3ed3-47f2-a859-c891c4f29434\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-92vh9" Nov 03 13:46:27.994840 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:27.994651 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b70e366-6fba-4bfa-ac3e-0f0c7b0b64d3-credential-socket\") pod \"maas-default-gateway-openshift-default-84587756dc-k4c4j\" (UID: \"4b70e366-6fba-4bfa-ac3e-0f0c7b0b64d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-k4c4j" Nov 03 13:46:27.994840 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:27.994741 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73f8f24f-b955-433b-95ed-5f441f7fceb2-istio-data\") pod \"maas-default-gateway-openshift-default-84587756dc-zndxf\" (UID: \"73f8f24f-b955-433b-95ed-5f441f7fceb2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-zndxf" Nov 03 13:46:27.995161 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:27.994976 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b70e366-6fba-4bfa-ac3e-0f0c7b0b64d3-credential-socket\") pod \"maas-default-gateway-openshift-default-84587756dc-k4c4j\" (UID: \"4b70e366-6fba-4bfa-ac3e-0f0c7b0b64d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-k4c4j" Nov 03 13:46:27.995161 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:27.995018 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73f8f24f-b955-433b-95ed-5f441f7fceb2-workload-socket\") pod \"maas-default-gateway-openshift-default-84587756dc-zndxf\" (UID: \"73f8f24f-b955-433b-95ed-5f441f7fceb2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-zndxf" Nov 03 13:46:27.995302 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:27.995189 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/825941b6-3ed3-47f2-a859-c891c4f29434-istio-data\") pod \"maas-default-gateway-openshift-default-84587756dc-92vh9\" (UID: \"825941b6-3ed3-47f2-a859-c891c4f29434\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-92vh9" Nov 03 13:46:27.995302 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:27.995229 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b70e366-6fba-4bfa-ac3e-0f0c7b0b64d3-workload-certs\") pod \"maas-default-gateway-openshift-default-84587756dc-k4c4j\" (UID: \"4b70e366-6fba-4bfa-ac3e-0f0c7b0b64d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-k4c4j" Nov 03 13:46:27.995302 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:27.995266 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/825941b6-3ed3-47f2-a859-c891c4f29434-workload-socket\") pod \"maas-default-gateway-openshift-default-84587756dc-92vh9\" (UID: \"825941b6-3ed3-47f2-a859-c891c4f29434\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-92vh9" Nov 03 13:46:27.995302 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:27.995272 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73f8f24f-b955-433b-95ed-5f441f7fceb2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-84587756dc-zndxf\" (UID: \"73f8f24f-b955-433b-95ed-5f441f7fceb2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-zndxf" Nov 03 13:46:27.995542 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:27.995317 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73f8f24f-b955-433b-95ed-5f441f7fceb2-workload-certs\") pod \"maas-default-gateway-openshift-default-84587756dc-zndxf\" (UID: \"73f8f24f-b955-433b-95ed-5f441f7fceb2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-zndxf" Nov 03 13:46:27.995542 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:27.995336 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/73f8f24f-b955-433b-95ed-5f441f7fceb2-istio-token\") pod \"maas-default-gateway-openshift-default-84587756dc-zndxf\" (UID: \"73f8f24f-b955-433b-95ed-5f441f7fceb2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-zndxf" Nov 03 13:46:27.995542 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:27.995380 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvqlg\" (UniqueName: \"kubernetes.io/projected/4b70e366-6fba-4bfa-ac3e-0f0c7b0b64d3-kube-api-access-zvqlg\") pod \"maas-default-gateway-openshift-default-84587756dc-k4c4j\" (UID: \"4b70e366-6fba-4bfa-ac3e-0f0c7b0b64d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-k4c4j" Nov 03 13:46:27.995542 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:27.995428 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b70e366-6fba-4bfa-ac3e-0f0c7b0b64d3-istio-data\") pod \"maas-default-gateway-openshift-default-84587756dc-k4c4j\" (UID: \"4b70e366-6fba-4bfa-ac3e-0f0c7b0b64d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-k4c4j" Nov 03 13:46:27.995542 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:27.995479 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73f8f24f-b955-433b-95ed-5f441f7fceb2-credential-socket\") pod \"maas-default-gateway-openshift-default-84587756dc-zndxf\" (UID: \"73f8f24f-b955-433b-95ed-5f441f7fceb2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-zndxf" Nov 03 13:46:27.995542 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:27.995507 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/825941b6-3ed3-47f2-a859-c891c4f29434-credential-socket\") pod \"maas-default-gateway-openshift-default-84587756dc-92vh9\" (UID: \"825941b6-3ed3-47f2-a859-c891c4f29434\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-92vh9" Nov 03 13:46:27.996056 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:27.995547 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/825941b6-3ed3-47f2-a859-c891c4f29434-workload-certs\") pod \"maas-default-gateway-openshift-default-84587756dc-92vh9\" (UID: \"825941b6-3ed3-47f2-a859-c891c4f29434\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-92vh9" Nov 03 13:46:27.996056 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:27.995583 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b70e366-6fba-4bfa-ac3e-0f0c7b0b64d3-istio-podinfo\") pod \"maas-default-gateway-openshift-default-84587756dc-k4c4j\" (UID: \"4b70e366-6fba-4bfa-ac3e-0f0c7b0b64d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-k4c4j" Nov 03 13:46:27.996056 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:27.995609 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b70e366-6fba-4bfa-ac3e-0f0c7b0b64d3-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-84587756dc-k4c4j\" (UID: \"4b70e366-6fba-4bfa-ac3e-0f0c7b0b64d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-k4c4j" Nov 03 13:46:27.996056 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:27.995652 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b70e366-6fba-4bfa-ac3e-0f0c7b0b64d3-workload-socket\") pod \"maas-default-gateway-openshift-default-84587756dc-k4c4j\" (UID: \"4b70e366-6fba-4bfa-ac3e-0f0c7b0b64d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-k4c4j" Nov 03 13:46:27.996056 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:27.995702 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/825941b6-3ed3-47f2-a859-c891c4f29434-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-84587756dc-92vh9\" (UID: \"825941b6-3ed3-47f2-a859-c891c4f29434\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-92vh9" Nov 03 13:46:27.996056 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:27.995800 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/825941b6-3ed3-47f2-a859-c891c4f29434-istio-data\") pod \"maas-default-gateway-openshift-default-84587756dc-92vh9\" (UID: \"825941b6-3ed3-47f2-a859-c891c4f29434\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-92vh9" Nov 03 13:46:27.996056 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:27.995949 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73f8f24f-b955-433b-95ed-5f441f7fceb2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-84587756dc-zndxf\" (UID: \"73f8f24f-b955-433b-95ed-5f441f7fceb2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-zndxf" Nov 03 13:46:27.996596 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:27.996061 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/825941b6-3ed3-47f2-a859-c891c4f29434-credential-socket\") pod \"maas-default-gateway-openshift-default-84587756dc-92vh9\" (UID: \"825941b6-3ed3-47f2-a859-c891c4f29434\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-92vh9" Nov 03 13:46:27.996596 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:27.996225 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b70e366-6fba-4bfa-ac3e-0f0c7b0b64d3-workload-certs\") pod \"maas-default-gateway-openshift-default-84587756dc-k4c4j\" (UID: \"4b70e366-6fba-4bfa-ac3e-0f0c7b0b64d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-k4c4j" Nov 03 13:46:27.996596 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:27.996492 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/825941b6-3ed3-47f2-a859-c891c4f29434-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-84587756dc-92vh9\" (UID: \"825941b6-3ed3-47f2-a859-c891c4f29434\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-92vh9" Nov 03 13:46:27.996596 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:27.996496 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b70e366-6fba-4bfa-ac3e-0f0c7b0b64d3-istio-data\") pod \"maas-default-gateway-openshift-default-84587756dc-k4c4j\" (UID: \"4b70e366-6fba-4bfa-ac3e-0f0c7b0b64d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-k4c4j" Nov 03 13:46:27.996596 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:27.996541 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73f8f24f-b955-433b-95ed-5f441f7fceb2-credential-socket\") pod \"maas-default-gateway-openshift-default-84587756dc-zndxf\" (UID: \"73f8f24f-b955-433b-95ed-5f441f7fceb2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-zndxf" Nov 03 13:46:27.997200 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:27.997178 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b70e366-6fba-4bfa-ac3e-0f0c7b0b64d3-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-84587756dc-k4c4j\" (UID: \"4b70e366-6fba-4bfa-ac3e-0f0c7b0b64d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-k4c4j" Nov 03 13:46:27.997329 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:27.997213 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/825941b6-3ed3-47f2-a859-c891c4f29434-workload-certs\") pod \"maas-default-gateway-openshift-default-84587756dc-92vh9\" (UID: \"825941b6-3ed3-47f2-a859-c891c4f29434\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-92vh9" Nov 03 13:46:27.997416 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:27.997361 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b70e366-6fba-4bfa-ac3e-0f0c7b0b64d3-workload-socket\") pod \"maas-default-gateway-openshift-default-84587756dc-k4c4j\" (UID: \"4b70e366-6fba-4bfa-ac3e-0f0c7b0b64d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-k4c4j" Nov 03 13:46:27.997541 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:27.997521 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b70e366-6fba-4bfa-ac3e-0f0c7b0b64d3-istio-envoy\") pod \"maas-default-gateway-openshift-default-84587756dc-k4c4j\" (UID: \"4b70e366-6fba-4bfa-ac3e-0f0c7b0b64d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-k4c4j" Nov 03 13:46:27.997608 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:27.997527 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/825941b6-3ed3-47f2-a859-c891c4f29434-istio-envoy\") pod \"maas-default-gateway-openshift-default-84587756dc-92vh9\" (UID: \"825941b6-3ed3-47f2-a859-c891c4f29434\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-92vh9" Nov 03 13:46:27.997654 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:27.997603 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/825941b6-3ed3-47f2-a859-c891c4f29434-istio-podinfo\") pod \"maas-default-gateway-openshift-default-84587756dc-92vh9\" (UID: \"825941b6-3ed3-47f2-a859-c891c4f29434\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-92vh9" Nov 03 13:46:27.997772 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:27.997755 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73f8f24f-b955-433b-95ed-5f441f7fceb2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-84587756dc-zndxf\" (UID: \"73f8f24f-b955-433b-95ed-5f441f7fceb2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-zndxf" Nov 03 13:46:27.998827 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:27.998810 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73f8f24f-b955-433b-95ed-5f441f7fceb2-istio-envoy\") pod \"maas-default-gateway-openshift-default-84587756dc-zndxf\" (UID: \"73f8f24f-b955-433b-95ed-5f441f7fceb2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-zndxf" Nov 03 13:46:27.999305 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:27.999260 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b70e366-6fba-4bfa-ac3e-0f0c7b0b64d3-istio-podinfo\") pod \"maas-default-gateway-openshift-default-84587756dc-k4c4j\" (UID: \"4b70e366-6fba-4bfa-ac3e-0f0c7b0b64d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-k4c4j" Nov 03 13:46:28.015698 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:28.015465 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhmxm\" (UniqueName: \"kubernetes.io/projected/825941b6-3ed3-47f2-a859-c891c4f29434-kube-api-access-hhmxm\") pod \"maas-default-gateway-openshift-default-84587756dc-92vh9\" (UID: \"825941b6-3ed3-47f2-a859-c891c4f29434\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-92vh9" Nov 03 13:46:28.015698 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:28.015664 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/825941b6-3ed3-47f2-a859-c891c4f29434-istio-token\") pod \"maas-default-gateway-openshift-default-84587756dc-92vh9\" (UID: \"825941b6-3ed3-47f2-a859-c891c4f29434\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-92vh9" Nov 03 13:46:28.015934 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:28.015910 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b70e366-6fba-4bfa-ac3e-0f0c7b0b64d3-istio-token\") pod \"maas-default-gateway-openshift-default-84587756dc-k4c4j\" (UID: \"4b70e366-6fba-4bfa-ac3e-0f0c7b0b64d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-k4c4j" Nov 03 13:46:28.016043 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:28.016027 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvqlg\" (UniqueName: \"kubernetes.io/projected/4b70e366-6fba-4bfa-ac3e-0f0c7b0b64d3-kube-api-access-zvqlg\") pod \"maas-default-gateway-openshift-default-84587756dc-k4c4j\" (UID: \"4b70e366-6fba-4bfa-ac3e-0f0c7b0b64d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-k4c4j" Nov 03 13:46:28.021856 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:28.021830 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/73f8f24f-b955-433b-95ed-5f441f7fceb2-istio-token\") pod \"maas-default-gateway-openshift-default-84587756dc-zndxf\" (UID: \"73f8f24f-b955-433b-95ed-5f441f7fceb2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-zndxf" Nov 03 13:46:28.022044 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:28.022021 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kb5h2\" (UniqueName: \"kubernetes.io/projected/73f8f24f-b955-433b-95ed-5f441f7fceb2-kube-api-access-kb5h2\") pod \"maas-default-gateway-openshift-default-84587756dc-zndxf\" (UID: \"73f8f24f-b955-433b-95ed-5f441f7fceb2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-zndxf" Nov 03 13:46:28.052938 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:28.052910 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-zndxf" Nov 03 13:46:28.074121 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:28.074095 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-92vh9" Nov 03 13:46:28.081212 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:28.081188 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-k4c4j" Nov 03 13:46:28.335072 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:28.335042 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-84587756dc-zndxf"] Nov 03 13:46:28.336883 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:46:28.336852 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73f8f24f_b955_433b_95ed_5f441f7fceb2.slice/crio-a1fb1415f7fdebe0db2d38f46a881e5c4ea3b557802063be416f23c7671382aa WatchSource:0}: Error finding container a1fb1415f7fdebe0db2d38f46a881e5c4ea3b557802063be416f23c7671382aa: Status 404 returned error can't find the container with id a1fb1415f7fdebe0db2d38f46a881e5c4ea3b557802063be416f23c7671382aa Nov 03 13:46:28.339206 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:28.339174 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:46:28.339334 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:28.339252 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:46:28.339334 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:28.339323 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:46:28.378836 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:28.378806 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-84587756dc-k4c4j"] Nov 03 13:46:28.380742 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:28.380714 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-84587756dc-92vh9"] Nov 03 13:46:28.384558 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:46:28.384531 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b70e366_6fba_4bfa_ac3e_0f0c7b0b64d3.slice/crio-b1c90c2184b7b612107518f04385871bbd67090776e38914aec987e40f02171c WatchSource:0}: Error finding container b1c90c2184b7b612107518f04385871bbd67090776e38914aec987e40f02171c: Status 404 returned error can't find the container with id b1c90c2184b7b612107518f04385871bbd67090776e38914aec987e40f02171c Nov 03 13:46:28.384805 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:46:28.384776 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod825941b6_3ed3_47f2_a859_c891c4f29434.slice/crio-8f430d44e06a5cf4ccc1a8dcfbcac52264cb1ae8bb1517c3c1bc7eeed5635215 WatchSource:0}: Error finding container 8f430d44e06a5cf4ccc1a8dcfbcac52264cb1ae8bb1517c3c1bc7eeed5635215: Status 404 returned error can't find the container with id 8f430d44e06a5cf4ccc1a8dcfbcac52264cb1ae8bb1517c3c1bc7eeed5635215 Nov 03 13:46:28.387081 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:28.387036 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:46:28.387168 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:28.387120 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:46:28.387168 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:28.387036 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:46:28.387168 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:28.387157 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:46:28.387302 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:28.387191 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:46:28.387302 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:28.387229 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:46:28.843817 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:28.843165 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-k4c4j" event={"ID":"4b70e366-6fba-4bfa-ac3e-0f0c7b0b64d3","Type":"ContainerStarted","Data":"620b24a909bac78d8faf332763593213552e9377724f87b563ece1abefcb6145"} Nov 03 13:46:28.844409 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:28.844381 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-k4c4j" event={"ID":"4b70e366-6fba-4bfa-ac3e-0f0c7b0b64d3","Type":"ContainerStarted","Data":"b1c90c2184b7b612107518f04385871bbd67090776e38914aec987e40f02171c"} Nov 03 13:46:28.855676 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:28.854886 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-92vh9" event={"ID":"825941b6-3ed3-47f2-a859-c891c4f29434","Type":"ContainerStarted","Data":"0ded7eeefd59bba13d70ae14c33f90ccb20399d4babb3d311e36f7c6d2e45518"} Nov 03 13:46:28.855676 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:28.854956 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-92vh9" event={"ID":"825941b6-3ed3-47f2-a859-c891c4f29434","Type":"ContainerStarted","Data":"8f430d44e06a5cf4ccc1a8dcfbcac52264cb1ae8bb1517c3c1bc7eeed5635215"} Nov 03 13:46:28.861114 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:28.860363 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-zndxf" event={"ID":"73f8f24f-b955-433b-95ed-5f441f7fceb2","Type":"ContainerStarted","Data":"1e6591578dd119b5e41c03f612c27f8c191fca89107e35ab62d0f4b1e710127a"} Nov 03 13:46:28.861114 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:28.860430 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-zndxf" event={"ID":"73f8f24f-b955-433b-95ed-5f441f7fceb2","Type":"ContainerStarted","Data":"a1fb1415f7fdebe0db2d38f46a881e5c4ea3b557802063be416f23c7671382aa"} Nov 03 13:46:28.883968 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:28.883899 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-k4c4j" podStartSLOduration=1.883879503 podStartE2EDuration="1.883879503s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:46:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:46:28.878766997 +0000 UTC m=+2268.886958292" watchObservedRunningTime="2026-11-03 13:46:28.883879503 +0000 UTC m=+2268.892070780" Nov 03 13:46:28.923252 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:28.923189 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-zndxf" podStartSLOduration=1.923169007 podStartE2EDuration="1.923169007s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:46:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:46:28.92277242 +0000 UTC m=+2268.930963699" watchObservedRunningTime="2026-11-03 13:46:28.923169007 +0000 UTC m=+2268.931360285" Nov 03 13:46:28.950543 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:28.950477 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-92vh9" podStartSLOduration=1.950462587 podStartE2EDuration="1.950462587s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:46:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:46:28.949590657 +0000 UTC m=+2268.957781933" watchObservedRunningTime="2026-11-03 13:46:28.950462587 +0000 UTC m=+2268.958653860" Nov 03 13:46:29.053597 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:29.053557 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-zndxf" Nov 03 13:46:29.055843 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:29.055592 5024 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-zndxf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" start-of-body= Nov 03 13:46:29.055843 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:29.055638 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-zndxf" podUID="73f8f24f-b955-433b-95ed-5f441f7fceb2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" Nov 03 13:46:29.076892 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:29.076814 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-92vh9" Nov 03 13:46:29.077140 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:29.077113 5024 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-92vh9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" start-of-body= Nov 03 13:46:29.077349 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:29.077319 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-92vh9" podUID="825941b6-3ed3-47f2-a859-c891c4f29434" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" Nov 03 13:46:29.082686 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:29.082664 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-k4c4j" Nov 03 13:46:29.082924 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:29.082905 5024 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-k4c4j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" start-of-body= Nov 03 13:46:29.083086 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:29.083063 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-k4c4j" podUID="4b70e366-6fba-4bfa-ac3e-0f0c7b0b64d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" Nov 03 13:46:30.054385 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:30.054336 5024 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-zndxf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" start-of-body= Nov 03 13:46:30.054893 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:30.054404 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-zndxf" podUID="73f8f24f-b955-433b-95ed-5f441f7fceb2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" Nov 03 13:46:30.074692 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:30.074655 5024 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-92vh9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" start-of-body= Nov 03 13:46:30.079309 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:30.074889 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-92vh9" podUID="825941b6-3ed3-47f2-a859-c891c4f29434" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" Nov 03 13:46:30.085667 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:30.085639 5024 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-k4c4j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" start-of-body= Nov 03 13:46:30.085851 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:30.085825 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-k4c4j" podUID="4b70e366-6fba-4bfa-ac3e-0f0c7b0b64d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" Nov 03 13:46:30.999304 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:30.999154 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-sv6fn"] Nov 03 13:46:31.017503 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:31.017465 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-sv6fn" Nov 03 13:46:31.020946 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:31.020917 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-sv6fn"] Nov 03 13:46:31.024127 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:31.024096 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 13:46:31.029840 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:31.028033 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-phqfq\"" Nov 03 13:46:31.058466 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:31.057200 5024 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-zndxf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" start-of-body= Nov 03 13:46:31.059142 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:31.059110 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-zndxf" podUID="73f8f24f-b955-433b-95ed-5f441f7fceb2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" Nov 03 13:46:31.063339 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:31.062902 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a71e9bee-8090-49bd-ad64-30fd30aabe7a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-sv6fn\" (UID: \"a71e9bee-8090-49bd-ad64-30fd30aabe7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-sv6fn" Nov 03 13:46:31.063339 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:31.063044 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rgx5h\" (UniqueName: \"kubernetes.io/projected/a71e9bee-8090-49bd-ad64-30fd30aabe7a-kube-api-access-rgx5h\") pod \"maas-api-revoke-sub-85237e45-sv6fn\" (UID: \"a71e9bee-8090-49bd-ad64-30fd30aabe7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-sv6fn" Nov 03 13:46:31.063339 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:31.063100 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a71e9bee-8090-49bd-ad64-30fd30aabe7a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-sv6fn\" (UID: \"a71e9bee-8090-49bd-ad64-30fd30aabe7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-sv6fn" Nov 03 13:46:31.066804 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:31.066764 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 13:46:31.066928 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:31.066867 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 13:46:31.083636 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:31.083596 5024 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-k4c4j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" start-of-body= Nov 03 13:46:31.084044 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:31.084014 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-k4c4j" podUID="4b70e366-6fba-4bfa-ac3e-0f0c7b0b64d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" Nov 03 13:46:31.084817 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:31.084795 5024 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-92vh9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" start-of-body= Nov 03 13:46:31.084982 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:31.084959 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-92vh9" podUID="825941b6-3ed3-47f2-a859-c891c4f29434" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" Nov 03 13:46:31.164596 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:31.164553 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rgx5h\" (UniqueName: \"kubernetes.io/projected/a71e9bee-8090-49bd-ad64-30fd30aabe7a-kube-api-access-rgx5h\") pod \"maas-api-revoke-sub-85237e45-sv6fn\" (UID: \"a71e9bee-8090-49bd-ad64-30fd30aabe7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-sv6fn" Nov 03 13:46:31.165620 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:31.164873 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a71e9bee-8090-49bd-ad64-30fd30aabe7a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-sv6fn\" (UID: \"a71e9bee-8090-49bd-ad64-30fd30aabe7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-sv6fn" Nov 03 13:46:31.165848 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:31.165815 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a71e9bee-8090-49bd-ad64-30fd30aabe7a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-sv6fn\" (UID: \"a71e9bee-8090-49bd-ad64-30fd30aabe7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-sv6fn" Nov 03 13:46:31.166491 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:31.166451 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a71e9bee-8090-49bd-ad64-30fd30aabe7a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-sv6fn\" (UID: \"a71e9bee-8090-49bd-ad64-30fd30aabe7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-sv6fn" Nov 03 13:46:31.167624 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:31.167597 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a71e9bee-8090-49bd-ad64-30fd30aabe7a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-sv6fn\" (UID: \"a71e9bee-8090-49bd-ad64-30fd30aabe7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-sv6fn" Nov 03 13:46:31.209307 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:31.207329 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rgx5h\" (UniqueName: \"kubernetes.io/projected/a71e9bee-8090-49bd-ad64-30fd30aabe7a-kube-api-access-rgx5h\") pod \"maas-api-revoke-sub-85237e45-sv6fn\" (UID: \"a71e9bee-8090-49bd-ad64-30fd30aabe7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-sv6fn" Nov 03 13:46:31.459308 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:31.458463 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-sv6fn" Nov 03 13:46:32.055126 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:32.054669 5024 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-zndxf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" start-of-body= Nov 03 13:46:32.055126 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:32.054734 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-zndxf" podUID="73f8f24f-b955-433b-95ed-5f441f7fceb2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" Nov 03 13:46:32.080316 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:32.076670 5024 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-92vh9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" start-of-body= Nov 03 13:46:32.080316 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:32.076752 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-92vh9" podUID="825941b6-3ed3-47f2-a859-c891c4f29434" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" Nov 03 13:46:32.082746 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:32.082638 5024 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-k4c4j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" start-of-body= Nov 03 13:46:32.082746 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:32.082694 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-k4c4j" podUID="4b70e366-6fba-4bfa-ac3e-0f0c7b0b64d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" Nov 03 13:46:32.160960 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:32.160933 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-sv6fn"] Nov 03 13:46:32.963756 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:32.963719 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-sv6fn" event={"ID":"a71e9bee-8090-49bd-ad64-30fd30aabe7a","Type":"ContainerStarted","Data":"87612cd87efdc500c002e0b19137253dc1e77e1d2629ffebb951c20aa6887506"} Nov 03 13:46:32.963943 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:32.963776 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-sv6fn" event={"ID":"a71e9bee-8090-49bd-ad64-30fd30aabe7a","Type":"ContainerStarted","Data":"3211d0cb95c6f57368489aa5a7885fe2ae30f30d83c5db7f879622a50aac4296"} Nov 03 13:46:33.006321 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:32.985682 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-sv6fn" podStartSLOduration=2.985645682 podStartE2EDuration="2.985645682s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:46:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:46:32.984607151 +0000 UTC m=+2272.992798429" watchObservedRunningTime="2026-11-03 13:46:32.985645682 +0000 UTC m=+2272.993836960" Nov 03 13:46:33.057320 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:33.055736 5024 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-zndxf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" start-of-body= Nov 03 13:46:33.057320 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:33.055802 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-zndxf" podUID="73f8f24f-b955-433b-95ed-5f441f7fceb2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" Nov 03 13:46:33.078986 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:33.078923 5024 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-92vh9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" start-of-body= Nov 03 13:46:33.079530 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:33.078992 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-92vh9" podUID="825941b6-3ed3-47f2-a859-c891c4f29434" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" Nov 03 13:46:33.087321 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:33.083629 5024 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-k4c4j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" start-of-body= Nov 03 13:46:33.087321 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:33.083687 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-k4c4j" podUID="4b70e366-6fba-4bfa-ac3e-0f0c7b0b64d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" Nov 03 13:46:33.972458 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:33.972418 5024 generic.go:356] "Generic (PLEG): container finished" podID="a71e9bee-8090-49bd-ad64-30fd30aabe7a" containerID="87612cd87efdc500c002e0b19137253dc1e77e1d2629ffebb951c20aa6887506" exitCode=0 Nov 03 13:46:33.972843 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:33.972815 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-sv6fn" event={"ID":"a71e9bee-8090-49bd-ad64-30fd30aabe7a","Type":"ContainerDied","Data":"87612cd87efdc500c002e0b19137253dc1e77e1d2629ffebb951c20aa6887506"} Nov 03 13:46:34.058833 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:34.058793 5024 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-zndxf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" start-of-body= Nov 03 13:46:34.059143 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:34.059095 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-zndxf" podUID="73f8f24f-b955-433b-95ed-5f441f7fceb2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" Nov 03 13:46:34.082974 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:34.082936 5024 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-92vh9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" start-of-body= Nov 03 13:46:34.083648 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:34.083615 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-92vh9" podUID="825941b6-3ed3-47f2-a859-c891c4f29434" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" Nov 03 13:46:34.087488 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:34.087381 5024 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-k4c4j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" start-of-body= Nov 03 13:46:34.087488 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:34.087434 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-k4c4j" podUID="4b70e366-6fba-4bfa-ac3e-0f0c7b0b64d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" Nov 03 13:46:35.054691 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:35.054195 5024 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-zndxf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" start-of-body= Nov 03 13:46:35.054691 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:35.054260 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-zndxf" podUID="73f8f24f-b955-433b-95ed-5f441f7fceb2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" Nov 03 13:46:35.081319 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:35.081031 5024 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-92vh9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" start-of-body= Nov 03 13:46:35.081319 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:35.081101 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-92vh9" podUID="825941b6-3ed3-47f2-a859-c891c4f29434" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" Nov 03 13:46:35.082481 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:35.082375 5024 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-k4c4j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" start-of-body= Nov 03 13:46:35.082481 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:35.082425 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-k4c4j" podUID="4b70e366-6fba-4bfa-ac3e-0f0c7b0b64d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" Nov 03 13:46:35.167344 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:35.167277 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-sv6fn" Nov 03 13:46:35.256482 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:35.256387 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a71e9bee-8090-49bd-ad64-30fd30aabe7a-kube-api-access-rgx5h\" (UniqueName: \"kubernetes.io/projected/a71e9bee-8090-49bd-ad64-30fd30aabe7a-kube-api-access-rgx5h\") pod \"a71e9bee-8090-49bd-ad64-30fd30aabe7a\" (UID: \"a71e9bee-8090-49bd-ad64-30fd30aabe7a\") " Nov 03 13:46:35.256482 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:35.256473 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a71e9bee-8090-49bd-ad64-30fd30aabe7a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a71e9bee-8090-49bd-ad64-30fd30aabe7a-maas-api-openshift-ca\") pod \"a71e9bee-8090-49bd-ad64-30fd30aabe7a\" (UID: \"a71e9bee-8090-49bd-ad64-30fd30aabe7a\") " Nov 03 13:46:35.256730 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:35.256573 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a71e9bee-8090-49bd-ad64-30fd30aabe7a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a71e9bee-8090-49bd-ad64-30fd30aabe7a-maas-api-xks-ca\") pod \"a71e9bee-8090-49bd-ad64-30fd30aabe7a\" (UID: \"a71e9bee-8090-49bd-ad64-30fd30aabe7a\") " Nov 03 13:46:35.256947 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:35.256912 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a71e9bee-8090-49bd-ad64-30fd30aabe7a-maas-api-openshift-ca" pod "a71e9bee-8090-49bd-ad64-30fd30aabe7a" (UID: "a71e9bee-8090-49bd-ad64-30fd30aabe7a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:46:35.259271 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:35.259239 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a71e9bee-8090-49bd-ad64-30fd30aabe7a-maas-api-xks-ca" pod "a71e9bee-8090-49bd-ad64-30fd30aabe7a" (UID: "a71e9bee-8090-49bd-ad64-30fd30aabe7a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 13:46:35.259440 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:35.259275 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a71e9bee-8090-49bd-ad64-30fd30aabe7a-kube-api-access-rgx5h" pod "a71e9bee-8090-49bd-ad64-30fd30aabe7a" (UID: "a71e9bee-8090-49bd-ad64-30fd30aabe7a"). InnerVolumeSpecName "kube-api-access-rgx5h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:46:35.358419 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:35.358378 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rgx5h\" (UniqueName: \"kubernetes.io/projected/a71e9bee-8090-49bd-ad64-30fd30aabe7a-kube-api-access-rgx5h\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:46:35.358419 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:35.358419 5024 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a71e9bee-8090-49bd-ad64-30fd30aabe7a-maas-api-openshift-ca\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:46:35.358703 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:35.358438 5024 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a71e9bee-8090-49bd-ad64-30fd30aabe7a-maas-api-xks-ca\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:46:35.989362 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:35.989271 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-sv6fn" event={"ID":"a71e9bee-8090-49bd-ad64-30fd30aabe7a","Type":"ContainerDied","Data":"3211d0cb95c6f57368489aa5a7885fe2ae30f30d83c5db7f879622a50aac4296"} Nov 03 13:46:35.989362 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:35.989339 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-sv6fn" Nov 03 13:46:35.989648 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:35.989435 5024 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3211d0cb95c6f57368489aa5a7885fe2ae30f30d83c5db7f879622a50aac4296" Nov 03 13:46:36.053401 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:36.053360 5024 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-zndxf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" start-of-body= Nov 03 13:46:36.053624 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:36.053431 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-zndxf" podUID="73f8f24f-b955-433b-95ed-5f441f7fceb2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" Nov 03 13:46:36.075165 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:36.075120 5024 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-92vh9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" start-of-body= Nov 03 13:46:36.075411 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:36.075189 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-92vh9" podUID="825941b6-3ed3-47f2-a859-c891c4f29434" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" Nov 03 13:46:36.082673 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:36.082633 5024 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-k4c4j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" start-of-body= Nov 03 13:46:36.082819 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:36.082696 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-k4c4j" podUID="4b70e366-6fba-4bfa-ac3e-0f0c7b0b64d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" Nov 03 13:46:38.054926 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:38.054836 5024 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-zndxf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.172:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 13:46:38.055423 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:38.055390 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-zndxf" podUID="73f8f24f-b955-433b-95ed-5f441f7fceb2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.172:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 13:46:38.077048 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:38.077006 5024 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-92vh9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.173:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 13:46:38.077382 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:38.077338 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-92vh9" podUID="825941b6-3ed3-47f2-a859-c891c4f29434" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.173:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 13:46:38.083122 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:38.083081 5024 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-k4c4j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.174:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 13:46:38.083246 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:38.083135 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-k4c4j" podUID="4b70e366-6fba-4bfa-ac3e-0f0c7b0b64d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.174:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 13:46:39.105310 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:39.103484 5024 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-zndxf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.172:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 13:46:39.105310 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:39.103582 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-zndxf" podUID="73f8f24f-b955-433b-95ed-5f441f7fceb2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.172:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 13:46:39.123522 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:39.123463 5024 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-92vh9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.173:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 13:46:39.123850 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:39.123802 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-92vh9" podUID="825941b6-3ed3-47f2-a859-c891c4f29434" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.173:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 13:46:39.132992 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:39.132937 5024 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-k4c4j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.174:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 13:46:39.133311 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:39.133256 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-k4c4j" podUID="4b70e366-6fba-4bfa-ac3e-0f0c7b0b64d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.174:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 13:46:39.560801 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:46:39.560709 5024 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d7mwz" containerName="extractor" Nov 03 13:46:40.124489 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:40.124448 5024 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-92vh9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.173:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 13:46:40.125094 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:40.125062 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-92vh9" podUID="825941b6-3ed3-47f2-a859-c891c4f29434" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.173:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 13:46:40.145435 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:40.145396 5024 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-zndxf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.172:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 13:46:40.145604 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:40.145449 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-zndxf" podUID="73f8f24f-b955-433b-95ed-5f441f7fceb2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.172:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 13:46:40.174497 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:40.174441 5024 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-k4c4j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.174:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 13:46:40.174688 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:40.174510 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-k4c4j" podUID="4b70e366-6fba-4bfa-ac3e-0f0c7b0b64d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.174:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 13:46:40.240486 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:40.240445 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-92vh9" Nov 03 13:46:40.241071 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:40.241037 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-92vh9" Nov 03 13:46:40.241753 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:40.241728 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-92vh9" Nov 03 13:46:40.263062 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:40.263030 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-zndxf" Nov 03 13:46:40.263470 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:40.263448 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-zndxf" Nov 03 13:46:40.264077 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:40.264054 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-zndxf" Nov 03 13:46:40.295943 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:40.295908 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-k4c4j" Nov 03 13:46:40.296326 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:40.296269 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-k4c4j" Nov 03 13:46:40.297401 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:40.297380 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-k4c4j" Nov 03 13:46:42.705588 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:42.705545 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-bc865c477-mhwxn"] Nov 03 13:46:42.706052 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:42.705795 5024 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-bc865c477-mhwxn" podUID="95d50d1a-1756-4b07-a016-a4c2cbd23a4d" containerName="manager" containerID="cri-o://6b7774c1aeb8429d5e974a29950f699c15451427d0f6c8523f5ee999411401fd" gracePeriod=30 Nov 03 13:46:42.966227 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:42.966171 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-bc865c477-mhwxn" Nov 03 13:46:43.043767 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:43.043732 5024 generic.go:356] "Generic (PLEG): container finished" podID="95d50d1a-1756-4b07-a016-a4c2cbd23a4d" containerID="6b7774c1aeb8429d5e974a29950f699c15451427d0f6c8523f5ee999411401fd" exitCode=0 Nov 03 13:46:43.043938 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:43.043812 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-bc865c477-mhwxn" Nov 03 13:46:43.043938 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:43.043818 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-bc865c477-mhwxn" event={"ID":"95d50d1a-1756-4b07-a016-a4c2cbd23a4d","Type":"ContainerDied","Data":"6b7774c1aeb8429d5e974a29950f699c15451427d0f6c8523f5ee999411401fd"} Nov 03 13:46:43.044015 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:43.043950 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-bc865c477-mhwxn" event={"ID":"95d50d1a-1756-4b07-a016-a4c2cbd23a4d","Type":"ContainerDied","Data":"b6cc476696064f201ecc1e5b2b60a4ebbd03e295d30e596a72e5a092bd80a969"} Nov 03 13:46:43.044015 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:43.043981 5024 scope.go:122] "RemoveContainer" containerID="6b7774c1aeb8429d5e974a29950f699c15451427d0f6c8523f5ee999411401fd" Nov 03 13:46:43.053101 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:43.053075 5024 scope.go:122] "RemoveContainer" containerID="6b7774c1aeb8429d5e974a29950f699c15451427d0f6c8523f5ee999411401fd" Nov 03 13:46:43.053464 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:46:43.053441 5024 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b7774c1aeb8429d5e974a29950f699c15451427d0f6c8523f5ee999411401fd\": container with ID starting with 6b7774c1aeb8429d5e974a29950f699c15451427d0f6c8523f5ee999411401fd not found: ID does not exist" containerID="6b7774c1aeb8429d5e974a29950f699c15451427d0f6c8523f5ee999411401fd" Nov 03 13:46:43.053549 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:43.053473 5024 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b7774c1aeb8429d5e974a29950f699c15451427d0f6c8523f5ee999411401fd"} err="failed to get container status \"6b7774c1aeb8429d5e974a29950f699c15451427d0f6c8523f5ee999411401fd\": rpc error: code = NotFound desc = could not find container \"6b7774c1aeb8429d5e974a29950f699c15451427d0f6c8523f5ee999411401fd\": container with ID starting with 6b7774c1aeb8429d5e974a29950f699c15451427d0f6c8523f5ee999411401fd not found: ID does not exist" Nov 03 13:46:43.082089 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:43.082036 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/95d50d1a-1756-4b07-a016-a4c2cbd23a4d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/95d50d1a-1756-4b07-a016-a4c2cbd23a4d-metrics-certs\") pod \"95d50d1a-1756-4b07-a016-a4c2cbd23a4d\" (UID: \"95d50d1a-1756-4b07-a016-a4c2cbd23a4d\") " Nov 03 13:46:43.082245 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:43.082097 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95d50d1a-1756-4b07-a016-a4c2cbd23a4d-kube-api-access-gv656\" (UniqueName: \"kubernetes.io/projected/95d50d1a-1756-4b07-a016-a4c2cbd23a4d-kube-api-access-gv656\") pod \"95d50d1a-1756-4b07-a016-a4c2cbd23a4d\" (UID: \"95d50d1a-1756-4b07-a016-a4c2cbd23a4d\") " Nov 03 13:46:43.082245 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:43.082133 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/95d50d1a-1756-4b07-a016-a4c2cbd23a4d-webhook-cert\" (UniqueName: \"kubernetes.io/secret/95d50d1a-1756-4b07-a016-a4c2cbd23a4d-webhook-cert\") pod \"95d50d1a-1756-4b07-a016-a4c2cbd23a4d\" (UID: \"95d50d1a-1756-4b07-a016-a4c2cbd23a4d\") " Nov 03 13:46:43.084402 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:43.084363 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95d50d1a-1756-4b07-a016-a4c2cbd23a4d-kube-api-access-gv656" pod "95d50d1a-1756-4b07-a016-a4c2cbd23a4d" (UID: "95d50d1a-1756-4b07-a016-a4c2cbd23a4d"). InnerVolumeSpecName "kube-api-access-gv656". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:46:43.084402 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:43.084400 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/95d50d1a-1756-4b07-a016-a4c2cbd23a4d-webhook-cert" pod "95d50d1a-1756-4b07-a016-a4c2cbd23a4d" (UID: "95d50d1a-1756-4b07-a016-a4c2cbd23a4d"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 13:46:43.084562 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:43.084424 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/95d50d1a-1756-4b07-a016-a4c2cbd23a4d-metrics-certs" pod "95d50d1a-1756-4b07-a016-a4c2cbd23a4d" (UID: "95d50d1a-1756-4b07-a016-a4c2cbd23a4d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 13:46:43.183787 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:43.183742 5024 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/95d50d1a-1756-4b07-a016-a4c2cbd23a4d-metrics-certs\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:46:43.183787 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:43.183781 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gv656\" (UniqueName: \"kubernetes.io/projected/95d50d1a-1756-4b07-a016-a4c2cbd23a4d-kube-api-access-gv656\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:46:43.184004 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:43.183798 5024 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/95d50d1a-1756-4b07-a016-a4c2cbd23a4d-webhook-cert\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:46:43.370624 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:43.370578 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-bc865c477-mhwxn"] Nov 03 13:46:43.373001 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:43.372970 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-bc865c477-mhwxn"] Nov 03 13:46:44.107117 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:44.107081 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="95d50d1a-1756-4b07-a016-a4c2cbd23a4d" path="/var/lib/kubelet/pods/95d50d1a-1756-4b07-a016-a4c2cbd23a4d/volumes" Nov 03 13:46:44.601584 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:44.601506 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-bc865c477-q4c9z"] Nov 03 13:46:44.601942 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:44.601929 5024 cpu_manager.go:414] "containerMap: removing container" podUID="95d50d1a-1756-4b07-a016-a4c2cbd23a4d" containerName="manager" Nov 03 13:46:44.601994 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:44.601943 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95d50d1a-1756-4b07-a016-a4c2cbd23a4d" containerName="manager" Nov 03 13:46:44.601994 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:44.601975 5024 cpu_manager.go:414] "containerMap: removing container" podUID="a71e9bee-8090-49bd-ad64-30fd30aabe7a" containerName="revoke-keys" Nov 03 13:46:44.601994 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:44.601981 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a71e9bee-8090-49bd-ad64-30fd30aabe7a" containerName="revoke-keys" Nov 03 13:46:44.602094 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:44.602037 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="a71e9bee-8090-49bd-ad64-30fd30aabe7a" containerName="revoke-keys" Nov 03 13:46:44.602094 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:44.602047 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="95d50d1a-1756-4b07-a016-a4c2cbd23a4d" containerName="manager" Nov 03 13:46:44.607895 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:44.607872 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-bc865c477-q4c9z" Nov 03 13:46:44.617093 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:44.617067 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Nov 03 13:46:44.617303 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:44.617127 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Nov 03 13:46:44.617303 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:44.617137 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Nov 03 13:46:44.617303 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:44.617236 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-qn5lb\"" Nov 03 13:46:44.618175 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:44.618145 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-bc865c477-q4c9z"] Nov 03 13:46:44.700323 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:44.700261 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8f8mn\" (UniqueName: \"kubernetes.io/projected/75c9ed72-4fa6-4d71-a8c9-6453fb9e2c1f-kube-api-access-8f8mn\") pod \"maas-controller-bc865c477-q4c9z\" (UID: \"75c9ed72-4fa6-4d71-a8c9-6453fb9e2c1f\") " pod="opendatahub/maas-controller-bc865c477-q4c9z" Nov 03 13:46:44.700484 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:44.700335 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/75c9ed72-4fa6-4d71-a8c9-6453fb9e2c1f-webhook-cert\") pod \"maas-controller-bc865c477-q4c9z\" (UID: \"75c9ed72-4fa6-4d71-a8c9-6453fb9e2c1f\") " pod="opendatahub/maas-controller-bc865c477-q4c9z" Nov 03 13:46:44.700484 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:44.700442 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/75c9ed72-4fa6-4d71-a8c9-6453fb9e2c1f-metrics-certs\") pod \"maas-controller-bc865c477-q4c9z\" (UID: \"75c9ed72-4fa6-4d71-a8c9-6453fb9e2c1f\") " pod="opendatahub/maas-controller-bc865c477-q4c9z" Nov 03 13:46:44.801624 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:44.801592 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8f8mn\" (UniqueName: \"kubernetes.io/projected/75c9ed72-4fa6-4d71-a8c9-6453fb9e2c1f-kube-api-access-8f8mn\") pod \"maas-controller-bc865c477-q4c9z\" (UID: \"75c9ed72-4fa6-4d71-a8c9-6453fb9e2c1f\") " pod="opendatahub/maas-controller-bc865c477-q4c9z" Nov 03 13:46:44.801783 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:44.801636 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/75c9ed72-4fa6-4d71-a8c9-6453fb9e2c1f-webhook-cert\") pod \"maas-controller-bc865c477-q4c9z\" (UID: \"75c9ed72-4fa6-4d71-a8c9-6453fb9e2c1f\") " pod="opendatahub/maas-controller-bc865c477-q4c9z" Nov 03 13:46:44.801783 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:44.801675 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/75c9ed72-4fa6-4d71-a8c9-6453fb9e2c1f-metrics-certs\") pod \"maas-controller-bc865c477-q4c9z\" (UID: \"75c9ed72-4fa6-4d71-a8c9-6453fb9e2c1f\") " pod="opendatahub/maas-controller-bc865c477-q4c9z" Nov 03 13:46:44.804078 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:44.804049 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/75c9ed72-4fa6-4d71-a8c9-6453fb9e2c1f-metrics-certs\") pod \"maas-controller-bc865c477-q4c9z\" (UID: \"75c9ed72-4fa6-4d71-a8c9-6453fb9e2c1f\") " pod="opendatahub/maas-controller-bc865c477-q4c9z" Nov 03 13:46:44.804192 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:44.804089 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/75c9ed72-4fa6-4d71-a8c9-6453fb9e2c1f-webhook-cert\") pod \"maas-controller-bc865c477-q4c9z\" (UID: \"75c9ed72-4fa6-4d71-a8c9-6453fb9e2c1f\") " pod="opendatahub/maas-controller-bc865c477-q4c9z" Nov 03 13:46:44.811497 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:44.811473 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8f8mn\" (UniqueName: \"kubernetes.io/projected/75c9ed72-4fa6-4d71-a8c9-6453fb9e2c1f-kube-api-access-8f8mn\") pod \"maas-controller-bc865c477-q4c9z\" (UID: \"75c9ed72-4fa6-4d71-a8c9-6453fb9e2c1f\") " pod="opendatahub/maas-controller-bc865c477-q4c9z" Nov 03 13:46:44.923601 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:44.923555 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-bc865c477-q4c9z" Nov 03 13:46:45.211113 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:45.211080 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-bc865c477-q4c9z"] Nov 03 13:46:45.211855 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:46:45.211831 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75c9ed72_4fa6_4d71_a8c9_6453fb9e2c1f.slice/crio-1e6b42aa8d4fec85245cc86262af4c57130dc9a82f751efef62e51182166c2ff WatchSource:0}: Error finding container 1e6b42aa8d4fec85245cc86262af4c57130dc9a82f751efef62e51182166c2ff: Status 404 returned error can't find the container with id 1e6b42aa8d4fec85245cc86262af4c57130dc9a82f751efef62e51182166c2ff Nov 03 13:46:46.062047 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:46.061960 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-bc865c477-q4c9z" event={"ID":"75c9ed72-4fa6-4d71-a8c9-6453fb9e2c1f","Type":"ContainerStarted","Data":"7166b7485b42bd986c7f3e3f891319da0b5e0baa9a317f7ad541f7fb09d83cfe"} Nov 03 13:46:46.062047 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:46.062026 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-bc865c477-q4c9z" event={"ID":"75c9ed72-4fa6-4d71-a8c9-6453fb9e2c1f","Type":"ContainerStarted","Data":"1e6b42aa8d4fec85245cc86262af4c57130dc9a82f751efef62e51182166c2ff"} Nov 03 13:46:46.062402 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:46.062076 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-bc865c477-q4c9z" Nov 03 13:46:46.083827 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:46.083770 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-bc865c477-q4c9z" podStartSLOduration=1.675433472 podStartE2EDuration="2.083758123s" totalImagesPullingTime="408.324651ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:46:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:46:46.081979103 +0000 UTC m=+2286.090170380" watchObservedRunningTime="2026-11-03 13:46:46.083758123 +0000 UTC m=+2286.091949401" Nov 03 13:46:57.068477 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:46:57.068445 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-bc865c477-q4c9z" Nov 03 13:47:09.506910 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:47:09.506870 5024 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d7mwz" containerName="extractor" Nov 03 13:47:15.204404 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:47:15.204369 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-bsvp9"] Nov 03 13:47:15.204807 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:47:15.204556 5024 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-bsvp9" podUID="8f111884-464c-4369-88ce-d3aed661014d" containerName="manager" containerID="cri-o://c7c61e5bb5a7db43d92df0fb35b1660b3d05d089a68ad3508eb55866317de7b3" gracePeriod=10 Nov 03 13:47:15.459384 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:47:15.459318 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-bsvp9" Nov 03 13:47:15.591458 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:47:15.591424 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f111884-464c-4369-88ce-d3aed661014d-kube-api-access-qvcc4\" (UniqueName: \"kubernetes.io/projected/8f111884-464c-4369-88ce-d3aed661014d-kube-api-access-qvcc4\") pod \"8f111884-464c-4369-88ce-d3aed661014d\" (UID: \"8f111884-464c-4369-88ce-d3aed661014d\") " Nov 03 13:47:15.591625 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:47:15.591510 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f111884-464c-4369-88ce-d3aed661014d-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8f111884-464c-4369-88ce-d3aed661014d-extensions-socket-volume\") pod \"8f111884-464c-4369-88ce-d3aed661014d\" (UID: \"8f111884-464c-4369-88ce-d3aed661014d\") " Nov 03 13:47:15.591891 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:47:15.591868 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f111884-464c-4369-88ce-d3aed661014d-extensions-socket-volume" pod "8f111884-464c-4369-88ce-d3aed661014d" (UID: "8f111884-464c-4369-88ce-d3aed661014d"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:47:15.593515 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:47:15.593489 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f111884-464c-4369-88ce-d3aed661014d-kube-api-access-qvcc4" pod "8f111884-464c-4369-88ce-d3aed661014d" (UID: "8f111884-464c-4369-88ce-d3aed661014d"). InnerVolumeSpecName "kube-api-access-qvcc4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:47:15.692942 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:47:15.692910 5024 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8f111884-464c-4369-88ce-d3aed661014d-extensions-socket-volume\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:47:15.692942 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:47:15.692937 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qvcc4\" (UniqueName: \"kubernetes.io/projected/8f111884-464c-4369-88ce-d3aed661014d-kube-api-access-qvcc4\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:47:16.207362 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:47:16.207320 5024 generic.go:356] "Generic (PLEG): container finished" podID="8f111884-464c-4369-88ce-d3aed661014d" containerID="c7c61e5bb5a7db43d92df0fb35b1660b3d05d089a68ad3508eb55866317de7b3" exitCode=0 Nov 03 13:47:16.207834 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:47:16.207397 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-bsvp9" Nov 03 13:47:16.207834 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:47:16.207406 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-bsvp9" event={"ID":"8f111884-464c-4369-88ce-d3aed661014d","Type":"ContainerDied","Data":"c7c61e5bb5a7db43d92df0fb35b1660b3d05d089a68ad3508eb55866317de7b3"} Nov 03 13:47:16.207834 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:47:16.207485 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-bsvp9" event={"ID":"8f111884-464c-4369-88ce-d3aed661014d","Type":"ContainerDied","Data":"b584e9db6edc1062407abb81d6beb8f2950e4b8a0918d96f85f43c0744fc3a43"} Nov 03 13:47:16.207834 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:47:16.207509 5024 scope.go:122] "RemoveContainer" containerID="c7c61e5bb5a7db43d92df0fb35b1660b3d05d089a68ad3508eb55866317de7b3" Nov 03 13:47:16.221618 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:47:16.221598 5024 scope.go:122] "RemoveContainer" containerID="c7c61e5bb5a7db43d92df0fb35b1660b3d05d089a68ad3508eb55866317de7b3" Nov 03 13:47:16.221873 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:47:16.221851 5024 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7c61e5bb5a7db43d92df0fb35b1660b3d05d089a68ad3508eb55866317de7b3\": container with ID starting with c7c61e5bb5a7db43d92df0fb35b1660b3d05d089a68ad3508eb55866317de7b3 not found: ID does not exist" containerID="c7c61e5bb5a7db43d92df0fb35b1660b3d05d089a68ad3508eb55866317de7b3" Nov 03 13:47:16.221923 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:47:16.221880 5024 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7c61e5bb5a7db43d92df0fb35b1660b3d05d089a68ad3508eb55866317de7b3"} err="failed to get container status \"c7c61e5bb5a7db43d92df0fb35b1660b3d05d089a68ad3508eb55866317de7b3\": rpc error: code = NotFound desc = could not find container \"c7c61e5bb5a7db43d92df0fb35b1660b3d05d089a68ad3508eb55866317de7b3\": container with ID starting with c7c61e5bb5a7db43d92df0fb35b1660b3d05d089a68ad3508eb55866317de7b3 not found: ID does not exist" Nov 03 13:47:16.230669 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:47:16.230646 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-bsvp9"] Nov 03 13:47:16.236253 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:47:16.236232 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-bsvp9"] Nov 03 13:47:18.106773 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:47:18.106740 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8f111884-464c-4369-88ce-d3aed661014d" path="/var/lib/kubelet/pods/8f111884-464c-4369-88ce-d3aed661014d/volumes" Nov 03 13:47:39.506541 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:47:39.506498 5024 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d7mwz" containerName="extractor" Nov 03 13:48:09.506170 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:48:09.506086 5024 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d7mwz" containerName="extractor" Nov 03 13:48:16.263917 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:48:16.263874 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-gpqgh"] Nov 03 13:48:16.264319 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:48:16.264278 5024 cpu_manager.go:414] "containerMap: removing container" podUID="8f111884-464c-4369-88ce-d3aed661014d" containerName="manager" Nov 03 13:48:16.264319 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:48:16.264308 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f111884-464c-4369-88ce-d3aed661014d" containerName="manager" Nov 03 13:48:16.264415 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:48:16.264405 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f111884-464c-4369-88ce-d3aed661014d" containerName="manager" Nov 03 13:48:16.267471 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:48:16.267451 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-gpqgh" Nov 03 13:48:16.276981 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:48:16.276961 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-5qr2g\"" Nov 03 13:48:16.288184 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:48:16.288160 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-gpqgh"] Nov 03 13:48:16.374402 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:48:16.374362 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q84b8\" (UniqueName: \"kubernetes.io/projected/6178a6a9-df55-4825-9df9-32a7f2bccad8-kube-api-access-q84b8\") pod \"kuadrant-operator-controller-manager-678d87766f-gpqgh\" (UID: \"6178a6a9-df55-4825-9df9-32a7f2bccad8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-gpqgh" Nov 03 13:48:16.374603 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:48:16.374489 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6178a6a9-df55-4825-9df9-32a7f2bccad8-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-gpqgh\" (UID: \"6178a6a9-df55-4825-9df9-32a7f2bccad8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-gpqgh" Nov 03 13:48:16.475276 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:48:16.475238 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q84b8\" (UniqueName: \"kubernetes.io/projected/6178a6a9-df55-4825-9df9-32a7f2bccad8-kube-api-access-q84b8\") pod \"kuadrant-operator-controller-manager-678d87766f-gpqgh\" (UID: \"6178a6a9-df55-4825-9df9-32a7f2bccad8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-gpqgh" Nov 03 13:48:16.475526 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:48:16.475327 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6178a6a9-df55-4825-9df9-32a7f2bccad8-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-gpqgh\" (UID: \"6178a6a9-df55-4825-9df9-32a7f2bccad8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-gpqgh" Nov 03 13:48:16.475666 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:48:16.475648 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6178a6a9-df55-4825-9df9-32a7f2bccad8-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-gpqgh\" (UID: \"6178a6a9-df55-4825-9df9-32a7f2bccad8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-gpqgh" Nov 03 13:48:16.484770 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:48:16.484745 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q84b8\" (UniqueName: \"kubernetes.io/projected/6178a6a9-df55-4825-9df9-32a7f2bccad8-kube-api-access-q84b8\") pod \"kuadrant-operator-controller-manager-678d87766f-gpqgh\" (UID: \"6178a6a9-df55-4825-9df9-32a7f2bccad8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-gpqgh" Nov 03 13:48:16.579900 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:48:16.579814 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-gpqgh" Nov 03 13:48:16.863717 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:48:16.863687 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-gpqgh"] Nov 03 13:48:16.865113 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:48:16.865076 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6178a6a9_df55_4825_9df9_32a7f2bccad8.slice/crio-db737f67cd803020c0247df87afe87fdef4353aa9ab6cc310792dd1b88656795 WatchSource:0}: Error finding container db737f67cd803020c0247df87afe87fdef4353aa9ab6cc310792dd1b88656795: Status 404 returned error can't find the container with id db737f67cd803020c0247df87afe87fdef4353aa9ab6cc310792dd1b88656795 Nov 03 13:48:17.485708 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:48:17.485659 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-gpqgh" event={"ID":"6178a6a9-df55-4825-9df9-32a7f2bccad8","Type":"ContainerStarted","Data":"48547e2702bc8c8fac75f1779c5a173b0fb894479c167d6210657eeba282c88a"} Nov 03 13:48:17.486164 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:48:17.485722 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-gpqgh" event={"ID":"6178a6a9-df55-4825-9df9-32a7f2bccad8","Type":"ContainerStarted","Data":"db737f67cd803020c0247df87afe87fdef4353aa9ab6cc310792dd1b88656795"} Nov 03 13:48:17.486164 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:48:17.485845 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-gpqgh" Nov 03 13:48:17.512023 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:48:17.511974 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-gpqgh" podStartSLOduration=1.511960822 podStartE2EDuration="1.511960822s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:48:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:48:17.508502841 +0000 UTC m=+2377.516694117" watchObservedRunningTime="2026-11-03 13:48:17.511960822 +0000 UTC m=+2377.520152098" Nov 03 13:48:28.493126 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:48:28.493091 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-gpqgh" Nov 03 13:48:39.506193 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:48:39.506153 5024 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d7mwz" containerName="extractor" Nov 03 13:48:39.672444 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:48:39.672400 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-8jf5w"] Nov 03 13:48:39.675954 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:48:39.675932 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-8jf5w" Nov 03 13:48:39.682569 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:48:39.682543 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 13:48:39.682741 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:48:39.682582 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 13:48:39.682741 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:48:39.682626 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 13:48:39.682741 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:48:39.682667 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-phqfq\"" Nov 03 13:48:39.686647 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:48:39.685164 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-8jf5w"] Nov 03 13:48:39.701584 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:48:39.701560 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/406b9b9c-d55b-416b-8a28-6f301e0339f8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-8jf5w\" (UID: \"406b9b9c-d55b-416b-8a28-6f301e0339f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-8jf5w" Nov 03 13:48:39.701682 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:48:39.701623 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qn92l\" (UniqueName: \"kubernetes.io/projected/406b9b9c-d55b-416b-8a28-6f301e0339f8-kube-api-access-qn92l\") pod \"maas-api-revoke-sub-d576e68b-8jf5w\" (UID: \"406b9b9c-d55b-416b-8a28-6f301e0339f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-8jf5w" Nov 03 13:48:39.701750 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:48:39.701734 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/406b9b9c-d55b-416b-8a28-6f301e0339f8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-8jf5w\" (UID: \"406b9b9c-d55b-416b-8a28-6f301e0339f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-8jf5w" Nov 03 13:48:39.803090 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:48:39.802991 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/406b9b9c-d55b-416b-8a28-6f301e0339f8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-8jf5w\" (UID: \"406b9b9c-d55b-416b-8a28-6f301e0339f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-8jf5w" Nov 03 13:48:39.803277 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:48:39.803098 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qn92l\" (UniqueName: \"kubernetes.io/projected/406b9b9c-d55b-416b-8a28-6f301e0339f8-kube-api-access-qn92l\") pod \"maas-api-revoke-sub-d576e68b-8jf5w\" (UID: \"406b9b9c-d55b-416b-8a28-6f301e0339f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-8jf5w" Nov 03 13:48:39.803277 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:48:39.803137 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/406b9b9c-d55b-416b-8a28-6f301e0339f8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-8jf5w\" (UID: \"406b9b9c-d55b-416b-8a28-6f301e0339f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-8jf5w" Nov 03 13:48:39.803766 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:48:39.803737 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/406b9b9c-d55b-416b-8a28-6f301e0339f8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-8jf5w\" (UID: \"406b9b9c-d55b-416b-8a28-6f301e0339f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-8jf5w" Nov 03 13:48:39.805537 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:48:39.805515 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/406b9b9c-d55b-416b-8a28-6f301e0339f8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-8jf5w\" (UID: \"406b9b9c-d55b-416b-8a28-6f301e0339f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-8jf5w" Nov 03 13:48:39.812507 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:48:39.812480 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qn92l\" (UniqueName: \"kubernetes.io/projected/406b9b9c-d55b-416b-8a28-6f301e0339f8-kube-api-access-qn92l\") pod \"maas-api-revoke-sub-d576e68b-8jf5w\" (UID: \"406b9b9c-d55b-416b-8a28-6f301e0339f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-8jf5w" Nov 03 13:48:39.992322 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:48:39.992249 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-8jf5w" Nov 03 13:48:40.269808 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:48:40.269777 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-8jf5w"] Nov 03 13:48:40.271135 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:48:40.271107 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod406b9b9c_d55b_416b_8a28_6f301e0339f8.slice/crio-409f184d1ed6cae54e930738c60c101defd7a73d9424d2c57cb4bcecc6121790 WatchSource:0}: Error finding container 409f184d1ed6cae54e930738c60c101defd7a73d9424d2c57cb4bcecc6121790: Status 404 returned error can't find the container with id 409f184d1ed6cae54e930738c60c101defd7a73d9424d2c57cb4bcecc6121790 Nov 03 13:48:40.728154 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:48:40.728112 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-8jf5w" event={"ID":"406b9b9c-d55b-416b-8a28-6f301e0339f8","Type":"ContainerStarted","Data":"d804da81ae33b107c6cd37730c5c3b22374cf3e854c004000b04ba74cb257fdc"} Nov 03 13:48:40.728618 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:48:40.728193 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-8jf5w" event={"ID":"406b9b9c-d55b-416b-8a28-6f301e0339f8","Type":"ContainerStarted","Data":"409f184d1ed6cae54e930738c60c101defd7a73d9424d2c57cb4bcecc6121790"} Nov 03 13:48:40.749319 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:48:40.749258 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-8jf5w" podStartSLOduration=1.7492458050000002 podStartE2EDuration="1.749245805s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:48:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:48:40.745909395 +0000 UTC m=+2400.754100679" watchObservedRunningTime="2026-11-03 13:48:40.749245805 +0000 UTC m=+2400.757437082" Nov 03 13:48:41.415633 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:48:41.415602 5024 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6htht_cd91de08-084e-4667-b13f-2e07b3cff624/console-operator/1.log" Nov 03 13:48:41.429119 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:48:41.429055 5024 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6htht_cd91de08-084e-4667-b13f-2e07b3cff624/console-operator/1.log" Nov 03 13:48:41.734002 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:48:41.733910 5024 generic.go:356] "Generic (PLEG): container finished" podID="406b9b9c-d55b-416b-8a28-6f301e0339f8" containerID="d804da81ae33b107c6cd37730c5c3b22374cf3e854c004000b04ba74cb257fdc" exitCode=0 Nov 03 13:48:41.734437 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:48:41.734003 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-8jf5w" event={"ID":"406b9b9c-d55b-416b-8a28-6f301e0339f8","Type":"ContainerDied","Data":"d804da81ae33b107c6cd37730c5c3b22374cf3e854c004000b04ba74cb257fdc"} Nov 03 13:48:42.872593 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:48:42.872567 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-8jf5w" Nov 03 13:48:42.929303 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:48:42.929250 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/406b9b9c-d55b-416b-8a28-6f301e0339f8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/406b9b9c-d55b-416b-8a28-6f301e0339f8-maas-api-openshift-ca\") pod \"406b9b9c-d55b-416b-8a28-6f301e0339f8\" (UID: \"406b9b9c-d55b-416b-8a28-6f301e0339f8\") " Nov 03 13:48:42.929441 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:48:42.929416 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/406b9b9c-d55b-416b-8a28-6f301e0339f8-kube-api-access-qn92l\" (UniqueName: \"kubernetes.io/projected/406b9b9c-d55b-416b-8a28-6f301e0339f8-kube-api-access-qn92l\") pod \"406b9b9c-d55b-416b-8a28-6f301e0339f8\" (UID: \"406b9b9c-d55b-416b-8a28-6f301e0339f8\") " Nov 03 13:48:42.929494 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:48:42.929448 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/406b9b9c-d55b-416b-8a28-6f301e0339f8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/406b9b9c-d55b-416b-8a28-6f301e0339f8-maas-api-xks-ca\") pod \"406b9b9c-d55b-416b-8a28-6f301e0339f8\" (UID: \"406b9b9c-d55b-416b-8a28-6f301e0339f8\") " Nov 03 13:48:42.929641 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:48:42.929614 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/406b9b9c-d55b-416b-8a28-6f301e0339f8-maas-api-openshift-ca" pod "406b9b9c-d55b-416b-8a28-6f301e0339f8" (UID: "406b9b9c-d55b-416b-8a28-6f301e0339f8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:48:42.931546 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:48:42.931517 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/406b9b9c-d55b-416b-8a28-6f301e0339f8-kube-api-access-qn92l" pod "406b9b9c-d55b-416b-8a28-6f301e0339f8" (UID: "406b9b9c-d55b-416b-8a28-6f301e0339f8"). InnerVolumeSpecName "kube-api-access-qn92l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:48:42.931667 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:48:42.931608 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/406b9b9c-d55b-416b-8a28-6f301e0339f8-maas-api-xks-ca" pod "406b9b9c-d55b-416b-8a28-6f301e0339f8" (UID: "406b9b9c-d55b-416b-8a28-6f301e0339f8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 13:48:43.030364 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:48:43.030269 5024 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/406b9b9c-d55b-416b-8a28-6f301e0339f8-maas-api-openshift-ca\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:48:43.030364 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:48:43.030320 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qn92l\" (UniqueName: \"kubernetes.io/projected/406b9b9c-d55b-416b-8a28-6f301e0339f8-kube-api-access-qn92l\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:48:43.030364 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:48:43.030331 5024 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/406b9b9c-d55b-416b-8a28-6f301e0339f8-maas-api-xks-ca\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:48:43.744253 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:48:43.744223 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-8jf5w" Nov 03 13:48:43.744457 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:48:43.744213 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-8jf5w" event={"ID":"406b9b9c-d55b-416b-8a28-6f301e0339f8","Type":"ContainerDied","Data":"409f184d1ed6cae54e930738c60c101defd7a73d9424d2c57cb4bcecc6121790"} Nov 03 13:48:43.744457 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:48:43.744365 5024 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="409f184d1ed6cae54e930738c60c101defd7a73d9424d2c57cb4bcecc6121790" Nov 03 13:49:09.521860 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:49:09.521787 5024 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d7mwz" containerName="extractor" Nov 03 13:49:12.747194 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:49:12.747153 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-751e2187-sjv8j"] Nov 03 13:49:12.747809 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:49:12.747665 5024 cpu_manager.go:414] "containerMap: removing container" podUID="406b9b9c-d55b-416b-8a28-6f301e0339f8" containerName="revoke-keys" Nov 03 13:49:12.747809 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:49:12.747687 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="406b9b9c-d55b-416b-8a28-6f301e0339f8" containerName="revoke-keys" Nov 03 13:49:12.747809 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:49:12.747772 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="406b9b9c-d55b-416b-8a28-6f301e0339f8" containerName="revoke-keys" Nov 03 13:49:12.750930 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:49:12.750910 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-751e2187-sjv8j" Nov 03 13:49:12.758125 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:49:12.758095 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 13:49:12.758482 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:49:12.758459 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-phqfq\"" Nov 03 13:49:12.758677 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:49:12.758655 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 13:49:12.758758 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:49:12.758662 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 13:49:12.759152 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:49:12.758967 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-751e2187-sjv8j"] Nov 03 13:49:12.876912 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:49:12.876867 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d5686fa-f7d5-41ff-8763-58e45c26fc70-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-751e2187-sjv8j\" (UID: \"8d5686fa-f7d5-41ff-8763-58e45c26fc70\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-751e2187-sjv8j" Nov 03 13:49:12.877134 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:49:12.876927 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d5686fa-f7d5-41ff-8763-58e45c26fc70-maas-api-xks-ca\") pod \"maas-api-revoke-sub-751e2187-sjv8j\" (UID: \"8d5686fa-f7d5-41ff-8763-58e45c26fc70\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-751e2187-sjv8j" Nov 03 13:49:12.877134 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:49:12.877042 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2vmd5\" (UniqueName: \"kubernetes.io/projected/8d5686fa-f7d5-41ff-8763-58e45c26fc70-kube-api-access-2vmd5\") pod \"maas-api-revoke-sub-751e2187-sjv8j\" (UID: \"8d5686fa-f7d5-41ff-8763-58e45c26fc70\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-751e2187-sjv8j" Nov 03 13:49:12.978567 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:49:12.978535 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d5686fa-f7d5-41ff-8763-58e45c26fc70-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-751e2187-sjv8j\" (UID: \"8d5686fa-f7d5-41ff-8763-58e45c26fc70\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-751e2187-sjv8j" Nov 03 13:49:12.978763 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:49:12.978581 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d5686fa-f7d5-41ff-8763-58e45c26fc70-maas-api-xks-ca\") pod \"maas-api-revoke-sub-751e2187-sjv8j\" (UID: \"8d5686fa-f7d5-41ff-8763-58e45c26fc70\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-751e2187-sjv8j" Nov 03 13:49:12.978763 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:49:12.978695 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vmd5\" (UniqueName: \"kubernetes.io/projected/8d5686fa-f7d5-41ff-8763-58e45c26fc70-kube-api-access-2vmd5\") pod \"maas-api-revoke-sub-751e2187-sjv8j\" (UID: \"8d5686fa-f7d5-41ff-8763-58e45c26fc70\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-751e2187-sjv8j" Nov 03 13:49:12.979315 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:49:12.979271 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d5686fa-f7d5-41ff-8763-58e45c26fc70-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-751e2187-sjv8j\" (UID: \"8d5686fa-f7d5-41ff-8763-58e45c26fc70\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-751e2187-sjv8j" Nov 03 13:49:12.980864 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:49:12.980833 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d5686fa-f7d5-41ff-8763-58e45c26fc70-maas-api-xks-ca\") pod \"maas-api-revoke-sub-751e2187-sjv8j\" (UID: \"8d5686fa-f7d5-41ff-8763-58e45c26fc70\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-751e2187-sjv8j" Nov 03 13:49:12.988917 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:49:12.988887 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vmd5\" (UniqueName: \"kubernetes.io/projected/8d5686fa-f7d5-41ff-8763-58e45c26fc70-kube-api-access-2vmd5\") pod \"maas-api-revoke-sub-751e2187-sjv8j\" (UID: \"8d5686fa-f7d5-41ff-8763-58e45c26fc70\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-751e2187-sjv8j" Nov 03 13:49:13.066160 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:49:13.066070 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-751e2187-sjv8j" Nov 03 13:49:13.343585 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:49:13.343552 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-751e2187-sjv8j"] Nov 03 13:49:13.344855 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:49:13.344827 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d5686fa_f7d5_41ff_8763_58e45c26fc70.slice/crio-a656ae5eb5721d53151571a713237d62f01f67be8dddbca9b4b75bf821d80dfa WatchSource:0}: Error finding container a656ae5eb5721d53151571a713237d62f01f67be8dddbca9b4b75bf821d80dfa: Status 404 returned error can't find the container with id a656ae5eb5721d53151571a713237d62f01f67be8dddbca9b4b75bf821d80dfa Nov 03 13:49:14.027619 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:49:14.027582 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-751e2187-sjv8j" event={"ID":"8d5686fa-f7d5-41ff-8763-58e45c26fc70","Type":"ContainerStarted","Data":"4394f8bb87b2acae3239088442fafe4c488da85ea60f858dc21248a0c33de53a"} Nov 03 13:49:14.028032 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:49:14.027640 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-751e2187-sjv8j" event={"ID":"8d5686fa-f7d5-41ff-8763-58e45c26fc70","Type":"ContainerStarted","Data":"a656ae5eb5721d53151571a713237d62f01f67be8dddbca9b4b75bf821d80dfa"} Nov 03 13:49:14.048538 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:49:14.048488 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-751e2187-sjv8j" podStartSLOduration=2.048476359 podStartE2EDuration="2.048476359s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:49:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:49:14.047131789 +0000 UTC m=+2434.055323066" watchObservedRunningTime="2026-11-03 13:49:14.048476359 +0000 UTC m=+2434.056667635" Nov 03 13:49:15.034924 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:49:15.034884 5024 generic.go:356] "Generic (PLEG): container finished" podID="8d5686fa-f7d5-41ff-8763-58e45c26fc70" containerID="4394f8bb87b2acae3239088442fafe4c488da85ea60f858dc21248a0c33de53a" exitCode=0 Nov 03 13:49:15.034924 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:49:15.034927 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-751e2187-sjv8j" event={"ID":"8d5686fa-f7d5-41ff-8763-58e45c26fc70","Type":"ContainerDied","Data":"4394f8bb87b2acae3239088442fafe4c488da85ea60f858dc21248a0c33de53a"} Nov 03 13:49:16.169476 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:49:16.169447 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-751e2187-sjv8j" Nov 03 13:49:16.313028 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:49:16.312932 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8d5686fa-f7d5-41ff-8763-58e45c26fc70-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d5686fa-f7d5-41ff-8763-58e45c26fc70-maas-api-xks-ca\") pod \"8d5686fa-f7d5-41ff-8763-58e45c26fc70\" (UID: \"8d5686fa-f7d5-41ff-8763-58e45c26fc70\") " Nov 03 13:49:16.313212 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:49:16.313074 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d5686fa-f7d5-41ff-8763-58e45c26fc70-kube-api-access-2vmd5\" (UniqueName: \"kubernetes.io/projected/8d5686fa-f7d5-41ff-8763-58e45c26fc70-kube-api-access-2vmd5\") pod \"8d5686fa-f7d5-41ff-8763-58e45c26fc70\" (UID: \"8d5686fa-f7d5-41ff-8763-58e45c26fc70\") " Nov 03 13:49:16.313212 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:49:16.313117 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d5686fa-f7d5-41ff-8763-58e45c26fc70-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d5686fa-f7d5-41ff-8763-58e45c26fc70-maas-api-openshift-ca\") pod \"8d5686fa-f7d5-41ff-8763-58e45c26fc70\" (UID: \"8d5686fa-f7d5-41ff-8763-58e45c26fc70\") " Nov 03 13:49:16.313546 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:49:16.313520 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d5686fa-f7d5-41ff-8763-58e45c26fc70-maas-api-openshift-ca" pod "8d5686fa-f7d5-41ff-8763-58e45c26fc70" (UID: "8d5686fa-f7d5-41ff-8763-58e45c26fc70"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:49:16.315001 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:49:16.314972 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d5686fa-f7d5-41ff-8763-58e45c26fc70-maas-api-xks-ca" pod "8d5686fa-f7d5-41ff-8763-58e45c26fc70" (UID: "8d5686fa-f7d5-41ff-8763-58e45c26fc70"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 13:49:16.315208 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:49:16.315186 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d5686fa-f7d5-41ff-8763-58e45c26fc70-kube-api-access-2vmd5" pod "8d5686fa-f7d5-41ff-8763-58e45c26fc70" (UID: "8d5686fa-f7d5-41ff-8763-58e45c26fc70"). InnerVolumeSpecName "kube-api-access-2vmd5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:49:16.414427 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:49:16.414397 5024 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d5686fa-f7d5-41ff-8763-58e45c26fc70-maas-api-xks-ca\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:49:16.414547 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:49:16.414428 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2vmd5\" (UniqueName: \"kubernetes.io/projected/8d5686fa-f7d5-41ff-8763-58e45c26fc70-kube-api-access-2vmd5\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:49:16.414547 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:49:16.414446 5024 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d5686fa-f7d5-41ff-8763-58e45c26fc70-maas-api-openshift-ca\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:49:17.045017 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:49:17.044985 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-751e2187-sjv8j" Nov 03 13:49:17.045216 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:49:17.045018 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-751e2187-sjv8j" event={"ID":"8d5686fa-f7d5-41ff-8763-58e45c26fc70","Type":"ContainerDied","Data":"a656ae5eb5721d53151571a713237d62f01f67be8dddbca9b4b75bf821d80dfa"} Nov 03 13:49:17.045216 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:49:17.045066 5024 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a656ae5eb5721d53151571a713237d62f01f67be8dddbca9b4b75bf821d80dfa" Nov 03 13:49:23.036052 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:49:23.036013 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cb53c5df-v52x2"] Nov 03 13:49:23.036536 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:49:23.036518 5024 cpu_manager.go:414] "containerMap: removing container" podUID="8d5686fa-f7d5-41ff-8763-58e45c26fc70" containerName="revoke-keys" Nov 03 13:49:23.036626 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:49:23.036537 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d5686fa-f7d5-41ff-8763-58e45c26fc70" containerName="revoke-keys" Nov 03 13:49:23.036710 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:49:23.036639 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d5686fa-f7d5-41ff-8763-58e45c26fc70" containerName="revoke-keys" Nov 03 13:49:23.039986 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:49:23.039963 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb53c5df-v52x2" Nov 03 13:49:23.047458 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:49:23.047430 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cb53c5df-v52x2"] Nov 03 13:49:23.047794 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:49:23.047654 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 13:49:23.047927 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:49:23.047691 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-phqfq\"" Nov 03 13:49:23.047927 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:49:23.047762 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 13:49:23.047927 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:49:23.047768 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 13:49:23.074897 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:49:23.074861 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-48rq9\" (UniqueName: \"kubernetes.io/projected/413b4679-30fe-421c-a1ef-4b463cc1603e-kube-api-access-48rq9\") pod \"maas-api-revoke-sub-cb53c5df-v52x2\" (UID: \"413b4679-30fe-421c-a1ef-4b463cc1603e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb53c5df-v52x2" Nov 03 13:49:23.075047 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:49:23.074993 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/413b4679-30fe-421c-a1ef-4b463cc1603e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cb53c5df-v52x2\" (UID: \"413b4679-30fe-421c-a1ef-4b463cc1603e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb53c5df-v52x2" Nov 03 13:49:23.075177 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:49:23.075157 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/413b4679-30fe-421c-a1ef-4b463cc1603e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cb53c5df-v52x2\" (UID: \"413b4679-30fe-421c-a1ef-4b463cc1603e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb53c5df-v52x2" Nov 03 13:49:23.176410 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:49:23.176373 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/413b4679-30fe-421c-a1ef-4b463cc1603e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cb53c5df-v52x2\" (UID: \"413b4679-30fe-421c-a1ef-4b463cc1603e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb53c5df-v52x2" Nov 03 13:49:23.176584 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:49:23.176514 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48rq9\" (UniqueName: \"kubernetes.io/projected/413b4679-30fe-421c-a1ef-4b463cc1603e-kube-api-access-48rq9\") pod \"maas-api-revoke-sub-cb53c5df-v52x2\" (UID: \"413b4679-30fe-421c-a1ef-4b463cc1603e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb53c5df-v52x2" Nov 03 13:49:23.176671 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:49:23.176582 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/413b4679-30fe-421c-a1ef-4b463cc1603e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cb53c5df-v52x2\" (UID: \"413b4679-30fe-421c-a1ef-4b463cc1603e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb53c5df-v52x2" Nov 03 13:49:23.177315 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:49:23.177269 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/413b4679-30fe-421c-a1ef-4b463cc1603e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cb53c5df-v52x2\" (UID: \"413b4679-30fe-421c-a1ef-4b463cc1603e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb53c5df-v52x2" Nov 03 13:49:23.179094 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:49:23.179068 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/413b4679-30fe-421c-a1ef-4b463cc1603e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cb53c5df-v52x2\" (UID: \"413b4679-30fe-421c-a1ef-4b463cc1603e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb53c5df-v52x2" Nov 03 13:49:23.186633 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:49:23.186608 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-48rq9\" (UniqueName: \"kubernetes.io/projected/413b4679-30fe-421c-a1ef-4b463cc1603e-kube-api-access-48rq9\") pod \"maas-api-revoke-sub-cb53c5df-v52x2\" (UID: \"413b4679-30fe-421c-a1ef-4b463cc1603e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb53c5df-v52x2" Nov 03 13:49:23.356191 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:49:23.356156 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb53c5df-v52x2" Nov 03 13:49:23.637090 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:49:23.637059 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cb53c5df-v52x2"] Nov 03 13:49:23.638804 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:49:23.638768 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod413b4679_30fe_421c_a1ef_4b463cc1603e.slice/crio-d05a72c0a62e2b828650817a127c6ac12cf246f237a9153a76120cf8114788e5 WatchSource:0}: Error finding container d05a72c0a62e2b828650817a127c6ac12cf246f237a9153a76120cf8114788e5: Status 404 returned error can't find the container with id d05a72c0a62e2b828650817a127c6ac12cf246f237a9153a76120cf8114788e5 Nov 03 13:49:24.082148 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:49:24.082053 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb53c5df-v52x2" event={"ID":"413b4679-30fe-421c-a1ef-4b463cc1603e","Type":"ContainerStarted","Data":"3faca8f151cfc8ce6630be326063f7f218d1bfe51bc6a3e812b14953373632ed"} Nov 03 13:49:24.082148 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:49:24.082124 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb53c5df-v52x2" event={"ID":"413b4679-30fe-421c-a1ef-4b463cc1603e","Type":"ContainerStarted","Data":"d05a72c0a62e2b828650817a127c6ac12cf246f237a9153a76120cf8114788e5"} Nov 03 13:49:24.102193 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:49:24.102132 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb53c5df-v52x2" podStartSLOduration=1.102117352 podStartE2EDuration="1.102117352s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:49:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:49:24.099565099 +0000 UTC m=+2444.107756377" watchObservedRunningTime="2026-11-03 13:49:24.102117352 +0000 UTC m=+2444.110308629" Nov 03 13:49:25.091929 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:49:25.091808 5024 generic.go:356] "Generic (PLEG): container finished" podID="413b4679-30fe-421c-a1ef-4b463cc1603e" containerID="3faca8f151cfc8ce6630be326063f7f218d1bfe51bc6a3e812b14953373632ed" exitCode=0 Nov 03 13:49:25.091929 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:49:25.091863 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb53c5df-v52x2" event={"ID":"413b4679-30fe-421c-a1ef-4b463cc1603e","Type":"ContainerDied","Data":"3faca8f151cfc8ce6630be326063f7f218d1bfe51bc6a3e812b14953373632ed"} Nov 03 13:49:26.966465 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:49:26.966438 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb53c5df-v52x2" Nov 03 13:49:27.037710 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:49:27.037668 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/413b4679-30fe-421c-a1ef-4b463cc1603e-kube-api-access-48rq9\" (UniqueName: \"kubernetes.io/projected/413b4679-30fe-421c-a1ef-4b463cc1603e-kube-api-access-48rq9\") pod \"413b4679-30fe-421c-a1ef-4b463cc1603e\" (UID: \"413b4679-30fe-421c-a1ef-4b463cc1603e\") " Nov 03 13:49:27.059509 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:49:27.059471 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/413b4679-30fe-421c-a1ef-4b463cc1603e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/413b4679-30fe-421c-a1ef-4b463cc1603e-maas-api-xks-ca\") pod \"413b4679-30fe-421c-a1ef-4b463cc1603e\" (UID: \"413b4679-30fe-421c-a1ef-4b463cc1603e\") " Nov 03 13:49:27.059729 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:49:27.059705 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/413b4679-30fe-421c-a1ef-4b463cc1603e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/413b4679-30fe-421c-a1ef-4b463cc1603e-maas-api-openshift-ca\") pod \"413b4679-30fe-421c-a1ef-4b463cc1603e\" (UID: \"413b4679-30fe-421c-a1ef-4b463cc1603e\") " Nov 03 13:49:27.060594 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:49:27.060570 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/413b4679-30fe-421c-a1ef-4b463cc1603e-maas-api-openshift-ca" pod "413b4679-30fe-421c-a1ef-4b463cc1603e" (UID: "413b4679-30fe-421c-a1ef-4b463cc1603e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:49:27.064665 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:49:27.064643 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/413b4679-30fe-421c-a1ef-4b463cc1603e-maas-api-xks-ca" pod "413b4679-30fe-421c-a1ef-4b463cc1603e" (UID: "413b4679-30fe-421c-a1ef-4b463cc1603e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 13:49:27.096744 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:49:27.096716 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/413b4679-30fe-421c-a1ef-4b463cc1603e-kube-api-access-48rq9" pod "413b4679-30fe-421c-a1ef-4b463cc1603e" (UID: "413b4679-30fe-421c-a1ef-4b463cc1603e"). InnerVolumeSpecName "kube-api-access-48rq9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:49:27.167199 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:49:27.167140 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-48rq9\" (UniqueName: \"kubernetes.io/projected/413b4679-30fe-421c-a1ef-4b463cc1603e-kube-api-access-48rq9\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:49:27.190454 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:49:27.190373 5024 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/413b4679-30fe-421c-a1ef-4b463cc1603e-maas-api-xks-ca\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:49:27.190741 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:49:27.190719 5024 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/413b4679-30fe-421c-a1ef-4b463cc1603e-maas-api-openshift-ca\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:49:27.320970 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:49:27.320911 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb53c5df-v52x2" event={"ID":"413b4679-30fe-421c-a1ef-4b463cc1603e","Type":"ContainerDied","Data":"d05a72c0a62e2b828650817a127c6ac12cf246f237a9153a76120cf8114788e5"} Nov 03 13:49:27.325312 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:49:27.321242 5024 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d05a72c0a62e2b828650817a127c6ac12cf246f237a9153a76120cf8114788e5" Nov 03 13:49:27.325511 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:49:27.322378 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb53c5df-v52x2" Nov 03 13:49:33.251295 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:49:33.251248 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-788f6f66-8ttzx"] Nov 03 13:49:33.251696 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:49:33.251681 5024 cpu_manager.go:414] "containerMap: removing container" podUID="413b4679-30fe-421c-a1ef-4b463cc1603e" containerName="revoke-keys" Nov 03 13:49:33.251739 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:49:33.251698 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="413b4679-30fe-421c-a1ef-4b463cc1603e" containerName="revoke-keys" Nov 03 13:49:33.251803 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:49:33.251794 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="413b4679-30fe-421c-a1ef-4b463cc1603e" containerName="revoke-keys" Nov 03 13:49:33.254642 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:49:33.254619 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-788f6f66-8ttzx" Nov 03 13:49:33.259324 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:49:33.259030 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 13:49:33.261456 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:49:33.261412 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-phqfq\"" Nov 03 13:49:33.261727 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:49:33.261578 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 13:49:33.261727 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:49:33.261681 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 13:49:33.264247 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:49:33.264225 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-788f6f66-8ttzx"] Nov 03 13:49:33.383200 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:49:33.383167 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b7prr\" (UniqueName: \"kubernetes.io/projected/58aa17f4-75c7-410b-ab1f-e05fdf6c626d-kube-api-access-b7prr\") pod \"maas-api-revoke-sub-788f6f66-8ttzx\" (UID: \"58aa17f4-75c7-410b-ab1f-e05fdf6c626d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-788f6f66-8ttzx" Nov 03 13:49:33.383200 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:49:33.383203 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58aa17f4-75c7-410b-ab1f-e05fdf6c626d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-788f6f66-8ttzx\" (UID: \"58aa17f4-75c7-410b-ab1f-e05fdf6c626d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-788f6f66-8ttzx" Nov 03 13:49:33.383450 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:49:33.383292 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58aa17f4-75c7-410b-ab1f-e05fdf6c626d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-788f6f66-8ttzx\" (UID: \"58aa17f4-75c7-410b-ab1f-e05fdf6c626d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-788f6f66-8ttzx" Nov 03 13:49:33.484692 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:49:33.484650 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7prr\" (UniqueName: \"kubernetes.io/projected/58aa17f4-75c7-410b-ab1f-e05fdf6c626d-kube-api-access-b7prr\") pod \"maas-api-revoke-sub-788f6f66-8ttzx\" (UID: \"58aa17f4-75c7-410b-ab1f-e05fdf6c626d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-788f6f66-8ttzx" Nov 03 13:49:33.484909 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:49:33.484702 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58aa17f4-75c7-410b-ab1f-e05fdf6c626d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-788f6f66-8ttzx\" (UID: \"58aa17f4-75c7-410b-ab1f-e05fdf6c626d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-788f6f66-8ttzx" Nov 03 13:49:33.484909 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:49:33.484739 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58aa17f4-75c7-410b-ab1f-e05fdf6c626d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-788f6f66-8ttzx\" (UID: \"58aa17f4-75c7-410b-ab1f-e05fdf6c626d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-788f6f66-8ttzx" Nov 03 13:49:33.485462 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:49:33.485436 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58aa17f4-75c7-410b-ab1f-e05fdf6c626d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-788f6f66-8ttzx\" (UID: \"58aa17f4-75c7-410b-ab1f-e05fdf6c626d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-788f6f66-8ttzx" Nov 03 13:49:33.487171 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:49:33.487146 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58aa17f4-75c7-410b-ab1f-e05fdf6c626d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-788f6f66-8ttzx\" (UID: \"58aa17f4-75c7-410b-ab1f-e05fdf6c626d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-788f6f66-8ttzx" Nov 03 13:49:33.493906 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:49:33.493882 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7prr\" (UniqueName: \"kubernetes.io/projected/58aa17f4-75c7-410b-ab1f-e05fdf6c626d-kube-api-access-b7prr\") pod \"maas-api-revoke-sub-788f6f66-8ttzx\" (UID: \"58aa17f4-75c7-410b-ab1f-e05fdf6c626d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-788f6f66-8ttzx" Nov 03 13:49:33.569960 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:49:33.569883 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-788f6f66-8ttzx" Nov 03 13:49:33.844767 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:49:33.844737 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-788f6f66-8ttzx"] Nov 03 13:49:33.846537 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:49:33.846491 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58aa17f4_75c7_410b_ab1f_e05fdf6c626d.slice/crio-74af3b99d7cd9b7e339e918fd5e9dbe011294e0877b05a17165e635282948582 WatchSource:0}: Error finding container 74af3b99d7cd9b7e339e918fd5e9dbe011294e0877b05a17165e635282948582: Status 404 returned error can't find the container with id 74af3b99d7cd9b7e339e918fd5e9dbe011294e0877b05a17165e635282948582 Nov 03 13:49:34.357742 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:49:34.357703 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-788f6f66-8ttzx" event={"ID":"58aa17f4-75c7-410b-ab1f-e05fdf6c626d","Type":"ContainerStarted","Data":"60baf7c2f3d913f953437449d73e1294f14168bd4ecb40323a483f704679fbfb"} Nov 03 13:49:34.358158 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:49:34.357758 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-788f6f66-8ttzx" event={"ID":"58aa17f4-75c7-410b-ab1f-e05fdf6c626d","Type":"ContainerStarted","Data":"74af3b99d7cd9b7e339e918fd5e9dbe011294e0877b05a17165e635282948582"} Nov 03 13:49:34.377969 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:49:34.377922 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-788f6f66-8ttzx" podStartSLOduration=1.377908498 podStartE2EDuration="1.377908498s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:49:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:49:34.377313269 +0000 UTC m=+2454.385504549" watchObservedRunningTime="2026-11-03 13:49:34.377908498 +0000 UTC m=+2454.386099801" Nov 03 13:49:35.364799 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:49:35.364752 5024 generic.go:356] "Generic (PLEG): container finished" podID="58aa17f4-75c7-410b-ab1f-e05fdf6c626d" containerID="60baf7c2f3d913f953437449d73e1294f14168bd4ecb40323a483f704679fbfb" exitCode=0 Nov 03 13:49:35.365308 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:49:35.364834 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-788f6f66-8ttzx" event={"ID":"58aa17f4-75c7-410b-ab1f-e05fdf6c626d","Type":"ContainerDied","Data":"60baf7c2f3d913f953437449d73e1294f14168bd4ecb40323a483f704679fbfb"} Nov 03 13:49:36.506986 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:49:36.506957 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-788f6f66-8ttzx" Nov 03 13:49:36.615214 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:49:36.615180 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/58aa17f4-75c7-410b-ab1f-e05fdf6c626d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58aa17f4-75c7-410b-ab1f-e05fdf6c626d-maas-api-xks-ca\") pod \"58aa17f4-75c7-410b-ab1f-e05fdf6c626d\" (UID: \"58aa17f4-75c7-410b-ab1f-e05fdf6c626d\") " Nov 03 13:49:36.615420 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:49:36.615243 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/58aa17f4-75c7-410b-ab1f-e05fdf6c626d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58aa17f4-75c7-410b-ab1f-e05fdf6c626d-maas-api-openshift-ca\") pod \"58aa17f4-75c7-410b-ab1f-e05fdf6c626d\" (UID: \"58aa17f4-75c7-410b-ab1f-e05fdf6c626d\") " Nov 03 13:49:36.615420 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:49:36.615336 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58aa17f4-75c7-410b-ab1f-e05fdf6c626d-kube-api-access-b7prr\" (UniqueName: \"kubernetes.io/projected/58aa17f4-75c7-410b-ab1f-e05fdf6c626d-kube-api-access-b7prr\") pod \"58aa17f4-75c7-410b-ab1f-e05fdf6c626d\" (UID: \"58aa17f4-75c7-410b-ab1f-e05fdf6c626d\") " Nov 03 13:49:36.615666 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:49:36.615643 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58aa17f4-75c7-410b-ab1f-e05fdf6c626d-maas-api-openshift-ca" pod "58aa17f4-75c7-410b-ab1f-e05fdf6c626d" (UID: "58aa17f4-75c7-410b-ab1f-e05fdf6c626d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:49:36.615783 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:49:36.615767 5024 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58aa17f4-75c7-410b-ab1f-e05fdf6c626d-maas-api-openshift-ca\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:49:36.617365 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:49:36.617338 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/58aa17f4-75c7-410b-ab1f-e05fdf6c626d-maas-api-xks-ca" pod "58aa17f4-75c7-410b-ab1f-e05fdf6c626d" (UID: "58aa17f4-75c7-410b-ab1f-e05fdf6c626d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 13:49:36.617476 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:49:36.617375 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58aa17f4-75c7-410b-ab1f-e05fdf6c626d-kube-api-access-b7prr" pod "58aa17f4-75c7-410b-ab1f-e05fdf6c626d" (UID: "58aa17f4-75c7-410b-ab1f-e05fdf6c626d"). InnerVolumeSpecName "kube-api-access-b7prr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:49:36.717059 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:49:36.717026 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b7prr\" (UniqueName: \"kubernetes.io/projected/58aa17f4-75c7-410b-ab1f-e05fdf6c626d-kube-api-access-b7prr\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:49:36.717059 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:49:36.717054 5024 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58aa17f4-75c7-410b-ab1f-e05fdf6c626d-maas-api-xks-ca\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:49:37.383427 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:49:37.383389 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-788f6f66-8ttzx" event={"ID":"58aa17f4-75c7-410b-ab1f-e05fdf6c626d","Type":"ContainerDied","Data":"74af3b99d7cd9b7e339e918fd5e9dbe011294e0877b05a17165e635282948582"} Nov 03 13:49:37.383624 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:49:37.383461 5024 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="74af3b99d7cd9b7e339e918fd5e9dbe011294e0877b05a17165e635282948582" Nov 03 13:49:37.383624 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:49:37.383405 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-788f6f66-8ttzx" Nov 03 13:49:39.506427 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:49:39.506391 5024 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d7mwz" containerName="extractor" Nov 03 13:49:43.488109 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:49:43.488066 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5076e05a-jgdkf"] Nov 03 13:49:43.488750 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:49:43.488726 5024 cpu_manager.go:414] "containerMap: removing container" podUID="58aa17f4-75c7-410b-ab1f-e05fdf6c626d" containerName="revoke-keys" Nov 03 13:49:43.488839 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:49:43.488752 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58aa17f4-75c7-410b-ab1f-e05fdf6c626d" containerName="revoke-keys" Nov 03 13:49:43.488920 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:49:43.488860 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="58aa17f4-75c7-410b-ab1f-e05fdf6c626d" containerName="revoke-keys" Nov 03 13:49:43.492640 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:49:43.492616 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5076e05a-jgdkf" Nov 03 13:49:43.498668 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:49:43.498640 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5076e05a-jgdkf"] Nov 03 13:49:43.498998 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:49:43.498972 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 13:49:43.498998 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:49:43.498919 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 13:49:43.500915 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:49:43.500882 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 13:49:43.502009 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:49:43.501020 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-phqfq\"" Nov 03 13:49:43.581310 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:49:43.581251 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6gmf\" (UniqueName: \"kubernetes.io/projected/4774b3fb-9cf8-4dda-a12c-89290fcf0ea4-kube-api-access-w6gmf\") pod \"maas-api-revoke-sub-5076e05a-jgdkf\" (UID: \"4774b3fb-9cf8-4dda-a12c-89290fcf0ea4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5076e05a-jgdkf" Nov 03 13:49:43.581496 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:49:43.581375 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4774b3fb-9cf8-4dda-a12c-89290fcf0ea4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5076e05a-jgdkf\" (UID: \"4774b3fb-9cf8-4dda-a12c-89290fcf0ea4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5076e05a-jgdkf" Nov 03 13:49:43.581496 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:49:43.581425 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4774b3fb-9cf8-4dda-a12c-89290fcf0ea4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5076e05a-jgdkf\" (UID: \"4774b3fb-9cf8-4dda-a12c-89290fcf0ea4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5076e05a-jgdkf" Nov 03 13:49:43.682669 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:49:43.682629 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6gmf\" (UniqueName: \"kubernetes.io/projected/4774b3fb-9cf8-4dda-a12c-89290fcf0ea4-kube-api-access-w6gmf\") pod \"maas-api-revoke-sub-5076e05a-jgdkf\" (UID: \"4774b3fb-9cf8-4dda-a12c-89290fcf0ea4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5076e05a-jgdkf" Nov 03 13:49:43.682876 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:49:43.682721 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4774b3fb-9cf8-4dda-a12c-89290fcf0ea4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5076e05a-jgdkf\" (UID: \"4774b3fb-9cf8-4dda-a12c-89290fcf0ea4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5076e05a-jgdkf" Nov 03 13:49:43.682876 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:49:43.682772 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4774b3fb-9cf8-4dda-a12c-89290fcf0ea4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5076e05a-jgdkf\" (UID: \"4774b3fb-9cf8-4dda-a12c-89290fcf0ea4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5076e05a-jgdkf" Nov 03 13:49:43.683444 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:49:43.683413 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4774b3fb-9cf8-4dda-a12c-89290fcf0ea4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5076e05a-jgdkf\" (UID: \"4774b3fb-9cf8-4dda-a12c-89290fcf0ea4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5076e05a-jgdkf" Nov 03 13:49:43.685168 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:49:43.685142 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4774b3fb-9cf8-4dda-a12c-89290fcf0ea4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5076e05a-jgdkf\" (UID: \"4774b3fb-9cf8-4dda-a12c-89290fcf0ea4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5076e05a-jgdkf" Nov 03 13:49:43.692545 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:49:43.692520 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6gmf\" (UniqueName: \"kubernetes.io/projected/4774b3fb-9cf8-4dda-a12c-89290fcf0ea4-kube-api-access-w6gmf\") pod \"maas-api-revoke-sub-5076e05a-jgdkf\" (UID: \"4774b3fb-9cf8-4dda-a12c-89290fcf0ea4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5076e05a-jgdkf" Nov 03 13:49:43.810610 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:49:43.810522 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5076e05a-jgdkf" Nov 03 13:49:44.090332 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:49:44.090295 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5076e05a-jgdkf"] Nov 03 13:49:44.091212 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:49:44.091179 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4774b3fb_9cf8_4dda_a12c_89290fcf0ea4.slice/crio-b999aacf39fc53bba94c38572b174536be33eb80ed53e3a62cb974a4149c350a WatchSource:0}: Error finding container b999aacf39fc53bba94c38572b174536be33eb80ed53e3a62cb974a4149c350a: Status 404 returned error can't find the container with id b999aacf39fc53bba94c38572b174536be33eb80ed53e3a62cb974a4149c350a Nov 03 13:49:44.417584 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:49:44.417544 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5076e05a-jgdkf" event={"ID":"4774b3fb-9cf8-4dda-a12c-89290fcf0ea4","Type":"ContainerStarted","Data":"6ddb347c0c969acd75b79cda529a539475c46cf5acbf7b48bc83e5fd4e3e2554"} Nov 03 13:49:44.417741 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:49:44.417608 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5076e05a-jgdkf" event={"ID":"4774b3fb-9cf8-4dda-a12c-89290fcf0ea4","Type":"ContainerStarted","Data":"b999aacf39fc53bba94c38572b174536be33eb80ed53e3a62cb974a4149c350a"} Nov 03 13:49:44.440984 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:49:44.440928 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5076e05a-jgdkf" podStartSLOduration=1.440914018 podStartE2EDuration="1.440914018s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:49:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:49:44.43947826 +0000 UTC m=+2464.447669540" watchObservedRunningTime="2026-11-03 13:49:44.440914018 +0000 UTC m=+2464.449105294" Nov 03 13:49:45.423771 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:49:45.423738 5024 generic.go:356] "Generic (PLEG): container finished" podID="4774b3fb-9cf8-4dda-a12c-89290fcf0ea4" containerID="6ddb347c0c969acd75b79cda529a539475c46cf5acbf7b48bc83e5fd4e3e2554" exitCode=0 Nov 03 13:49:45.424229 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:49:45.423807 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5076e05a-jgdkf" event={"ID":"4774b3fb-9cf8-4dda-a12c-89290fcf0ea4","Type":"ContainerDied","Data":"6ddb347c0c969acd75b79cda529a539475c46cf5acbf7b48bc83e5fd4e3e2554"} Nov 03 13:49:46.566923 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:49:46.566896 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5076e05a-jgdkf" Nov 03 13:49:46.608827 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:49:46.608798 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4774b3fb-9cf8-4dda-a12c-89290fcf0ea4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4774b3fb-9cf8-4dda-a12c-89290fcf0ea4-maas-api-openshift-ca\") pod \"4774b3fb-9cf8-4dda-a12c-89290fcf0ea4\" (UID: \"4774b3fb-9cf8-4dda-a12c-89290fcf0ea4\") " Nov 03 13:49:46.609000 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:49:46.608840 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4774b3fb-9cf8-4dda-a12c-89290fcf0ea4-kube-api-access-w6gmf\" (UniqueName: \"kubernetes.io/projected/4774b3fb-9cf8-4dda-a12c-89290fcf0ea4-kube-api-access-w6gmf\") pod \"4774b3fb-9cf8-4dda-a12c-89290fcf0ea4\" (UID: \"4774b3fb-9cf8-4dda-a12c-89290fcf0ea4\") " Nov 03 13:49:46.609000 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:49:46.608918 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4774b3fb-9cf8-4dda-a12c-89290fcf0ea4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4774b3fb-9cf8-4dda-a12c-89290fcf0ea4-maas-api-xks-ca\") pod \"4774b3fb-9cf8-4dda-a12c-89290fcf0ea4\" (UID: \"4774b3fb-9cf8-4dda-a12c-89290fcf0ea4\") " Nov 03 13:49:46.609160 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:49:46.609132 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4774b3fb-9cf8-4dda-a12c-89290fcf0ea4-maas-api-openshift-ca" pod "4774b3fb-9cf8-4dda-a12c-89290fcf0ea4" (UID: "4774b3fb-9cf8-4dda-a12c-89290fcf0ea4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:49:46.609253 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:49:46.609230 5024 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4774b3fb-9cf8-4dda-a12c-89290fcf0ea4-maas-api-openshift-ca\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:49:46.611051 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:49:46.611028 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4774b3fb-9cf8-4dda-a12c-89290fcf0ea4-maas-api-xks-ca" pod "4774b3fb-9cf8-4dda-a12c-89290fcf0ea4" (UID: "4774b3fb-9cf8-4dda-a12c-89290fcf0ea4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 13:49:46.611137 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:49:46.611059 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4774b3fb-9cf8-4dda-a12c-89290fcf0ea4-kube-api-access-w6gmf" pod "4774b3fb-9cf8-4dda-a12c-89290fcf0ea4" (UID: "4774b3fb-9cf8-4dda-a12c-89290fcf0ea4"). InnerVolumeSpecName "kube-api-access-w6gmf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:49:46.710541 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:49:46.710472 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w6gmf\" (UniqueName: \"kubernetes.io/projected/4774b3fb-9cf8-4dda-a12c-89290fcf0ea4-kube-api-access-w6gmf\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:49:46.710541 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:49:46.710508 5024 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4774b3fb-9cf8-4dda-a12c-89290fcf0ea4-maas-api-xks-ca\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:49:47.434821 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:49:47.434789 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5076e05a-jgdkf" Nov 03 13:49:47.434997 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:49:47.434823 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5076e05a-jgdkf" event={"ID":"4774b3fb-9cf8-4dda-a12c-89290fcf0ea4","Type":"ContainerDied","Data":"b999aacf39fc53bba94c38572b174536be33eb80ed53e3a62cb974a4149c350a"} Nov 03 13:49:47.434997 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:49:47.434894 5024 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b999aacf39fc53bba94c38572b174536be33eb80ed53e3a62cb974a4149c350a" Nov 03 13:49:53.728534 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:49:53.728500 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fbfb876e-gv9v5"] Nov 03 13:49:53.729099 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:49:53.729081 5024 cpu_manager.go:414] "containerMap: removing container" podUID="4774b3fb-9cf8-4dda-a12c-89290fcf0ea4" containerName="revoke-keys" Nov 03 13:49:53.729179 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:49:53.729099 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4774b3fb-9cf8-4dda-a12c-89290fcf0ea4" containerName="revoke-keys" Nov 03 13:49:53.729238 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:49:53.729219 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="4774b3fb-9cf8-4dda-a12c-89290fcf0ea4" containerName="revoke-keys" Nov 03 13:49:53.732739 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:49:53.732716 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbfb876e-gv9v5" Nov 03 13:49:53.739314 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:49:53.738717 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 13:49:53.739314 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:49:53.738797 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 13:49:53.739314 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:49:53.738884 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 13:49:53.740570 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:49:53.740536 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fbfb876e-gv9v5"] Nov 03 13:49:53.740704 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:49:53.740681 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-phqfq\"" Nov 03 13:49:53.777245 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:49:53.777208 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af1113d7-cd0a-439b-a323-7f9ca5eb6723-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fbfb876e-gv9v5\" (UID: \"af1113d7-cd0a-439b-a323-7f9ca5eb6723\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbfb876e-gv9v5" Nov 03 13:49:53.777245 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:49:53.777251 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rfzqk\" (UniqueName: \"kubernetes.io/projected/af1113d7-cd0a-439b-a323-7f9ca5eb6723-kube-api-access-rfzqk\") pod \"maas-api-revoke-sub-fbfb876e-gv9v5\" (UID: \"af1113d7-cd0a-439b-a323-7f9ca5eb6723\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbfb876e-gv9v5" Nov 03 13:49:53.777479 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:49:53.777274 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af1113d7-cd0a-439b-a323-7f9ca5eb6723-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fbfb876e-gv9v5\" (UID: \"af1113d7-cd0a-439b-a323-7f9ca5eb6723\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbfb876e-gv9v5" Nov 03 13:49:53.878394 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:49:53.878353 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af1113d7-cd0a-439b-a323-7f9ca5eb6723-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fbfb876e-gv9v5\" (UID: \"af1113d7-cd0a-439b-a323-7f9ca5eb6723\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbfb876e-gv9v5" Nov 03 13:49:53.878575 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:49:53.878418 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rfzqk\" (UniqueName: \"kubernetes.io/projected/af1113d7-cd0a-439b-a323-7f9ca5eb6723-kube-api-access-rfzqk\") pod \"maas-api-revoke-sub-fbfb876e-gv9v5\" (UID: \"af1113d7-cd0a-439b-a323-7f9ca5eb6723\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbfb876e-gv9v5" Nov 03 13:49:53.878575 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:49:53.878455 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af1113d7-cd0a-439b-a323-7f9ca5eb6723-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fbfb876e-gv9v5\" (UID: \"af1113d7-cd0a-439b-a323-7f9ca5eb6723\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbfb876e-gv9v5" Nov 03 13:49:53.879096 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:49:53.879062 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af1113d7-cd0a-439b-a323-7f9ca5eb6723-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fbfb876e-gv9v5\" (UID: \"af1113d7-cd0a-439b-a323-7f9ca5eb6723\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbfb876e-gv9v5" Nov 03 13:49:53.881074 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:49:53.881044 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af1113d7-cd0a-439b-a323-7f9ca5eb6723-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fbfb876e-gv9v5\" (UID: \"af1113d7-cd0a-439b-a323-7f9ca5eb6723\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbfb876e-gv9v5" Nov 03 13:49:53.887956 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:49:53.887924 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rfzqk\" (UniqueName: \"kubernetes.io/projected/af1113d7-cd0a-439b-a323-7f9ca5eb6723-kube-api-access-rfzqk\") pod \"maas-api-revoke-sub-fbfb876e-gv9v5\" (UID: \"af1113d7-cd0a-439b-a323-7f9ca5eb6723\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbfb876e-gv9v5" Nov 03 13:49:54.049904 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:49:54.049817 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbfb876e-gv9v5" Nov 03 13:49:54.329175 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:49:54.329144 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fbfb876e-gv9v5"] Nov 03 13:49:54.331225 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:49:54.331196 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf1113d7_cd0a_439b_a323_7f9ca5eb6723.slice/crio-b3e7259b1b905b92d64e17af774bf71164d1fd97a8a18a5468e24d958c762aad WatchSource:0}: Error finding container b3e7259b1b905b92d64e17af774bf71164d1fd97a8a18a5468e24d958c762aad: Status 404 returned error can't find the container with id b3e7259b1b905b92d64e17af774bf71164d1fd97a8a18a5468e24d958c762aad Nov 03 13:49:54.470372 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:49:54.470323 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbfb876e-gv9v5" event={"ID":"af1113d7-cd0a-439b-a323-7f9ca5eb6723","Type":"ContainerStarted","Data":"b3e7259b1b905b92d64e17af774bf71164d1fd97a8a18a5468e24d958c762aad"} Nov 03 13:49:55.479850 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:49:55.479799 5024 generic.go:356] "Generic (PLEG): container finished" podID="af1113d7-cd0a-439b-a323-7f9ca5eb6723" containerID="0e7c06a760570d9d832349a69a0ccdc8e0dbc070e230c562feabfbb42f390b84" exitCode=0 Nov 03 13:49:55.480344 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:49:55.479963 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbfb876e-gv9v5" event={"ID":"af1113d7-cd0a-439b-a323-7f9ca5eb6723","Type":"ContainerDied","Data":"0e7c06a760570d9d832349a69a0ccdc8e0dbc070e230c562feabfbb42f390b84"} Nov 03 13:49:57.226411 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:49:57.226372 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbfb876e-gv9v5" Nov 03 13:49:57.372309 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:49:57.371219 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af1113d7-cd0a-439b-a323-7f9ca5eb6723-kube-api-access-rfzqk\" (UniqueName: \"kubernetes.io/projected/af1113d7-cd0a-439b-a323-7f9ca5eb6723-kube-api-access-rfzqk\") pod \"af1113d7-cd0a-439b-a323-7f9ca5eb6723\" (UID: \"af1113d7-cd0a-439b-a323-7f9ca5eb6723\") " Nov 03 13:49:57.372309 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:49:57.371342 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/af1113d7-cd0a-439b-a323-7f9ca5eb6723-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af1113d7-cd0a-439b-a323-7f9ca5eb6723-maas-api-openshift-ca\") pod \"af1113d7-cd0a-439b-a323-7f9ca5eb6723\" (UID: \"af1113d7-cd0a-439b-a323-7f9ca5eb6723\") " Nov 03 13:49:57.372309 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:49:57.371416 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/af1113d7-cd0a-439b-a323-7f9ca5eb6723-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af1113d7-cd0a-439b-a323-7f9ca5eb6723-maas-api-xks-ca\") pod \"af1113d7-cd0a-439b-a323-7f9ca5eb6723\" (UID: \"af1113d7-cd0a-439b-a323-7f9ca5eb6723\") " Nov 03 13:49:57.383310 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:49:57.373531 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af1113d7-cd0a-439b-a323-7f9ca5eb6723-maas-api-openshift-ca" pod "af1113d7-cd0a-439b-a323-7f9ca5eb6723" (UID: "af1113d7-cd0a-439b-a323-7f9ca5eb6723"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:49:57.412461 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:49:57.412412 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af1113d7-cd0a-439b-a323-7f9ca5eb6723-maas-api-xks-ca" pod "af1113d7-cd0a-439b-a323-7f9ca5eb6723" (UID: "af1113d7-cd0a-439b-a323-7f9ca5eb6723"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 13:49:57.413106 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:49:57.413064 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af1113d7-cd0a-439b-a323-7f9ca5eb6723-kube-api-access-rfzqk" pod "af1113d7-cd0a-439b-a323-7f9ca5eb6723" (UID: "af1113d7-cd0a-439b-a323-7f9ca5eb6723"). InnerVolumeSpecName "kube-api-access-rfzqk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:49:57.485039 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:49:57.484937 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rfzqk\" (UniqueName: \"kubernetes.io/projected/af1113d7-cd0a-439b-a323-7f9ca5eb6723-kube-api-access-rfzqk\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:49:57.485039 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:49:57.484979 5024 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af1113d7-cd0a-439b-a323-7f9ca5eb6723-maas-api-openshift-ca\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:49:57.485039 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:49:57.484997 5024 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af1113d7-cd0a-439b-a323-7f9ca5eb6723-maas-api-xks-ca\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:49:57.554849 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:49:57.554729 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbfb876e-gv9v5" event={"ID":"af1113d7-cd0a-439b-a323-7f9ca5eb6723","Type":"ContainerDied","Data":"b3e7259b1b905b92d64e17af774bf71164d1fd97a8a18a5468e24d958c762aad"} Nov 03 13:49:57.555073 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:49:57.554989 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbfb876e-gv9v5" Nov 03 13:49:57.555217 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:49:57.555182 5024 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b3e7259b1b905b92d64e17af774bf71164d1fd97a8a18a5468e24d958c762aad" Nov 03 13:50:04.078391 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:50:04.078352 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e0f27240-9pkw2"] Nov 03 13:50:04.078882 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:50:04.078768 5024 cpu_manager.go:414] "containerMap: removing container" podUID="af1113d7-cd0a-439b-a323-7f9ca5eb6723" containerName="revoke-keys" Nov 03 13:50:04.078882 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:50:04.078781 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af1113d7-cd0a-439b-a323-7f9ca5eb6723" containerName="revoke-keys" Nov 03 13:50:04.078882 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:50:04.078852 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="af1113d7-cd0a-439b-a323-7f9ca5eb6723" containerName="revoke-keys" Nov 03 13:50:04.082052 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:50:04.082027 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0f27240-9pkw2" Nov 03 13:50:04.087141 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:50:04.087118 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 13:50:04.090976 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:50:04.089064 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 13:50:04.090976 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:50:04.089418 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-phqfq\"" Nov 03 13:50:04.090976 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:50:04.089637 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 13:50:04.092543 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:50:04.092522 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e0f27240-9pkw2"] Nov 03 13:50:04.145520 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:50:04.145484 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3ae5602-f8f2-42a7-bbb9-a878e9a984ab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e0f27240-9pkw2\" (UID: \"c3ae5602-f8f2-42a7-bbb9-a878e9a984ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0f27240-9pkw2" Nov 03 13:50:04.145520 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:50:04.145525 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j5htx\" (UniqueName: \"kubernetes.io/projected/c3ae5602-f8f2-42a7-bbb9-a878e9a984ab-kube-api-access-j5htx\") pod \"maas-api-revoke-sub-e0f27240-9pkw2\" (UID: \"c3ae5602-f8f2-42a7-bbb9-a878e9a984ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0f27240-9pkw2" Nov 03 13:50:04.145784 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:50:04.145650 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3ae5602-f8f2-42a7-bbb9-a878e9a984ab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e0f27240-9pkw2\" (UID: \"c3ae5602-f8f2-42a7-bbb9-a878e9a984ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0f27240-9pkw2" Nov 03 13:50:04.246461 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:50:04.246421 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3ae5602-f8f2-42a7-bbb9-a878e9a984ab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e0f27240-9pkw2\" (UID: \"c3ae5602-f8f2-42a7-bbb9-a878e9a984ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0f27240-9pkw2" Nov 03 13:50:04.246652 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:50:04.246523 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3ae5602-f8f2-42a7-bbb9-a878e9a984ab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e0f27240-9pkw2\" (UID: \"c3ae5602-f8f2-42a7-bbb9-a878e9a984ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0f27240-9pkw2" Nov 03 13:50:04.246652 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:50:04.246554 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j5htx\" (UniqueName: \"kubernetes.io/projected/c3ae5602-f8f2-42a7-bbb9-a878e9a984ab-kube-api-access-j5htx\") pod \"maas-api-revoke-sub-e0f27240-9pkw2\" (UID: \"c3ae5602-f8f2-42a7-bbb9-a878e9a984ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0f27240-9pkw2" Nov 03 13:50:04.247114 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:50:04.247091 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3ae5602-f8f2-42a7-bbb9-a878e9a984ab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e0f27240-9pkw2\" (UID: \"c3ae5602-f8f2-42a7-bbb9-a878e9a984ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0f27240-9pkw2" Nov 03 13:50:04.248788 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:50:04.248767 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3ae5602-f8f2-42a7-bbb9-a878e9a984ab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e0f27240-9pkw2\" (UID: \"c3ae5602-f8f2-42a7-bbb9-a878e9a984ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0f27240-9pkw2" Nov 03 13:50:04.256824 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:50:04.256796 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j5htx\" (UniqueName: \"kubernetes.io/projected/c3ae5602-f8f2-42a7-bbb9-a878e9a984ab-kube-api-access-j5htx\") pod \"maas-api-revoke-sub-e0f27240-9pkw2\" (UID: \"c3ae5602-f8f2-42a7-bbb9-a878e9a984ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0f27240-9pkw2" Nov 03 13:50:04.397352 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:50:04.397316 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0f27240-9pkw2" Nov 03 13:50:04.676892 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:50:04.676866 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e0f27240-9pkw2"] Nov 03 13:50:04.679433 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:50:04.679399 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3ae5602_f8f2_42a7_bbb9_a878e9a984ab.slice/crio-aa5e767c71c56322d4baa5857401dd8d5752b48ea52a4b6877c78774cf01ce92 WatchSource:0}: Error finding container aa5e767c71c56322d4baa5857401dd8d5752b48ea52a4b6877c78774cf01ce92: Status 404 returned error can't find the container with id aa5e767c71c56322d4baa5857401dd8d5752b48ea52a4b6877c78774cf01ce92 Nov 03 13:50:05.597103 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:50:05.597067 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0f27240-9pkw2" event={"ID":"c3ae5602-f8f2-42a7-bbb9-a878e9a984ab","Type":"ContainerStarted","Data":"32440bdebfea8f756fbcfb365e13d7dd69e77d99dd6719d8bf3cc16929229446"} Nov 03 13:50:05.597533 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:50:05.597139 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0f27240-9pkw2" event={"ID":"c3ae5602-f8f2-42a7-bbb9-a878e9a984ab","Type":"ContainerStarted","Data":"aa5e767c71c56322d4baa5857401dd8d5752b48ea52a4b6877c78774cf01ce92"} Nov 03 13:50:06.618107 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:50:06.618070 5024 generic.go:356] "Generic (PLEG): container finished" podID="c3ae5602-f8f2-42a7-bbb9-a878e9a984ab" containerID="32440bdebfea8f756fbcfb365e13d7dd69e77d99dd6719d8bf3cc16929229446" exitCode=0 Nov 03 13:50:06.619026 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:50:06.618922 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0f27240-9pkw2" event={"ID":"c3ae5602-f8f2-42a7-bbb9-a878e9a984ab","Type":"ContainerDied","Data":"32440bdebfea8f756fbcfb365e13d7dd69e77d99dd6719d8bf3cc16929229446"} Nov 03 13:50:07.341488 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:50:07.341454 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0f27240-9pkw2" Nov 03 13:50:07.539383 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:50:07.538478 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3ae5602-f8f2-42a7-bbb9-a878e9a984ab-kube-api-access-j5htx\" (UniqueName: \"kubernetes.io/projected/c3ae5602-f8f2-42a7-bbb9-a878e9a984ab-kube-api-access-j5htx\") pod \"c3ae5602-f8f2-42a7-bbb9-a878e9a984ab\" (UID: \"c3ae5602-f8f2-42a7-bbb9-a878e9a984ab\") " Nov 03 13:50:07.539383 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:50:07.538553 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c3ae5602-f8f2-42a7-bbb9-a878e9a984ab-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3ae5602-f8f2-42a7-bbb9-a878e9a984ab-maas-api-openshift-ca\") pod \"c3ae5602-f8f2-42a7-bbb9-a878e9a984ab\" (UID: \"c3ae5602-f8f2-42a7-bbb9-a878e9a984ab\") " Nov 03 13:50:07.539383 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:50:07.538670 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c3ae5602-f8f2-42a7-bbb9-a878e9a984ab-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3ae5602-f8f2-42a7-bbb9-a878e9a984ab-maas-api-xks-ca\") pod \"c3ae5602-f8f2-42a7-bbb9-a878e9a984ab\" (UID: \"c3ae5602-f8f2-42a7-bbb9-a878e9a984ab\") " Nov 03 13:50:07.541023 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:50:07.540990 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3ae5602-f8f2-42a7-bbb9-a878e9a984ab-maas-api-openshift-ca" pod "c3ae5602-f8f2-42a7-bbb9-a878e9a984ab" (UID: "c3ae5602-f8f2-42a7-bbb9-a878e9a984ab"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:50:07.647050 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:50:07.646993 5024 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3ae5602-f8f2-42a7-bbb9-a878e9a984ab-maas-api-openshift-ca\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:50:07.665803 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:50:07.665075 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c3ae5602-f8f2-42a7-bbb9-a878e9a984ab-maas-api-xks-ca" pod "c3ae5602-f8f2-42a7-bbb9-a878e9a984ab" (UID: "c3ae5602-f8f2-42a7-bbb9-a878e9a984ab"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 13:50:07.665803 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:50:07.665392 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0f27240-9pkw2" event={"ID":"c3ae5602-f8f2-42a7-bbb9-a878e9a984ab","Type":"ContainerDied","Data":"aa5e767c71c56322d4baa5857401dd8d5752b48ea52a4b6877c78774cf01ce92"} Nov 03 13:50:07.665803 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:50:07.665451 5024 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aa5e767c71c56322d4baa5857401dd8d5752b48ea52a4b6877c78774cf01ce92" Nov 03 13:50:07.665803 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:50:07.665525 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0f27240-9pkw2" Nov 03 13:50:07.666467 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:50:07.666418 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3ae5602-f8f2-42a7-bbb9-a878e9a984ab-kube-api-access-j5htx" pod "c3ae5602-f8f2-42a7-bbb9-a878e9a984ab" (UID: "c3ae5602-f8f2-42a7-bbb9-a878e9a984ab"). InnerVolumeSpecName "kube-api-access-j5htx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:50:07.753015 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:50:07.752935 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j5htx\" (UniqueName: \"kubernetes.io/projected/c3ae5602-f8f2-42a7-bbb9-a878e9a984ab-kube-api-access-j5htx\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:50:07.753015 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:50:07.752975 5024 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3ae5602-f8f2-42a7-bbb9-a878e9a984ab-maas-api-xks-ca\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:50:09.508133 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:50:09.508093 5024 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d7mwz" containerName="extractor" Nov 03 13:50:14.323739 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:50:14.323701 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c4a5d20d-vtczd"] Nov 03 13:50:14.324266 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:50:14.324099 5024 cpu_manager.go:414] "containerMap: removing container" podUID="c3ae5602-f8f2-42a7-bbb9-a878e9a984ab" containerName="revoke-keys" Nov 03 13:50:14.324266 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:50:14.324110 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3ae5602-f8f2-42a7-bbb9-a878e9a984ab" containerName="revoke-keys" Nov 03 13:50:14.324266 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:50:14.324200 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3ae5602-f8f2-42a7-bbb9-a878e9a984ab" containerName="revoke-keys" Nov 03 13:50:14.327398 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:50:14.327378 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4a5d20d-vtczd" Nov 03 13:50:14.333547 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:50:14.333523 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 13:50:14.333742 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:50:14.333610 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 13:50:14.341495 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:50:14.341468 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c4a5d20d-vtczd"] Nov 03 13:50:14.390324 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:50:14.390271 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 13:50:14.390476 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:50:14.390349 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-phqfq\"" Nov 03 13:50:14.422466 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:50:14.422437 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0412dc2-841f-4005-a841-128786fce7be-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c4a5d20d-vtczd\" (UID: \"e0412dc2-841f-4005-a841-128786fce7be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4a5d20d-vtczd" Nov 03 13:50:14.422633 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:50:14.422472 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bgjjm\" (UniqueName: \"kubernetes.io/projected/e0412dc2-841f-4005-a841-128786fce7be-kube-api-access-bgjjm\") pod \"maas-api-revoke-sub-c4a5d20d-vtczd\" (UID: \"e0412dc2-841f-4005-a841-128786fce7be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4a5d20d-vtczd" Nov 03 13:50:14.422633 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:50:14.422495 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0412dc2-841f-4005-a841-128786fce7be-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c4a5d20d-vtczd\" (UID: \"e0412dc2-841f-4005-a841-128786fce7be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4a5d20d-vtczd" Nov 03 13:50:14.523656 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:50:14.523608 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0412dc2-841f-4005-a841-128786fce7be-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c4a5d20d-vtczd\" (UID: \"e0412dc2-841f-4005-a841-128786fce7be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4a5d20d-vtczd" Nov 03 13:50:14.523873 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:50:14.523668 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgjjm\" (UniqueName: \"kubernetes.io/projected/e0412dc2-841f-4005-a841-128786fce7be-kube-api-access-bgjjm\") pod \"maas-api-revoke-sub-c4a5d20d-vtczd\" (UID: \"e0412dc2-841f-4005-a841-128786fce7be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4a5d20d-vtczd" Nov 03 13:50:14.523873 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:50:14.523707 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0412dc2-841f-4005-a841-128786fce7be-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c4a5d20d-vtczd\" (UID: \"e0412dc2-841f-4005-a841-128786fce7be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4a5d20d-vtczd" Nov 03 13:50:14.524394 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:50:14.524359 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0412dc2-841f-4005-a841-128786fce7be-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c4a5d20d-vtczd\" (UID: \"e0412dc2-841f-4005-a841-128786fce7be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4a5d20d-vtczd" Nov 03 13:50:14.526352 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:50:14.526262 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0412dc2-841f-4005-a841-128786fce7be-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c4a5d20d-vtczd\" (UID: \"e0412dc2-841f-4005-a841-128786fce7be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4a5d20d-vtczd" Nov 03 13:50:14.533224 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:50:14.533192 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bgjjm\" (UniqueName: \"kubernetes.io/projected/e0412dc2-841f-4005-a841-128786fce7be-kube-api-access-bgjjm\") pod \"maas-api-revoke-sub-c4a5d20d-vtczd\" (UID: \"e0412dc2-841f-4005-a841-128786fce7be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4a5d20d-vtczd" Nov 03 13:50:14.641297 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:50:14.641262 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4a5d20d-vtczd" Nov 03 13:50:14.922194 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:50:14.922166 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c4a5d20d-vtczd"] Nov 03 13:50:14.924253 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:50:14.924225 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0412dc2_841f_4005_a841_128786fce7be.slice/crio-efee002524b20fedf8f6f73bf3879dc6fb79ef2430c28455aeacba6402120a6a WatchSource:0}: Error finding container efee002524b20fedf8f6f73bf3879dc6fb79ef2430c28455aeacba6402120a6a: Status 404 returned error can't find the container with id efee002524b20fedf8f6f73bf3879dc6fb79ef2430c28455aeacba6402120a6a Nov 03 13:50:15.721894 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:50:15.721837 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4a5d20d-vtczd" event={"ID":"e0412dc2-841f-4005-a841-128786fce7be","Type":"ContainerStarted","Data":"5733abf9a4f0cb407abdac0bc99053883f842a2ae96889fe77e4638d0f4dd1a1"} Nov 03 13:50:15.722515 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:50:15.721913 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4a5d20d-vtczd" event={"ID":"e0412dc2-841f-4005-a841-128786fce7be","Type":"ContainerStarted","Data":"efee002524b20fedf8f6f73bf3879dc6fb79ef2430c28455aeacba6402120a6a"} Nov 03 13:50:15.742972 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:50:15.742919 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4a5d20d-vtczd" podStartSLOduration=1.742904766 podStartE2EDuration="1.742904766s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:50:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:50:15.742258837 +0000 UTC m=+2495.750450114" watchObservedRunningTime="2026-11-03 13:50:15.742904766 +0000 UTC m=+2495.751096043" Nov 03 13:50:16.764509 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:50:16.764468 5024 generic.go:356] "Generic (PLEG): container finished" podID="e0412dc2-841f-4005-a841-128786fce7be" containerID="5733abf9a4f0cb407abdac0bc99053883f842a2ae96889fe77e4638d0f4dd1a1" exitCode=0 Nov 03 13:50:16.765237 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:50:16.765207 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4a5d20d-vtczd" event={"ID":"e0412dc2-841f-4005-a841-128786fce7be","Type":"ContainerDied","Data":"5733abf9a4f0cb407abdac0bc99053883f842a2ae96889fe77e4638d0f4dd1a1"} Nov 03 13:50:18.400904 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:50:18.400866 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4a5d20d-vtczd" Nov 03 13:50:18.429489 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:50:18.429452 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e0412dc2-841f-4005-a841-128786fce7be-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0412dc2-841f-4005-a841-128786fce7be-maas-api-openshift-ca\") pod \"e0412dc2-841f-4005-a841-128786fce7be\" (UID: \"e0412dc2-841f-4005-a841-128786fce7be\") " Nov 03 13:50:18.429912 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:50:18.429885 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e0412dc2-841f-4005-a841-128786fce7be-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0412dc2-841f-4005-a841-128786fce7be-maas-api-xks-ca\") pod \"e0412dc2-841f-4005-a841-128786fce7be\" (UID: \"e0412dc2-841f-4005-a841-128786fce7be\") " Nov 03 13:50:18.430458 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:50:18.430413 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0412dc2-841f-4005-a841-128786fce7be-maas-api-openshift-ca" pod "e0412dc2-841f-4005-a841-128786fce7be" (UID: "e0412dc2-841f-4005-a841-128786fce7be"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:50:18.430458 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:50:18.430415 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0412dc2-841f-4005-a841-128786fce7be-kube-api-access-bgjjm\" (UniqueName: \"kubernetes.io/projected/e0412dc2-841f-4005-a841-128786fce7be-kube-api-access-bgjjm\") pod \"e0412dc2-841f-4005-a841-128786fce7be\" (UID: \"e0412dc2-841f-4005-a841-128786fce7be\") " Nov 03 13:50:18.433034 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:50:18.430771 5024 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0412dc2-841f-4005-a841-128786fce7be-maas-api-openshift-ca\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:50:18.434421 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:50:18.434393 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0412dc2-841f-4005-a841-128786fce7be-maas-api-xks-ca" pod "e0412dc2-841f-4005-a841-128786fce7be" (UID: "e0412dc2-841f-4005-a841-128786fce7be"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 13:50:18.436441 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:50:18.436403 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0412dc2-841f-4005-a841-128786fce7be-kube-api-access-bgjjm" pod "e0412dc2-841f-4005-a841-128786fce7be" (UID: "e0412dc2-841f-4005-a841-128786fce7be"). InnerVolumeSpecName "kube-api-access-bgjjm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:50:18.532173 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:50:18.532137 5024 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0412dc2-841f-4005-a841-128786fce7be-maas-api-xks-ca\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:50:18.532173 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:50:18.532170 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bgjjm\" (UniqueName: \"kubernetes.io/projected/e0412dc2-841f-4005-a841-128786fce7be-kube-api-access-bgjjm\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:50:18.788515 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:50:18.788438 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4a5d20d-vtczd" Nov 03 13:50:18.788662 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:50:18.788434 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4a5d20d-vtczd" event={"ID":"e0412dc2-841f-4005-a841-128786fce7be","Type":"ContainerDied","Data":"efee002524b20fedf8f6f73bf3879dc6fb79ef2430c28455aeacba6402120a6a"} Nov 03 13:50:18.788662 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:50:18.788572 5024 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="efee002524b20fedf8f6f73bf3879dc6fb79ef2430c28455aeacba6402120a6a" Nov 03 13:50:24.515859 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:50:24.515798 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9bf1edb6-pdnkf"] Nov 03 13:50:24.516366 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:50:24.516244 5024 cpu_manager.go:414] "containerMap: removing container" podUID="e0412dc2-841f-4005-a841-128786fce7be" containerName="revoke-keys" Nov 03 13:50:24.516366 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:50:24.516260 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0412dc2-841f-4005-a841-128786fce7be" containerName="revoke-keys" Nov 03 13:50:24.516552 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:50:24.516379 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0412dc2-841f-4005-a841-128786fce7be" containerName="revoke-keys" Nov 03 13:50:24.519800 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:50:24.519771 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bf1edb6-pdnkf" Nov 03 13:50:24.526089 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:50:24.526038 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 13:50:24.526089 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:50:24.526038 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 13:50:24.526324 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:50:24.526144 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 13:50:24.526324 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:50:24.526253 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-phqfq\"" Nov 03 13:50:24.526561 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:50:24.526537 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9bf1edb6-pdnkf"] Nov 03 13:50:24.585421 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:50:24.585386 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c179e407-4918-4bdb-9f6b-3ccdd219f40c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9bf1edb6-pdnkf\" (UID: \"c179e407-4918-4bdb-9f6b-3ccdd219f40c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bf1edb6-pdnkf" Nov 03 13:50:24.585591 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:50:24.585506 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c179e407-4918-4bdb-9f6b-3ccdd219f40c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9bf1edb6-pdnkf\" (UID: \"c179e407-4918-4bdb-9f6b-3ccdd219f40c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bf1edb6-pdnkf" Nov 03 13:50:24.585591 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:50:24.585542 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2wzvh\" (UniqueName: \"kubernetes.io/projected/c179e407-4918-4bdb-9f6b-3ccdd219f40c-kube-api-access-2wzvh\") pod \"maas-api-revoke-sub-9bf1edb6-pdnkf\" (UID: \"c179e407-4918-4bdb-9f6b-3ccdd219f40c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bf1edb6-pdnkf" Nov 03 13:50:24.685995 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:50:24.685959 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2wzvh\" (UniqueName: \"kubernetes.io/projected/c179e407-4918-4bdb-9f6b-3ccdd219f40c-kube-api-access-2wzvh\") pod \"maas-api-revoke-sub-9bf1edb6-pdnkf\" (UID: \"c179e407-4918-4bdb-9f6b-3ccdd219f40c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bf1edb6-pdnkf" Nov 03 13:50:24.686191 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:50:24.686054 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c179e407-4918-4bdb-9f6b-3ccdd219f40c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9bf1edb6-pdnkf\" (UID: \"c179e407-4918-4bdb-9f6b-3ccdd219f40c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bf1edb6-pdnkf" Nov 03 13:50:24.686191 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:50:24.686154 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c179e407-4918-4bdb-9f6b-3ccdd219f40c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9bf1edb6-pdnkf\" (UID: \"c179e407-4918-4bdb-9f6b-3ccdd219f40c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bf1edb6-pdnkf" Nov 03 13:50:24.686743 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:50:24.686717 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c179e407-4918-4bdb-9f6b-3ccdd219f40c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9bf1edb6-pdnkf\" (UID: \"c179e407-4918-4bdb-9f6b-3ccdd219f40c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bf1edb6-pdnkf" Nov 03 13:50:24.688510 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:50:24.688488 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c179e407-4918-4bdb-9f6b-3ccdd219f40c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9bf1edb6-pdnkf\" (UID: \"c179e407-4918-4bdb-9f6b-3ccdd219f40c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bf1edb6-pdnkf" Nov 03 13:50:24.696524 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:50:24.696491 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2wzvh\" (UniqueName: \"kubernetes.io/projected/c179e407-4918-4bdb-9f6b-3ccdd219f40c-kube-api-access-2wzvh\") pod \"maas-api-revoke-sub-9bf1edb6-pdnkf\" (UID: \"c179e407-4918-4bdb-9f6b-3ccdd219f40c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bf1edb6-pdnkf" Nov 03 13:50:24.836770 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:50:24.836689 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bf1edb6-pdnkf" Nov 03 13:50:25.118109 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:50:25.117739 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9bf1edb6-pdnkf"] Nov 03 13:50:25.123824 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:50:25.123792 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc179e407_4918_4bdb_9f6b_3ccdd219f40c.slice/crio-ae4799098490f71fb57e280b8c31c6da58b9219dac067a12d0b4df836996e5d3 WatchSource:0}: Error finding container ae4799098490f71fb57e280b8c31c6da58b9219dac067a12d0b4df836996e5d3: Status 404 returned error can't find the container with id ae4799098490f71fb57e280b8c31c6da58b9219dac067a12d0b4df836996e5d3 Nov 03 13:50:25.828360 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:50:25.827975 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bf1edb6-pdnkf" event={"ID":"c179e407-4918-4bdb-9f6b-3ccdd219f40c","Type":"ContainerStarted","Data":"4480a3386279d7f4ec5991a10d26cca8b2a510e758f3ab3aa8ac7f8dfb36c18a"} Nov 03 13:50:25.828360 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:50:25.828097 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bf1edb6-pdnkf" event={"ID":"c179e407-4918-4bdb-9f6b-3ccdd219f40c","Type":"ContainerStarted","Data":"ae4799098490f71fb57e280b8c31c6da58b9219dac067a12d0b4df836996e5d3"} Nov 03 13:50:25.849441 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:50:25.849374 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bf1edb6-pdnkf" podStartSLOduration=1.84935597 podStartE2EDuration="1.84935597s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:50:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:50:25.848025649 +0000 UTC m=+2505.856216927" watchObservedRunningTime="2026-11-03 13:50:25.84935597 +0000 UTC m=+2505.857547250" Nov 03 13:50:26.845880 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:50:26.845415 5024 generic.go:356] "Generic (PLEG): container finished" podID="c179e407-4918-4bdb-9f6b-3ccdd219f40c" containerID="4480a3386279d7f4ec5991a10d26cca8b2a510e758f3ab3aa8ac7f8dfb36c18a" exitCode=0 Nov 03 13:50:26.845880 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:50:26.845499 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bf1edb6-pdnkf" event={"ID":"c179e407-4918-4bdb-9f6b-3ccdd219f40c","Type":"ContainerDied","Data":"4480a3386279d7f4ec5991a10d26cca8b2a510e758f3ab3aa8ac7f8dfb36c18a"} Nov 03 13:50:28.487448 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:50:28.487418 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bf1edb6-pdnkf" Nov 03 13:50:28.514887 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:50:28.514846 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c179e407-4918-4bdb-9f6b-3ccdd219f40c-kube-api-access-2wzvh\" (UniqueName: \"kubernetes.io/projected/c179e407-4918-4bdb-9f6b-3ccdd219f40c-kube-api-access-2wzvh\") pod \"c179e407-4918-4bdb-9f6b-3ccdd219f40c\" (UID: \"c179e407-4918-4bdb-9f6b-3ccdd219f40c\") " Nov 03 13:50:28.515064 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:50:28.514906 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c179e407-4918-4bdb-9f6b-3ccdd219f40c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c179e407-4918-4bdb-9f6b-3ccdd219f40c-maas-api-xks-ca\") pod \"c179e407-4918-4bdb-9f6b-3ccdd219f40c\" (UID: \"c179e407-4918-4bdb-9f6b-3ccdd219f40c\") " Nov 03 13:50:28.515064 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:50:28.515013 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c179e407-4918-4bdb-9f6b-3ccdd219f40c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c179e407-4918-4bdb-9f6b-3ccdd219f40c-maas-api-openshift-ca\") pod \"c179e407-4918-4bdb-9f6b-3ccdd219f40c\" (UID: \"c179e407-4918-4bdb-9f6b-3ccdd219f40c\") " Nov 03 13:50:28.515708 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:50:28.515677 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c179e407-4918-4bdb-9f6b-3ccdd219f40c-maas-api-openshift-ca" pod "c179e407-4918-4bdb-9f6b-3ccdd219f40c" (UID: "c179e407-4918-4bdb-9f6b-3ccdd219f40c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:50:28.518506 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:50:28.518472 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c179e407-4918-4bdb-9f6b-3ccdd219f40c-kube-api-access-2wzvh" pod "c179e407-4918-4bdb-9f6b-3ccdd219f40c" (UID: "c179e407-4918-4bdb-9f6b-3ccdd219f40c"). InnerVolumeSpecName "kube-api-access-2wzvh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:50:28.519316 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:50:28.519269 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c179e407-4918-4bdb-9f6b-3ccdd219f40c-maas-api-xks-ca" pod "c179e407-4918-4bdb-9f6b-3ccdd219f40c" (UID: "c179e407-4918-4bdb-9f6b-3ccdd219f40c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 13:50:28.615808 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:50:28.615772 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2wzvh\" (UniqueName: \"kubernetes.io/projected/c179e407-4918-4bdb-9f6b-3ccdd219f40c-kube-api-access-2wzvh\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:50:28.615808 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:50:28.615810 5024 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c179e407-4918-4bdb-9f6b-3ccdd219f40c-maas-api-xks-ca\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:50:28.616017 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:50:28.615825 5024 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c179e407-4918-4bdb-9f6b-3ccdd219f40c-maas-api-openshift-ca\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:50:28.878265 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:50:28.878234 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bf1edb6-pdnkf" Nov 03 13:50:28.878265 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:50:28.878249 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bf1edb6-pdnkf" event={"ID":"c179e407-4918-4bdb-9f6b-3ccdd219f40c","Type":"ContainerDied","Data":"ae4799098490f71fb57e280b8c31c6da58b9219dac067a12d0b4df836996e5d3"} Nov 03 13:50:28.878551 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:50:28.878318 5024 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ae4799098490f71fb57e280b8c31c6da58b9219dac067a12d0b4df836996e5d3" Nov 03 13:50:34.736644 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:50:34.736610 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d31f7f2a-nv2ng"] Nov 03 13:50:34.737050 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:50:34.737038 5024 cpu_manager.go:414] "containerMap: removing container" podUID="c179e407-4918-4bdb-9f6b-3ccdd219f40c" containerName="revoke-keys" Nov 03 13:50:34.737112 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:50:34.737051 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c179e407-4918-4bdb-9f6b-3ccdd219f40c" containerName="revoke-keys" Nov 03 13:50:34.737178 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:50:34.737166 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="c179e407-4918-4bdb-9f6b-3ccdd219f40c" containerName="revoke-keys" Nov 03 13:50:34.740673 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:50:34.740644 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d31f7f2a-nv2ng" Nov 03 13:50:34.748270 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:50:34.748242 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 13:50:34.748444 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:50:34.748337 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 13:50:34.749446 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:50:34.749422 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-phqfq\"" Nov 03 13:50:34.749593 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:50:34.749521 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 13:50:34.753096 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:50:34.753069 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d31f7f2a-nv2ng"] Nov 03 13:50:34.764127 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:50:34.764095 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4983c732-1ead-4068-9bae-511cfd3ae7c3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d31f7f2a-nv2ng\" (UID: \"4983c732-1ead-4068-9bae-511cfd3ae7c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d31f7f2a-nv2ng" Nov 03 13:50:34.764240 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:50:34.764180 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4983c732-1ead-4068-9bae-511cfd3ae7c3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d31f7f2a-nv2ng\" (UID: \"4983c732-1ead-4068-9bae-511cfd3ae7c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d31f7f2a-nv2ng" Nov 03 13:50:34.764326 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:50:34.764251 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hg7p6\" (UniqueName: \"kubernetes.io/projected/4983c732-1ead-4068-9bae-511cfd3ae7c3-kube-api-access-hg7p6\") pod \"maas-api-revoke-sub-d31f7f2a-nv2ng\" (UID: \"4983c732-1ead-4068-9bae-511cfd3ae7c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d31f7f2a-nv2ng" Nov 03 13:50:34.865340 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:50:34.865265 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4983c732-1ead-4068-9bae-511cfd3ae7c3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d31f7f2a-nv2ng\" (UID: \"4983c732-1ead-4068-9bae-511cfd3ae7c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d31f7f2a-nv2ng" Nov 03 13:50:34.865514 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:50:34.865386 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4983c732-1ead-4068-9bae-511cfd3ae7c3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d31f7f2a-nv2ng\" (UID: \"4983c732-1ead-4068-9bae-511cfd3ae7c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d31f7f2a-nv2ng" Nov 03 13:50:34.865514 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:50:34.865458 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hg7p6\" (UniqueName: \"kubernetes.io/projected/4983c732-1ead-4068-9bae-511cfd3ae7c3-kube-api-access-hg7p6\") pod \"maas-api-revoke-sub-d31f7f2a-nv2ng\" (UID: \"4983c732-1ead-4068-9bae-511cfd3ae7c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d31f7f2a-nv2ng" Nov 03 13:50:34.865990 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:50:34.865959 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4983c732-1ead-4068-9bae-511cfd3ae7c3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d31f7f2a-nv2ng\" (UID: \"4983c732-1ead-4068-9bae-511cfd3ae7c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d31f7f2a-nv2ng" Nov 03 13:50:34.867727 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:50:34.867703 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4983c732-1ead-4068-9bae-511cfd3ae7c3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d31f7f2a-nv2ng\" (UID: \"4983c732-1ead-4068-9bae-511cfd3ae7c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d31f7f2a-nv2ng" Nov 03 13:50:34.875345 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:50:34.875321 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hg7p6\" (UniqueName: \"kubernetes.io/projected/4983c732-1ead-4068-9bae-511cfd3ae7c3-kube-api-access-hg7p6\") pod \"maas-api-revoke-sub-d31f7f2a-nv2ng\" (UID: \"4983c732-1ead-4068-9bae-511cfd3ae7c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d31f7f2a-nv2ng" Nov 03 13:50:35.057241 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:50:35.057153 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d31f7f2a-nv2ng" Nov 03 13:50:35.332601 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:50:35.332573 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d31f7f2a-nv2ng"] Nov 03 13:50:35.334548 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:50:35.334508 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4983c732_1ead_4068_9bae_511cfd3ae7c3.slice/crio-ca57c8d46fc47bc6ee11514a8543d1d860b578942551752ff9fd7fafce412eda WatchSource:0}: Error finding container ca57c8d46fc47bc6ee11514a8543d1d860b578942551752ff9fd7fafce412eda: Status 404 returned error can't find the container with id ca57c8d46fc47bc6ee11514a8543d1d860b578942551752ff9fd7fafce412eda Nov 03 13:50:35.917580 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:50:35.917531 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d31f7f2a-nv2ng" event={"ID":"4983c732-1ead-4068-9bae-511cfd3ae7c3","Type":"ContainerStarted","Data":"2cc39713c2ecf611cb970f890e4b1cfff92a73d04feef518d0d4f48f07e52726"} Nov 03 13:50:35.918167 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:50:35.917612 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d31f7f2a-nv2ng" event={"ID":"4983c732-1ead-4068-9bae-511cfd3ae7c3","Type":"ContainerStarted","Data":"ca57c8d46fc47bc6ee11514a8543d1d860b578942551752ff9fd7fafce412eda"} Nov 03 13:50:35.942992 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:50:35.942922 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d31f7f2a-nv2ng" podStartSLOduration=1.9429004220000001 podStartE2EDuration="1.942900422s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:50:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:50:35.938455226 +0000 UTC m=+2515.946646504" watchObservedRunningTime="2026-11-03 13:50:35.942900422 +0000 UTC m=+2515.951091700" Nov 03 13:50:36.939063 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:50:36.939021 5024 generic.go:356] "Generic (PLEG): container finished" podID="4983c732-1ead-4068-9bae-511cfd3ae7c3" containerID="2cc39713c2ecf611cb970f890e4b1cfff92a73d04feef518d0d4f48f07e52726" exitCode=0 Nov 03 13:50:36.939726 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:50:36.939698 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d31f7f2a-nv2ng" event={"ID":"4983c732-1ead-4068-9bae-511cfd3ae7c3","Type":"ContainerDied","Data":"2cc39713c2ecf611cb970f890e4b1cfff92a73d04feef518d0d4f48f07e52726"} Nov 03 13:50:38.679213 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:50:38.679182 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d31f7f2a-nv2ng" Nov 03 13:50:38.823296 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:50:38.823249 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4983c732-1ead-4068-9bae-511cfd3ae7c3-kube-api-access-hg7p6\" (UniqueName: \"kubernetes.io/projected/4983c732-1ead-4068-9bae-511cfd3ae7c3-kube-api-access-hg7p6\") pod \"4983c732-1ead-4068-9bae-511cfd3ae7c3\" (UID: \"4983c732-1ead-4068-9bae-511cfd3ae7c3\") " Nov 03 13:50:38.823487 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:50:38.823427 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4983c732-1ead-4068-9bae-511cfd3ae7c3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4983c732-1ead-4068-9bae-511cfd3ae7c3-maas-api-openshift-ca\") pod \"4983c732-1ead-4068-9bae-511cfd3ae7c3\" (UID: \"4983c732-1ead-4068-9bae-511cfd3ae7c3\") " Nov 03 13:50:38.823567 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:50:38.823533 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4983c732-1ead-4068-9bae-511cfd3ae7c3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4983c732-1ead-4068-9bae-511cfd3ae7c3-maas-api-xks-ca\") pod \"4983c732-1ead-4068-9bae-511cfd3ae7c3\" (UID: \"4983c732-1ead-4068-9bae-511cfd3ae7c3\") " Nov 03 13:50:38.823789 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:50:38.823758 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4983c732-1ead-4068-9bae-511cfd3ae7c3-maas-api-openshift-ca" pod "4983c732-1ead-4068-9bae-511cfd3ae7c3" (UID: "4983c732-1ead-4068-9bae-511cfd3ae7c3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:50:38.825805 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:50:38.825783 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4983c732-1ead-4068-9bae-511cfd3ae7c3-kube-api-access-hg7p6" pod "4983c732-1ead-4068-9bae-511cfd3ae7c3" (UID: "4983c732-1ead-4068-9bae-511cfd3ae7c3"). InnerVolumeSpecName "kube-api-access-hg7p6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:50:38.826277 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:50:38.826257 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4983c732-1ead-4068-9bae-511cfd3ae7c3-maas-api-xks-ca" pod "4983c732-1ead-4068-9bae-511cfd3ae7c3" (UID: "4983c732-1ead-4068-9bae-511cfd3ae7c3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 13:50:38.924997 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:50:38.924922 5024 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4983c732-1ead-4068-9bae-511cfd3ae7c3-maas-api-openshift-ca\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:50:38.924997 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:50:38.924953 5024 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4983c732-1ead-4068-9bae-511cfd3ae7c3-maas-api-xks-ca\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:50:38.924997 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:50:38.924966 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hg7p6\" (UniqueName: \"kubernetes.io/projected/4983c732-1ead-4068-9bae-511cfd3ae7c3-kube-api-access-hg7p6\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:50:38.956271 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:50:38.956243 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d31f7f2a-nv2ng" Nov 03 13:50:38.956427 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:50:38.956274 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d31f7f2a-nv2ng" event={"ID":"4983c732-1ead-4068-9bae-511cfd3ae7c3","Type":"ContainerDied","Data":"ca57c8d46fc47bc6ee11514a8543d1d860b578942551752ff9fd7fafce412eda"} Nov 03 13:50:38.956427 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:50:38.956335 5024 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ca57c8d46fc47bc6ee11514a8543d1d860b578942551752ff9fd7fafce412eda" Nov 03 13:50:39.506936 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:50:39.506900 5024 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d7mwz" containerName="extractor" Nov 03 13:50:44.951843 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:50:44.951808 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a80c6bd9-qpdr2"] Nov 03 13:50:44.952327 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:50:44.952304 5024 cpu_manager.go:414] "containerMap: removing container" podUID="4983c732-1ead-4068-9bae-511cfd3ae7c3" containerName="revoke-keys" Nov 03 13:50:44.952327 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:50:44.952323 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4983c732-1ead-4068-9bae-511cfd3ae7c3" containerName="revoke-keys" Nov 03 13:50:44.952478 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:50:44.952448 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="4983c732-1ead-4068-9bae-511cfd3ae7c3" containerName="revoke-keys" Nov 03 13:50:44.955988 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:50:44.955953 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a80c6bd9-qpdr2" Nov 03 13:50:44.964382 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:50:44.964355 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 13:50:44.964382 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:50:44.964355 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 13:50:44.966267 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:50:44.965723 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a80c6bd9-qpdr2"] Nov 03 13:50:44.966267 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:50:44.965867 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 13:50:44.966826 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:50:44.966806 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-phqfq\"" Nov 03 13:50:44.980168 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:50:44.980133 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dcffa56d-1f8e-4aab-9105-db5888e2bf4a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a80c6bd9-qpdr2\" (UID: \"dcffa56d-1f8e-4aab-9105-db5888e2bf4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a80c6bd9-qpdr2" Nov 03 13:50:44.980320 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:50:44.980170 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dh4gm\" (UniqueName: \"kubernetes.io/projected/dcffa56d-1f8e-4aab-9105-db5888e2bf4a-kube-api-access-dh4gm\") pod \"maas-api-revoke-sub-a80c6bd9-qpdr2\" (UID: \"dcffa56d-1f8e-4aab-9105-db5888e2bf4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a80c6bd9-qpdr2" Nov 03 13:50:44.980320 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:50:44.980202 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dcffa56d-1f8e-4aab-9105-db5888e2bf4a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a80c6bd9-qpdr2\" (UID: \"dcffa56d-1f8e-4aab-9105-db5888e2bf4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a80c6bd9-qpdr2" Nov 03 13:50:45.081254 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:50:45.081209 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dcffa56d-1f8e-4aab-9105-db5888e2bf4a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a80c6bd9-qpdr2\" (UID: \"dcffa56d-1f8e-4aab-9105-db5888e2bf4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a80c6bd9-qpdr2" Nov 03 13:50:45.081254 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:50:45.081255 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dh4gm\" (UniqueName: \"kubernetes.io/projected/dcffa56d-1f8e-4aab-9105-db5888e2bf4a-kube-api-access-dh4gm\") pod \"maas-api-revoke-sub-a80c6bd9-qpdr2\" (UID: \"dcffa56d-1f8e-4aab-9105-db5888e2bf4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a80c6bd9-qpdr2" Nov 03 13:50:45.081508 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:50:45.081295 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dcffa56d-1f8e-4aab-9105-db5888e2bf4a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a80c6bd9-qpdr2\" (UID: \"dcffa56d-1f8e-4aab-9105-db5888e2bf4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a80c6bd9-qpdr2" Nov 03 13:50:45.082018 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:50:45.081987 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dcffa56d-1f8e-4aab-9105-db5888e2bf4a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a80c6bd9-qpdr2\" (UID: \"dcffa56d-1f8e-4aab-9105-db5888e2bf4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a80c6bd9-qpdr2" Nov 03 13:50:45.083604 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:50:45.083584 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dcffa56d-1f8e-4aab-9105-db5888e2bf4a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a80c6bd9-qpdr2\" (UID: \"dcffa56d-1f8e-4aab-9105-db5888e2bf4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a80c6bd9-qpdr2" Nov 03 13:50:45.091680 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:50:45.091652 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dh4gm\" (UniqueName: \"kubernetes.io/projected/dcffa56d-1f8e-4aab-9105-db5888e2bf4a-kube-api-access-dh4gm\") pod \"maas-api-revoke-sub-a80c6bd9-qpdr2\" (UID: \"dcffa56d-1f8e-4aab-9105-db5888e2bf4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a80c6bd9-qpdr2" Nov 03 13:50:45.275207 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:50:45.275118 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a80c6bd9-qpdr2" Nov 03 13:50:45.557437 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:50:45.557411 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a80c6bd9-qpdr2"] Nov 03 13:50:45.559955 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:50:45.559919 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddcffa56d_1f8e_4aab_9105_db5888e2bf4a.slice/crio-49aab783ceeb87df98d559822962a3cf7b8e3b6978e1daabc1f1e7ffaf5b2020 WatchSource:0}: Error finding container 49aab783ceeb87df98d559822962a3cf7b8e3b6978e1daabc1f1e7ffaf5b2020: Status 404 returned error can't find the container with id 49aab783ceeb87df98d559822962a3cf7b8e3b6978e1daabc1f1e7ffaf5b2020 Nov 03 13:50:45.997231 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:50:45.997194 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a80c6bd9-qpdr2" event={"ID":"dcffa56d-1f8e-4aab-9105-db5888e2bf4a","Type":"ContainerStarted","Data":"a60f05635491f6a858b65a039c18c13423efa3069de8c5356b613340e6db9f5f"} Nov 03 13:50:45.997723 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:50:45.997252 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a80c6bd9-qpdr2" event={"ID":"dcffa56d-1f8e-4aab-9105-db5888e2bf4a","Type":"ContainerStarted","Data":"49aab783ceeb87df98d559822962a3cf7b8e3b6978e1daabc1f1e7ffaf5b2020"} Nov 03 13:50:46.019684 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:50:46.019635 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a80c6bd9-qpdr2" podStartSLOduration=2.019621785 podStartE2EDuration="2.019621785s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:50:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:50:46.017317575 +0000 UTC m=+2526.025508855" watchObservedRunningTime="2026-11-03 13:50:46.019621785 +0000 UTC m=+2526.027813061" Nov 03 13:50:47.016308 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:50:47.015410 5024 generic.go:356] "Generic (PLEG): container finished" podID="dcffa56d-1f8e-4aab-9105-db5888e2bf4a" containerID="a60f05635491f6a858b65a039c18c13423efa3069de8c5356b613340e6db9f5f" exitCode=0 Nov 03 13:50:47.016308 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:50:47.015566 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a80c6bd9-qpdr2" event={"ID":"dcffa56d-1f8e-4aab-9105-db5888e2bf4a","Type":"ContainerDied","Data":"a60f05635491f6a858b65a039c18c13423efa3069de8c5356b613340e6db9f5f"} Nov 03 13:50:48.779691 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:50:48.779654 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a80c6bd9-qpdr2" Nov 03 13:50:48.921690 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:50:48.920751 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dcffa56d-1f8e-4aab-9105-db5888e2bf4a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dcffa56d-1f8e-4aab-9105-db5888e2bf4a-maas-api-openshift-ca\") pod \"dcffa56d-1f8e-4aab-9105-db5888e2bf4a\" (UID: \"dcffa56d-1f8e-4aab-9105-db5888e2bf4a\") " Nov 03 13:50:48.921690 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:50:48.920836 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dcffa56d-1f8e-4aab-9105-db5888e2bf4a-kube-api-access-dh4gm\" (UniqueName: \"kubernetes.io/projected/dcffa56d-1f8e-4aab-9105-db5888e2bf4a-kube-api-access-dh4gm\") pod \"dcffa56d-1f8e-4aab-9105-db5888e2bf4a\" (UID: \"dcffa56d-1f8e-4aab-9105-db5888e2bf4a\") " Nov 03 13:50:48.921690 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:50:48.920969 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dcffa56d-1f8e-4aab-9105-db5888e2bf4a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dcffa56d-1f8e-4aab-9105-db5888e2bf4a-maas-api-xks-ca\") pod \"dcffa56d-1f8e-4aab-9105-db5888e2bf4a\" (UID: \"dcffa56d-1f8e-4aab-9105-db5888e2bf4a\") " Nov 03 13:50:48.944045 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:50:48.943703 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dcffa56d-1f8e-4aab-9105-db5888e2bf4a-maas-api-openshift-ca" pod "dcffa56d-1f8e-4aab-9105-db5888e2bf4a" (UID: "dcffa56d-1f8e-4aab-9105-db5888e2bf4a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:50:48.947686 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:50:48.947596 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dcffa56d-1f8e-4aab-9105-db5888e2bf4a-kube-api-access-dh4gm" pod "dcffa56d-1f8e-4aab-9105-db5888e2bf4a" (UID: "dcffa56d-1f8e-4aab-9105-db5888e2bf4a"). InnerVolumeSpecName "kube-api-access-dh4gm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:50:48.962353 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:50:48.961477 5024 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-vrbqc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 13:50:48.962353 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:50:48.961552 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-vrbqc" podUID="d12a0b88-93e3-4542-90b0-56a37dd91387" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 13:50:48.962553 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:50:48.962445 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dcffa56d-1f8e-4aab-9105-db5888e2bf4a-maas-api-xks-ca" pod "dcffa56d-1f8e-4aab-9105-db5888e2bf4a" (UID: "dcffa56d-1f8e-4aab-9105-db5888e2bf4a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 13:50:49.022302 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:50:49.022196 5024 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dcffa56d-1f8e-4aab-9105-db5888e2bf4a-maas-api-xks-ca\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:50:49.022302 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:50:49.022229 5024 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dcffa56d-1f8e-4aab-9105-db5888e2bf4a-maas-api-openshift-ca\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:50:49.022302 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:50:49.022239 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dh4gm\" (UniqueName: \"kubernetes.io/projected/dcffa56d-1f8e-4aab-9105-db5888e2bf4a-kube-api-access-dh4gm\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:50:49.038322 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:50:49.037954 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a80c6bd9-qpdr2" event={"ID":"dcffa56d-1f8e-4aab-9105-db5888e2bf4a","Type":"ContainerDied","Data":"49aab783ceeb87df98d559822962a3cf7b8e3b6978e1daabc1f1e7ffaf5b2020"} Nov 03 13:50:49.038322 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:50:49.038005 5024 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="49aab783ceeb87df98d559822962a3cf7b8e3b6978e1daabc1f1e7ffaf5b2020" Nov 03 13:50:49.038322 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:50:49.038060 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a80c6bd9-qpdr2" Nov 03 13:50:55.170081 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:50:55.170035 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c8af12ec-qvb2g"] Nov 03 13:50:55.170791 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:50:55.170764 5024 cpu_manager.go:414] "containerMap: removing container" podUID="dcffa56d-1f8e-4aab-9105-db5888e2bf4a" containerName="revoke-keys" Nov 03 13:50:55.170844 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:50:55.170796 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dcffa56d-1f8e-4aab-9105-db5888e2bf4a" containerName="revoke-keys" Nov 03 13:50:55.170928 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:50:55.170916 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="dcffa56d-1f8e-4aab-9105-db5888e2bf4a" containerName="revoke-keys" Nov 03 13:50:55.174659 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:50:55.174633 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8af12ec-qvb2g" Nov 03 13:50:55.180894 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:50:55.180860 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c8af12ec-qvb2g"] Nov 03 13:50:55.181768 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:50:55.181735 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 13:50:55.181915 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:50:55.181816 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 13:50:55.181993 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:50:55.181935 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-phqfq\"" Nov 03 13:50:55.181993 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:50:55.181985 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 13:50:55.280855 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:50:55.280811 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wbkfp\" (UniqueName: \"kubernetes.io/projected/57920e64-cc2d-476d-8ede-92077229ab21-kube-api-access-wbkfp\") pod \"maas-api-revoke-sub-c8af12ec-qvb2g\" (UID: \"57920e64-cc2d-476d-8ede-92077229ab21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8af12ec-qvb2g" Nov 03 13:50:55.281060 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:50:55.280864 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/57920e64-cc2d-476d-8ede-92077229ab21-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c8af12ec-qvb2g\" (UID: \"57920e64-cc2d-476d-8ede-92077229ab21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8af12ec-qvb2g" Nov 03 13:50:55.281060 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:50:55.280969 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/57920e64-cc2d-476d-8ede-92077229ab21-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c8af12ec-qvb2g\" (UID: \"57920e64-cc2d-476d-8ede-92077229ab21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8af12ec-qvb2g" Nov 03 13:50:55.381836 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:50:55.381802 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wbkfp\" (UniqueName: \"kubernetes.io/projected/57920e64-cc2d-476d-8ede-92077229ab21-kube-api-access-wbkfp\") pod \"maas-api-revoke-sub-c8af12ec-qvb2g\" (UID: \"57920e64-cc2d-476d-8ede-92077229ab21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8af12ec-qvb2g" Nov 03 13:50:55.382040 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:50:55.381857 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/57920e64-cc2d-476d-8ede-92077229ab21-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c8af12ec-qvb2g\" (UID: \"57920e64-cc2d-476d-8ede-92077229ab21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8af12ec-qvb2g" Nov 03 13:50:55.382040 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:50:55.381919 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/57920e64-cc2d-476d-8ede-92077229ab21-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c8af12ec-qvb2g\" (UID: \"57920e64-cc2d-476d-8ede-92077229ab21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8af12ec-qvb2g" Nov 03 13:50:55.382804 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:50:55.382772 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/57920e64-cc2d-476d-8ede-92077229ab21-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c8af12ec-qvb2g\" (UID: \"57920e64-cc2d-476d-8ede-92077229ab21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8af12ec-qvb2g" Nov 03 13:50:55.384413 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:50:55.384384 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/57920e64-cc2d-476d-8ede-92077229ab21-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c8af12ec-qvb2g\" (UID: \"57920e64-cc2d-476d-8ede-92077229ab21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8af12ec-qvb2g" Nov 03 13:50:55.391410 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:50:55.391384 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wbkfp\" (UniqueName: \"kubernetes.io/projected/57920e64-cc2d-476d-8ede-92077229ab21-kube-api-access-wbkfp\") pod \"maas-api-revoke-sub-c8af12ec-qvb2g\" (UID: \"57920e64-cc2d-476d-8ede-92077229ab21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8af12ec-qvb2g" Nov 03 13:50:55.492714 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:50:55.492644 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8af12ec-qvb2g" Nov 03 13:50:55.767428 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:50:55.767400 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c8af12ec-qvb2g"] Nov 03 13:50:55.768471 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:50:55.768442 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57920e64_cc2d_476d_8ede_92077229ab21.slice/crio-d99ee7a7f634f445a8abc5cc51ef23e8c0e954776e2f47518e82edebbbb3c0fd WatchSource:0}: Error finding container d99ee7a7f634f445a8abc5cc51ef23e8c0e954776e2f47518e82edebbbb3c0fd: Status 404 returned error can't find the container with id d99ee7a7f634f445a8abc5cc51ef23e8c0e954776e2f47518e82edebbbb3c0fd Nov 03 13:50:56.077232 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:50:56.077141 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8af12ec-qvb2g" event={"ID":"57920e64-cc2d-476d-8ede-92077229ab21","Type":"ContainerStarted","Data":"d99ee7a7f634f445a8abc5cc51ef23e8c0e954776e2f47518e82edebbbb3c0fd"} Nov 03 13:50:57.099362 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:50:57.092817 5024 generic.go:356] "Generic (PLEG): container finished" podID="57920e64-cc2d-476d-8ede-92077229ab21" containerID="ab1de31f036da9d5134c9dd10d23365e3fed4330366df5804af2ee642948d920" exitCode=0 Nov 03 13:50:57.099362 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:50:57.092980 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8af12ec-qvb2g" event={"ID":"57920e64-cc2d-476d-8ede-92077229ab21","Type":"ContainerDied","Data":"ab1de31f036da9d5134c9dd10d23365e3fed4330366df5804af2ee642948d920"} Nov 03 13:50:58.939479 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:50:58.939027 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8af12ec-qvb2g" Nov 03 13:50:59.043317 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:50:59.041359 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/57920e64-cc2d-476d-8ede-92077229ab21-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/57920e64-cc2d-476d-8ede-92077229ab21-maas-api-openshift-ca\") pod \"57920e64-cc2d-476d-8ede-92077229ab21\" (UID: \"57920e64-cc2d-476d-8ede-92077229ab21\") " Nov 03 13:50:59.043317 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:50:59.041501 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/57920e64-cc2d-476d-8ede-92077229ab21-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/57920e64-cc2d-476d-8ede-92077229ab21-maas-api-xks-ca\") pod \"57920e64-cc2d-476d-8ede-92077229ab21\" (UID: \"57920e64-cc2d-476d-8ede-92077229ab21\") " Nov 03 13:50:59.043317 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:50:59.041569 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57920e64-cc2d-476d-8ede-92077229ab21-kube-api-access-wbkfp\" (UniqueName: \"kubernetes.io/projected/57920e64-cc2d-476d-8ede-92077229ab21-kube-api-access-wbkfp\") pod \"57920e64-cc2d-476d-8ede-92077229ab21\" (UID: \"57920e64-cc2d-476d-8ede-92077229ab21\") " Nov 03 13:50:59.043317 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:50:59.042865 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57920e64-cc2d-476d-8ede-92077229ab21-maas-api-openshift-ca" pod "57920e64-cc2d-476d-8ede-92077229ab21" (UID: "57920e64-cc2d-476d-8ede-92077229ab21"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:50:59.055592 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:50:59.055559 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/57920e64-cc2d-476d-8ede-92077229ab21-maas-api-xks-ca" pod "57920e64-cc2d-476d-8ede-92077229ab21" (UID: "57920e64-cc2d-476d-8ede-92077229ab21"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 13:50:59.057155 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:50:59.056923 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57920e64-cc2d-476d-8ede-92077229ab21-kube-api-access-wbkfp" pod "57920e64-cc2d-476d-8ede-92077229ab21" (UID: "57920e64-cc2d-476d-8ede-92077229ab21"). InnerVolumeSpecName "kube-api-access-wbkfp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:50:59.117205 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:50:59.117165 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8af12ec-qvb2g" Nov 03 13:50:59.117205 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:50:59.117179 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8af12ec-qvb2g" event={"ID":"57920e64-cc2d-476d-8ede-92077229ab21","Type":"ContainerDied","Data":"d99ee7a7f634f445a8abc5cc51ef23e8c0e954776e2f47518e82edebbbb3c0fd"} Nov 03 13:50:59.117425 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:50:59.117238 5024 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d99ee7a7f634f445a8abc5cc51ef23e8c0e954776e2f47518e82edebbbb3c0fd" Nov 03 13:50:59.143432 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:50:59.143396 5024 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/57920e64-cc2d-476d-8ede-92077229ab21-maas-api-openshift-ca\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:50:59.143574 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:50:59.143435 5024 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/57920e64-cc2d-476d-8ede-92077229ab21-maas-api-xks-ca\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:50:59.143574 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:50:59.143453 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wbkfp\" (UniqueName: \"kubernetes.io/projected/57920e64-cc2d-476d-8ede-92077229ab21-kube-api-access-wbkfp\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:51:05.410306 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:05.410253 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a9c0c6e9-frrsm"] Nov 03 13:51:05.411530 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:05.411489 5024 cpu_manager.go:414] "containerMap: removing container" podUID="57920e64-cc2d-476d-8ede-92077229ab21" containerName="revoke-keys" Nov 03 13:51:05.411530 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:05.411521 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57920e64-cc2d-476d-8ede-92077229ab21" containerName="revoke-keys" Nov 03 13:51:05.411812 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:05.411652 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="57920e64-cc2d-476d-8ede-92077229ab21" containerName="revoke-keys" Nov 03 13:51:05.415033 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:05.415011 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9c0c6e9-frrsm" Nov 03 13:51:05.420563 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:05.420539 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 13:51:05.420959 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:05.420934 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-phqfq\"" Nov 03 13:51:05.421068 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:05.421022 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 13:51:05.421954 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:05.421936 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 13:51:05.422892 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:05.422867 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a9c0c6e9-frrsm"] Nov 03 13:51:05.503377 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:05.503331 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0919e155-6632-4b77-bccd-de89cc4847ad-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a9c0c6e9-frrsm\" (UID: \"0919e155-6632-4b77-bccd-de89cc4847ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9c0c6e9-frrsm" Nov 03 13:51:05.503542 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:05.503473 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0919e155-6632-4b77-bccd-de89cc4847ad-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a9c0c6e9-frrsm\" (UID: \"0919e155-6632-4b77-bccd-de89cc4847ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9c0c6e9-frrsm" Nov 03 13:51:05.503542 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:05.503524 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2cl8m\" (UniqueName: \"kubernetes.io/projected/0919e155-6632-4b77-bccd-de89cc4847ad-kube-api-access-2cl8m\") pod \"maas-api-revoke-sub-a9c0c6e9-frrsm\" (UID: \"0919e155-6632-4b77-bccd-de89cc4847ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9c0c6e9-frrsm" Nov 03 13:51:05.604751 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:05.604714 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0919e155-6632-4b77-bccd-de89cc4847ad-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a9c0c6e9-frrsm\" (UID: \"0919e155-6632-4b77-bccd-de89cc4847ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9c0c6e9-frrsm" Nov 03 13:51:05.604939 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:05.604759 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2cl8m\" (UniqueName: \"kubernetes.io/projected/0919e155-6632-4b77-bccd-de89cc4847ad-kube-api-access-2cl8m\") pod \"maas-api-revoke-sub-a9c0c6e9-frrsm\" (UID: \"0919e155-6632-4b77-bccd-de89cc4847ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9c0c6e9-frrsm" Nov 03 13:51:05.604939 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:05.604845 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0919e155-6632-4b77-bccd-de89cc4847ad-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a9c0c6e9-frrsm\" (UID: \"0919e155-6632-4b77-bccd-de89cc4847ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9c0c6e9-frrsm" Nov 03 13:51:05.605584 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:05.605562 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0919e155-6632-4b77-bccd-de89cc4847ad-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a9c0c6e9-frrsm\" (UID: \"0919e155-6632-4b77-bccd-de89cc4847ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9c0c6e9-frrsm" Nov 03 13:51:05.607259 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:05.607235 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0919e155-6632-4b77-bccd-de89cc4847ad-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a9c0c6e9-frrsm\" (UID: \"0919e155-6632-4b77-bccd-de89cc4847ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9c0c6e9-frrsm" Nov 03 13:51:05.614907 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:05.614880 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2cl8m\" (UniqueName: \"kubernetes.io/projected/0919e155-6632-4b77-bccd-de89cc4847ad-kube-api-access-2cl8m\") pod \"maas-api-revoke-sub-a9c0c6e9-frrsm\" (UID: \"0919e155-6632-4b77-bccd-de89cc4847ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9c0c6e9-frrsm" Nov 03 13:51:05.738701 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:05.738614 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9c0c6e9-frrsm" Nov 03 13:51:06.022383 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:06.022357 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a9c0c6e9-frrsm"] Nov 03 13:51:06.024245 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:51:06.024209 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0919e155_6632_4b77_bccd_de89cc4847ad.slice/crio-66cc067b62a2e3e0bffe121088743e8b17b9831b92b311ba2e3d07d5aa23c644 WatchSource:0}: Error finding container 66cc067b62a2e3e0bffe121088743e8b17b9831b92b311ba2e3d07d5aa23c644: Status 404 returned error can't find the container with id 66cc067b62a2e3e0bffe121088743e8b17b9831b92b311ba2e3d07d5aa23c644 Nov 03 13:51:06.153137 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:06.153103 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9c0c6e9-frrsm" event={"ID":"0919e155-6632-4b77-bccd-de89cc4847ad","Type":"ContainerStarted","Data":"66cc067b62a2e3e0bffe121088743e8b17b9831b92b311ba2e3d07d5aa23c644"} Nov 03 13:51:07.172117 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:07.171585 5024 generic.go:356] "Generic (PLEG): container finished" podID="0919e155-6632-4b77-bccd-de89cc4847ad" containerID="32743266377cb4a0d9f5b923a71523eea2de494a3486e963c83ff38fef94bc03" exitCode=0 Nov 03 13:51:07.172117 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:07.171740 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9c0c6e9-frrsm" event={"ID":"0919e155-6632-4b77-bccd-de89cc4847ad","Type":"ContainerDied","Data":"32743266377cb4a0d9f5b923a71523eea2de494a3486e963c83ff38fef94bc03"} Nov 03 13:51:08.565138 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:08.565113 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9c0c6e9-frrsm" Nov 03 13:51:08.663222 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:08.663187 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0919e155-6632-4b77-bccd-de89cc4847ad-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0919e155-6632-4b77-bccd-de89cc4847ad-maas-api-xks-ca\") pod \"0919e155-6632-4b77-bccd-de89cc4847ad\" (UID: \"0919e155-6632-4b77-bccd-de89cc4847ad\") " Nov 03 13:51:08.663437 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:08.663261 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0919e155-6632-4b77-bccd-de89cc4847ad-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0919e155-6632-4b77-bccd-de89cc4847ad-maas-api-openshift-ca\") pod \"0919e155-6632-4b77-bccd-de89cc4847ad\" (UID: \"0919e155-6632-4b77-bccd-de89cc4847ad\") " Nov 03 13:51:08.663437 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:08.663302 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0919e155-6632-4b77-bccd-de89cc4847ad-kube-api-access-2cl8m\" (UniqueName: \"kubernetes.io/projected/0919e155-6632-4b77-bccd-de89cc4847ad-kube-api-access-2cl8m\") pod \"0919e155-6632-4b77-bccd-de89cc4847ad\" (UID: \"0919e155-6632-4b77-bccd-de89cc4847ad\") " Nov 03 13:51:08.663682 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:08.663658 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0919e155-6632-4b77-bccd-de89cc4847ad-maas-api-openshift-ca" pod "0919e155-6632-4b77-bccd-de89cc4847ad" (UID: "0919e155-6632-4b77-bccd-de89cc4847ad"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:51:08.665380 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:08.665359 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0919e155-6632-4b77-bccd-de89cc4847ad-maas-api-xks-ca" pod "0919e155-6632-4b77-bccd-de89cc4847ad" (UID: "0919e155-6632-4b77-bccd-de89cc4847ad"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 13:51:08.665523 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:08.665504 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0919e155-6632-4b77-bccd-de89cc4847ad-kube-api-access-2cl8m" pod "0919e155-6632-4b77-bccd-de89cc4847ad" (UID: "0919e155-6632-4b77-bccd-de89cc4847ad"). InnerVolumeSpecName "kube-api-access-2cl8m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:51:08.764784 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:08.764724 5024 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0919e155-6632-4b77-bccd-de89cc4847ad-maas-api-xks-ca\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:51:08.764784 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:08.764750 5024 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0919e155-6632-4b77-bccd-de89cc4847ad-maas-api-openshift-ca\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:51:08.764784 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:08.764760 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2cl8m\" (UniqueName: \"kubernetes.io/projected/0919e155-6632-4b77-bccd-de89cc4847ad-kube-api-access-2cl8m\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:51:09.195749 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:09.195713 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9c0c6e9-frrsm" Nov 03 13:51:09.195749 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:09.195731 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9c0c6e9-frrsm" event={"ID":"0919e155-6632-4b77-bccd-de89cc4847ad","Type":"ContainerDied","Data":"66cc067b62a2e3e0bffe121088743e8b17b9831b92b311ba2e3d07d5aa23c644"} Nov 03 13:51:09.195959 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:09.195822 5024 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="66cc067b62a2e3e0bffe121088743e8b17b9831b92b311ba2e3d07d5aa23c644" Nov 03 13:51:09.506642 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:51:09.506563 5024 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d7mwz" containerName="extractor" Nov 03 13:51:22.138790 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:22.138746 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-d969013e-openshift-default-6ff77f8c6c-cl26n"] Nov 03 13:51:22.139464 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:22.139214 5024 cpu_manager.go:414] "containerMap: removing container" podUID="0919e155-6632-4b77-bccd-de89cc4847ad" containerName="revoke-keys" Nov 03 13:51:22.139464 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:22.139229 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0919e155-6632-4b77-bccd-de89cc4847ad" containerName="revoke-keys" Nov 03 13:51:22.139464 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:22.139333 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="0919e155-6632-4b77-bccd-de89cc4847ad" containerName="revoke-keys" Nov 03 13:51:22.142864 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:22.142844 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-d969013e-openshift-default-6ff77f8c6c-cl26n" Nov 03 13:51:22.154060 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:22.154037 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-d969013e-openshift-default-dockercfg-sdr8m\"" Nov 03 13:51:22.157114 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:22.157090 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-d969013e-openshift-default-6ff77f8c6c-cl26n"] Nov 03 13:51:22.231827 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:22.231784 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11fc0ab3-5590-4613-8f68-cf4c75791d3a-istiod-ca-cert\") pod \"e2e-ait-d969013e-openshift-default-6ff77f8c6c-cl26n\" (UID: \"11fc0ab3-5590-4613-8f68-cf4c75791d3a\") " pod="openshift-ingress/e2e-ait-d969013e-openshift-default-6ff77f8c6c-cl26n" Nov 03 13:51:22.232038 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:22.231844 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11fc0ab3-5590-4613-8f68-cf4c75791d3a-credential-socket\") pod \"e2e-ait-d969013e-openshift-default-6ff77f8c6c-cl26n\" (UID: \"11fc0ab3-5590-4613-8f68-cf4c75791d3a\") " pod="openshift-ingress/e2e-ait-d969013e-openshift-default-6ff77f8c6c-cl26n" Nov 03 13:51:22.232038 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:22.231964 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11fc0ab3-5590-4613-8f68-cf4c75791d3a-istio-token\") pod \"e2e-ait-d969013e-openshift-default-6ff77f8c6c-cl26n\" (UID: \"11fc0ab3-5590-4613-8f68-cf4c75791d3a\") " pod="openshift-ingress/e2e-ait-d969013e-openshift-default-6ff77f8c6c-cl26n" Nov 03 13:51:22.232038 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:22.232012 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11fc0ab3-5590-4613-8f68-cf4c75791d3a-istio-envoy\") pod \"e2e-ait-d969013e-openshift-default-6ff77f8c6c-cl26n\" (UID: \"11fc0ab3-5590-4613-8f68-cf4c75791d3a\") " pod="openshift-ingress/e2e-ait-d969013e-openshift-default-6ff77f8c6c-cl26n" Nov 03 13:51:22.232230 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:22.232128 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11fc0ab3-5590-4613-8f68-cf4c75791d3a-workload-socket\") pod \"e2e-ait-d969013e-openshift-default-6ff77f8c6c-cl26n\" (UID: \"11fc0ab3-5590-4613-8f68-cf4c75791d3a\") " pod="openshift-ingress/e2e-ait-d969013e-openshift-default-6ff77f8c6c-cl26n" Nov 03 13:51:22.232230 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:22.232203 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11fc0ab3-5590-4613-8f68-cf4c75791d3a-istio-podinfo\") pod \"e2e-ait-d969013e-openshift-default-6ff77f8c6c-cl26n\" (UID: \"11fc0ab3-5590-4613-8f68-cf4c75791d3a\") " pod="openshift-ingress/e2e-ait-d969013e-openshift-default-6ff77f8c6c-cl26n" Nov 03 13:51:22.234419 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:22.232471 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11fc0ab3-5590-4613-8f68-cf4c75791d3a-istio-data\") pod \"e2e-ait-d969013e-openshift-default-6ff77f8c6c-cl26n\" (UID: \"11fc0ab3-5590-4613-8f68-cf4c75791d3a\") " pod="openshift-ingress/e2e-ait-d969013e-openshift-default-6ff77f8c6c-cl26n" Nov 03 13:51:22.234419 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:22.232584 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xn625\" (UniqueName: \"kubernetes.io/projected/11fc0ab3-5590-4613-8f68-cf4c75791d3a-kube-api-access-xn625\") pod \"e2e-ait-d969013e-openshift-default-6ff77f8c6c-cl26n\" (UID: \"11fc0ab3-5590-4613-8f68-cf4c75791d3a\") " pod="openshift-ingress/e2e-ait-d969013e-openshift-default-6ff77f8c6c-cl26n" Nov 03 13:51:22.234419 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:22.232659 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11fc0ab3-5590-4613-8f68-cf4c75791d3a-workload-certs\") pod \"e2e-ait-d969013e-openshift-default-6ff77f8c6c-cl26n\" (UID: \"11fc0ab3-5590-4613-8f68-cf4c75791d3a\") " pod="openshift-ingress/e2e-ait-d969013e-openshift-default-6ff77f8c6c-cl26n" Nov 03 13:51:22.334365 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:22.334327 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11fc0ab3-5590-4613-8f68-cf4c75791d3a-istio-data\") pod \"e2e-ait-d969013e-openshift-default-6ff77f8c6c-cl26n\" (UID: \"11fc0ab3-5590-4613-8f68-cf4c75791d3a\") " pod="openshift-ingress/e2e-ait-d969013e-openshift-default-6ff77f8c6c-cl26n" Nov 03 13:51:22.334365 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:22.334365 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xn625\" (UniqueName: \"kubernetes.io/projected/11fc0ab3-5590-4613-8f68-cf4c75791d3a-kube-api-access-xn625\") pod \"e2e-ait-d969013e-openshift-default-6ff77f8c6c-cl26n\" (UID: \"11fc0ab3-5590-4613-8f68-cf4c75791d3a\") " pod="openshift-ingress/e2e-ait-d969013e-openshift-default-6ff77f8c6c-cl26n" Nov 03 13:51:22.334642 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:22.334386 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11fc0ab3-5590-4613-8f68-cf4c75791d3a-workload-certs\") pod \"e2e-ait-d969013e-openshift-default-6ff77f8c6c-cl26n\" (UID: \"11fc0ab3-5590-4613-8f68-cf4c75791d3a\") " pod="openshift-ingress/e2e-ait-d969013e-openshift-default-6ff77f8c6c-cl26n" Nov 03 13:51:22.334642 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:22.334564 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11fc0ab3-5590-4613-8f68-cf4c75791d3a-istiod-ca-cert\") pod \"e2e-ait-d969013e-openshift-default-6ff77f8c6c-cl26n\" (UID: \"11fc0ab3-5590-4613-8f68-cf4c75791d3a\") " pod="openshift-ingress/e2e-ait-d969013e-openshift-default-6ff77f8c6c-cl26n" Nov 03 13:51:22.334642 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:22.334611 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11fc0ab3-5590-4613-8f68-cf4c75791d3a-credential-socket\") pod \"e2e-ait-d969013e-openshift-default-6ff77f8c6c-cl26n\" (UID: \"11fc0ab3-5590-4613-8f68-cf4c75791d3a\") " pod="openshift-ingress/e2e-ait-d969013e-openshift-default-6ff77f8c6c-cl26n" Nov 03 13:51:22.334815 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:22.334650 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11fc0ab3-5590-4613-8f68-cf4c75791d3a-istio-token\") pod \"e2e-ait-d969013e-openshift-default-6ff77f8c6c-cl26n\" (UID: \"11fc0ab3-5590-4613-8f68-cf4c75791d3a\") " pod="openshift-ingress/e2e-ait-d969013e-openshift-default-6ff77f8c6c-cl26n" Nov 03 13:51:22.334815 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:22.334678 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11fc0ab3-5590-4613-8f68-cf4c75791d3a-istio-envoy\") pod \"e2e-ait-d969013e-openshift-default-6ff77f8c6c-cl26n\" (UID: \"11fc0ab3-5590-4613-8f68-cf4c75791d3a\") " pod="openshift-ingress/e2e-ait-d969013e-openshift-default-6ff77f8c6c-cl26n" Nov 03 13:51:22.334815 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:22.334768 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11fc0ab3-5590-4613-8f68-cf4c75791d3a-workload-socket\") pod \"e2e-ait-d969013e-openshift-default-6ff77f8c6c-cl26n\" (UID: \"11fc0ab3-5590-4613-8f68-cf4c75791d3a\") " pod="openshift-ingress/e2e-ait-d969013e-openshift-default-6ff77f8c6c-cl26n" Nov 03 13:51:22.334984 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:22.334818 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11fc0ab3-5590-4613-8f68-cf4c75791d3a-workload-certs\") pod \"e2e-ait-d969013e-openshift-default-6ff77f8c6c-cl26n\" (UID: \"11fc0ab3-5590-4613-8f68-cf4c75791d3a\") " pod="openshift-ingress/e2e-ait-d969013e-openshift-default-6ff77f8c6c-cl26n" Nov 03 13:51:22.334984 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:22.334832 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11fc0ab3-5590-4613-8f68-cf4c75791d3a-istio-podinfo\") pod \"e2e-ait-d969013e-openshift-default-6ff77f8c6c-cl26n\" (UID: \"11fc0ab3-5590-4613-8f68-cf4c75791d3a\") " pod="openshift-ingress/e2e-ait-d969013e-openshift-default-6ff77f8c6c-cl26n" Nov 03 13:51:22.334984 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:22.334844 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11fc0ab3-5590-4613-8f68-cf4c75791d3a-istio-data\") pod \"e2e-ait-d969013e-openshift-default-6ff77f8c6c-cl26n\" (UID: \"11fc0ab3-5590-4613-8f68-cf4c75791d3a\") " pod="openshift-ingress/e2e-ait-d969013e-openshift-default-6ff77f8c6c-cl26n" Nov 03 13:51:22.335145 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:22.334978 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11fc0ab3-5590-4613-8f68-cf4c75791d3a-credential-socket\") pod \"e2e-ait-d969013e-openshift-default-6ff77f8c6c-cl26n\" (UID: \"11fc0ab3-5590-4613-8f68-cf4c75791d3a\") " pod="openshift-ingress/e2e-ait-d969013e-openshift-default-6ff77f8c6c-cl26n" Nov 03 13:51:22.335206 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:22.335156 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11fc0ab3-5590-4613-8f68-cf4c75791d3a-workload-socket\") pod \"e2e-ait-d969013e-openshift-default-6ff77f8c6c-cl26n\" (UID: \"11fc0ab3-5590-4613-8f68-cf4c75791d3a\") " pod="openshift-ingress/e2e-ait-d969013e-openshift-default-6ff77f8c6c-cl26n" Nov 03 13:51:22.335380 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:22.335360 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11fc0ab3-5590-4613-8f68-cf4c75791d3a-istiod-ca-cert\") pod \"e2e-ait-d969013e-openshift-default-6ff77f8c6c-cl26n\" (UID: \"11fc0ab3-5590-4613-8f68-cf4c75791d3a\") " pod="openshift-ingress/e2e-ait-d969013e-openshift-default-6ff77f8c6c-cl26n" Nov 03 13:51:22.337145 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:22.337115 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11fc0ab3-5590-4613-8f68-cf4c75791d3a-istio-envoy\") pod \"e2e-ait-d969013e-openshift-default-6ff77f8c6c-cl26n\" (UID: \"11fc0ab3-5590-4613-8f68-cf4c75791d3a\") " pod="openshift-ingress/e2e-ait-d969013e-openshift-default-6ff77f8c6c-cl26n" Nov 03 13:51:22.337492 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:22.337471 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11fc0ab3-5590-4613-8f68-cf4c75791d3a-istio-podinfo\") pod \"e2e-ait-d969013e-openshift-default-6ff77f8c6c-cl26n\" (UID: \"11fc0ab3-5590-4613-8f68-cf4c75791d3a\") " pod="openshift-ingress/e2e-ait-d969013e-openshift-default-6ff77f8c6c-cl26n" Nov 03 13:51:22.345048 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:22.345026 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11fc0ab3-5590-4613-8f68-cf4c75791d3a-istio-token\") pod \"e2e-ait-d969013e-openshift-default-6ff77f8c6c-cl26n\" (UID: \"11fc0ab3-5590-4613-8f68-cf4c75791d3a\") " pod="openshift-ingress/e2e-ait-d969013e-openshift-default-6ff77f8c6c-cl26n" Nov 03 13:51:22.345151 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:22.345102 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xn625\" (UniqueName: \"kubernetes.io/projected/11fc0ab3-5590-4613-8f68-cf4c75791d3a-kube-api-access-xn625\") pod \"e2e-ait-d969013e-openshift-default-6ff77f8c6c-cl26n\" (UID: \"11fc0ab3-5590-4613-8f68-cf4c75791d3a\") " pod="openshift-ingress/e2e-ait-d969013e-openshift-default-6ff77f8c6c-cl26n" Nov 03 13:51:22.456472 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:22.456392 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-d969013e-openshift-default-6ff77f8c6c-cl26n" Nov 03 13:51:22.737916 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:22.737874 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-d969013e-openshift-default-6ff77f8c6c-cl26n"] Nov 03 13:51:22.739114 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:51:22.739081 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11fc0ab3_5590_4613_8f68_cf4c75791d3a.slice/crio-e56a0bd7600423294476803d7dd74c020d80a436a789a21bdbdb82f391c27450 WatchSource:0}: Error finding container e56a0bd7600423294476803d7dd74c020d80a436a789a21bdbdb82f391c27450: Status 404 returned error can't find the container with id e56a0bd7600423294476803d7dd74c020d80a436a789a21bdbdb82f391c27450 Nov 03 13:51:22.741426 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:22.741387 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:51:22.741530 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:22.741473 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:51:22.741575 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:22.741527 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:51:23.292134 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:23.292096 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d969013e-openshift-default-6ff77f8c6c-cl26n" event={"ID":"11fc0ab3-5590-4613-8f68-cf4c75791d3a","Type":"ContainerStarted","Data":"4feb3e110f9f7aa45cacffefb63d1feb4143a18cb54e83cb1ad195d5a4ccbbb8"} Nov 03 13:51:23.292882 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:23.292157 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d969013e-openshift-default-6ff77f8c6c-cl26n" event={"ID":"11fc0ab3-5590-4613-8f68-cf4c75791d3a","Type":"ContainerStarted","Data":"e56a0bd7600423294476803d7dd74c020d80a436a789a21bdbdb82f391c27450"} Nov 03 13:51:23.320649 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:23.320586 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-d969013e-openshift-default-6ff77f8c6c-cl26n" podStartSLOduration=1.320567562 podStartE2EDuration="1.320567562s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:51:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:51:23.319121882 +0000 UTC m=+2563.327313159" watchObservedRunningTime="2026-11-03 13:51:23.320567562 +0000 UTC m=+2563.328758839" Nov 03 13:51:23.456983 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:23.456948 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-d969013e-openshift-default-6ff77f8c6c-cl26n" Nov 03 13:51:23.462276 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:23.462249 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-d969013e-openshift-default-6ff77f8c6c-cl26n" Nov 03 13:51:24.299827 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:24.299789 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-d969013e-openshift-default-6ff77f8c6c-cl26n" Nov 03 13:51:24.300982 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:24.300952 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-d969013e-openshift-default-6ff77f8c6c-cl26n" Nov 03 13:51:25.963911 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:25.963868 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d969013e-575cbbfb6-6wf8s"] Nov 03 13:51:25.967583 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:25.967563 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d969013e-575cbbfb6-6wf8s" Nov 03 13:51:25.976967 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:25.976923 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 13:51:25.977829 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:25.977807 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 13:51:25.979133 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:25.977853 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-p5qd5\"" Nov 03 13:51:25.979133 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:25.977902 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-d969013e-serving-cert\"" Nov 03 13:51:25.979133 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:25.977981 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 03 13:51:25.981831 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:25.981683 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d969013e-575cbbfb6-6wf8s"] Nov 03 13:51:26.076514 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:26.076480 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9338282-ec5b-4805-9d2d-f8be31525f8a-openshift-service-ca\") pod \"maas-api-e2e-ait-d969013e-575cbbfb6-6wf8s\" (UID: \"f9338282-ec5b-4805-9d2d-f8be31525f8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d969013e-575cbbfb6-6wf8s" Nov 03 13:51:26.076514 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:26.076523 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9338282-ec5b-4805-9d2d-f8be31525f8a-metrics-certs\") pod \"maas-api-e2e-ait-d969013e-575cbbfb6-6wf8s\" (UID: \"f9338282-ec5b-4805-9d2d-f8be31525f8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d969013e-575cbbfb6-6wf8s" Nov 03 13:51:26.076731 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:26.076670 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k98rq\" (UniqueName: \"kubernetes.io/projected/f9338282-ec5b-4805-9d2d-f8be31525f8a-kube-api-access-k98rq\") pod \"maas-api-e2e-ait-d969013e-575cbbfb6-6wf8s\" (UID: \"f9338282-ec5b-4805-9d2d-f8be31525f8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d969013e-575cbbfb6-6wf8s" Nov 03 13:51:26.076731 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:26.076703 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f9338282-ec5b-4805-9d2d-f8be31525f8a-maas-api-tls\") pod \"maas-api-e2e-ait-d969013e-575cbbfb6-6wf8s\" (UID: \"f9338282-ec5b-4805-9d2d-f8be31525f8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d969013e-575cbbfb6-6wf8s" Nov 03 13:51:26.135170 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:26.135129 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-d969013e-85c7895599-g4z9h"] Nov 03 13:51:26.138768 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:26.138749 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-d969013e-85c7895599-g4z9h" Nov 03 13:51:26.147274 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:26.147234 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-d969013e\"" Nov 03 13:51:26.147483 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:26.147242 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-d969013e-dockercfg-dn27b\"" Nov 03 13:51:26.149677 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:26.149651 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-d969013e-85c7895599-g4z9h"] Nov 03 13:51:26.163539 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:26.163517 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-d969013e-69795cd9c-fpcj9"] Nov 03 13:51:26.167125 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:26.167101 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-d969013e-69795cd9c-fpcj9" Nov 03 13:51:26.182965 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:26.182939 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9338282-ec5b-4805-9d2d-f8be31525f8a-openshift-service-ca\") pod \"maas-api-e2e-ait-d969013e-575cbbfb6-6wf8s\" (UID: \"f9338282-ec5b-4805-9d2d-f8be31525f8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d969013e-575cbbfb6-6wf8s" Nov 03 13:51:26.183079 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:26.182987 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9338282-ec5b-4805-9d2d-f8be31525f8a-metrics-certs\") pod \"maas-api-e2e-ait-d969013e-575cbbfb6-6wf8s\" (UID: \"f9338282-ec5b-4805-9d2d-f8be31525f8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d969013e-575cbbfb6-6wf8s" Nov 03 13:51:26.183150 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:26.183107 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k98rq\" (UniqueName: \"kubernetes.io/projected/f9338282-ec5b-4805-9d2d-f8be31525f8a-kube-api-access-k98rq\") pod \"maas-api-e2e-ait-d969013e-575cbbfb6-6wf8s\" (UID: \"f9338282-ec5b-4805-9d2d-f8be31525f8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d969013e-575cbbfb6-6wf8s" Nov 03 13:51:26.183150 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:26.183143 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f9338282-ec5b-4805-9d2d-f8be31525f8a-maas-api-tls\") pod \"maas-api-e2e-ait-d969013e-575cbbfb6-6wf8s\" (UID: \"f9338282-ec5b-4805-9d2d-f8be31525f8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d969013e-575cbbfb6-6wf8s" Nov 03 13:51:26.183722 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:26.183697 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9338282-ec5b-4805-9d2d-f8be31525f8a-openshift-service-ca\") pod \"maas-api-e2e-ait-d969013e-575cbbfb6-6wf8s\" (UID: \"f9338282-ec5b-4805-9d2d-f8be31525f8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d969013e-575cbbfb6-6wf8s" Nov 03 13:51:26.186117 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:26.186091 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9338282-ec5b-4805-9d2d-f8be31525f8a-metrics-certs\") pod \"maas-api-e2e-ait-d969013e-575cbbfb6-6wf8s\" (UID: \"f9338282-ec5b-4805-9d2d-f8be31525f8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d969013e-575cbbfb6-6wf8s" Nov 03 13:51:26.186266 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:26.186237 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f9338282-ec5b-4805-9d2d-f8be31525f8a-maas-api-tls\") pod \"maas-api-e2e-ait-d969013e-575cbbfb6-6wf8s\" (UID: \"f9338282-ec5b-4805-9d2d-f8be31525f8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d969013e-575cbbfb6-6wf8s" Nov 03 13:51:26.186408 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:26.186384 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-d969013e-69795cd9c-fpcj9"] Nov 03 13:51:26.192844 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:26.192822 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k98rq\" (UniqueName: \"kubernetes.io/projected/f9338282-ec5b-4805-9d2d-f8be31525f8a-kube-api-access-k98rq\") pod \"maas-api-e2e-ait-d969013e-575cbbfb6-6wf8s\" (UID: \"f9338282-ec5b-4805-9d2d-f8be31525f8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d969013e-575cbbfb6-6wf8s" Nov 03 13:51:26.283778 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:26.283686 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ed4d963-93ff-4553-a493-c8426d121216-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-d969013e-85c7895599-g4z9h\" (UID: \"3ed4d963-93ff-4553-a493-c8426d121216\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-d969013e-85c7895599-g4z9h" Nov 03 13:51:26.283778 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:26.283746 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rbwwx\" (UniqueName: \"kubernetes.io/projected/951922e7-2b2b-496a-9edb-31af5071228b-kube-api-access-rbwwx\") pod \"payload-processing-e2e-ait-d969013e-69795cd9c-fpcj9\" (UID: \"951922e7-2b2b-496a-9edb-31af5071228b\") " pod="openshift-ingress/payload-processing-e2e-ait-d969013e-69795cd9c-fpcj9" Nov 03 13:51:26.284016 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:26.283904 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/951922e7-2b2b-496a-9edb-31af5071228b-plugins-config-volume\") pod \"payload-processing-e2e-ait-d969013e-69795cd9c-fpcj9\" (UID: \"951922e7-2b2b-496a-9edb-31af5071228b\") " pod="openshift-ingress/payload-processing-e2e-ait-d969013e-69795cd9c-fpcj9" Nov 03 13:51:26.284072 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:26.284017 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b54gh\" (UniqueName: \"kubernetes.io/projected/3ed4d963-93ff-4553-a493-c8426d121216-kube-api-access-b54gh\") pod \"payload-pre-processing-e2e-ait-d969013e-85c7895599-g4z9h\" (UID: \"3ed4d963-93ff-4553-a493-c8426d121216\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-d969013e-85c7895599-g4z9h" Nov 03 13:51:26.290568 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:26.290544 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d969013e-575cbbfb6-6wf8s" Nov 03 13:51:26.385156 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:26.385121 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ed4d963-93ff-4553-a493-c8426d121216-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-d969013e-85c7895599-g4z9h\" (UID: \"3ed4d963-93ff-4553-a493-c8426d121216\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-d969013e-85c7895599-g4z9h" Nov 03 13:51:26.385330 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:26.385167 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rbwwx\" (UniqueName: \"kubernetes.io/projected/951922e7-2b2b-496a-9edb-31af5071228b-kube-api-access-rbwwx\") pod \"payload-processing-e2e-ait-d969013e-69795cd9c-fpcj9\" (UID: \"951922e7-2b2b-496a-9edb-31af5071228b\") " pod="openshift-ingress/payload-processing-e2e-ait-d969013e-69795cd9c-fpcj9" Nov 03 13:51:26.385330 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:26.385316 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/951922e7-2b2b-496a-9edb-31af5071228b-plugins-config-volume\") pod \"payload-processing-e2e-ait-d969013e-69795cd9c-fpcj9\" (UID: \"951922e7-2b2b-496a-9edb-31af5071228b\") " pod="openshift-ingress/payload-processing-e2e-ait-d969013e-69795cd9c-fpcj9" Nov 03 13:51:26.385475 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:26.385446 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b54gh\" (UniqueName: \"kubernetes.io/projected/3ed4d963-93ff-4553-a493-c8426d121216-kube-api-access-b54gh\") pod \"payload-pre-processing-e2e-ait-d969013e-85c7895599-g4z9h\" (UID: \"3ed4d963-93ff-4553-a493-c8426d121216\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-d969013e-85c7895599-g4z9h" Nov 03 13:51:26.386251 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:26.386228 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/951922e7-2b2b-496a-9edb-31af5071228b-plugins-config-volume\") pod \"payload-processing-e2e-ait-d969013e-69795cd9c-fpcj9\" (UID: \"951922e7-2b2b-496a-9edb-31af5071228b\") " pod="openshift-ingress/payload-processing-e2e-ait-d969013e-69795cd9c-fpcj9" Nov 03 13:51:26.386835 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:26.386818 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ed4d963-93ff-4553-a493-c8426d121216-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-d969013e-85c7895599-g4z9h\" (UID: \"3ed4d963-93ff-4553-a493-c8426d121216\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-d969013e-85c7895599-g4z9h" Nov 03 13:51:26.394328 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:26.394302 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rbwwx\" (UniqueName: \"kubernetes.io/projected/951922e7-2b2b-496a-9edb-31af5071228b-kube-api-access-rbwwx\") pod \"payload-processing-e2e-ait-d969013e-69795cd9c-fpcj9\" (UID: \"951922e7-2b2b-496a-9edb-31af5071228b\") " pod="openshift-ingress/payload-processing-e2e-ait-d969013e-69795cd9c-fpcj9" Nov 03 13:51:26.395671 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:26.395641 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b54gh\" (UniqueName: \"kubernetes.io/projected/3ed4d963-93ff-4553-a493-c8426d121216-kube-api-access-b54gh\") pod \"payload-pre-processing-e2e-ait-d969013e-85c7895599-g4z9h\" (UID: \"3ed4d963-93ff-4553-a493-c8426d121216\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-d969013e-85c7895599-g4z9h" Nov 03 13:51:26.452151 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:26.452112 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-d969013e-85c7895599-g4z9h" Nov 03 13:51:26.489719 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:26.489682 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-d969013e-69795cd9c-fpcj9" Nov 03 13:51:26.578557 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:26.578518 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d969013e-575cbbfb6-6wf8s"] Nov 03 13:51:26.580936 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:51:26.580904 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9338282_ec5b_4805_9d2d_f8be31525f8a.slice/crio-5342f866598a6d3d5808583eb7d2baf72b52ea1c67e83e575013600fc0fc572c WatchSource:0}: Error finding container 5342f866598a6d3d5808583eb7d2baf72b52ea1c67e83e575013600fc0fc572c: Status 404 returned error can't find the container with id 5342f866598a6d3d5808583eb7d2baf72b52ea1c67e83e575013600fc0fc572c Nov 03 13:51:26.582967 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:26.582947 5024 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 13:51:26.732147 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:26.732106 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-d969013e-85c7895599-g4z9h"] Nov 03 13:51:26.735428 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:51:26.735393 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ed4d963_93ff_4553_a493_c8426d121216.slice/crio-46cc23b17641a3dc98200faf3ec4184e104613ad0d4213468fcaee44249b0c61 WatchSource:0}: Error finding container 46cc23b17641a3dc98200faf3ec4184e104613ad0d4213468fcaee44249b0c61: Status 404 returned error can't find the container with id 46cc23b17641a3dc98200faf3ec4184e104613ad0d4213468fcaee44249b0c61 Nov 03 13:51:26.781551 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:26.781518 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-d969013e-69795cd9c-fpcj9"] Nov 03 13:51:26.787796 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:51:26.787754 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod951922e7_2b2b_496a_9edb_31af5071228b.slice/crio-f1a912a843f01b40e4c54032e46e83a6cb2902e9dc3103be7299af52c2a4693d WatchSource:0}: Error finding container f1a912a843f01b40e4c54032e46e83a6cb2902e9dc3103be7299af52c2a4693d: Status 404 returned error can't find the container with id f1a912a843f01b40e4c54032e46e83a6cb2902e9dc3103be7299af52c2a4693d Nov 03 13:51:27.316728 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:27.316685 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d969013e-575cbbfb6-6wf8s" event={"ID":"f9338282-ec5b-4805-9d2d-f8be31525f8a","Type":"ContainerStarted","Data":"d7bfdf84d52dfa814c670a9af1e89a635b31528098f9638f198aa8742bf60035"} Nov 03 13:51:27.317192 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:27.316755 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d969013e-575cbbfb6-6wf8s" event={"ID":"f9338282-ec5b-4805-9d2d-f8be31525f8a","Type":"ContainerStarted","Data":"5342f866598a6d3d5808583eb7d2baf72b52ea1c67e83e575013600fc0fc572c"} Nov 03 13:51:27.317192 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:27.316803 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d969013e-575cbbfb6-6wf8s" Nov 03 13:51:27.318362 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:27.318331 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-d969013e-69795cd9c-fpcj9" event={"ID":"951922e7-2b2b-496a-9edb-31af5071228b","Type":"ContainerStarted","Data":"2902a6e485e289cf1eeda922cf4f4c2067d3a2bcbfd748040ae382c586f488cf"} Nov 03 13:51:27.318515 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:27.318385 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-d969013e-69795cd9c-fpcj9" event={"ID":"951922e7-2b2b-496a-9edb-31af5071228b","Type":"ContainerStarted","Data":"f1a912a843f01b40e4c54032e46e83a6cb2902e9dc3103be7299af52c2a4693d"} Nov 03 13:51:27.318515 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:27.318413 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-d969013e-69795cd9c-fpcj9" Nov 03 13:51:27.320108 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:27.320085 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-d969013e-85c7895599-g4z9h" event={"ID":"3ed4d963-93ff-4553-a493-c8426d121216","Type":"ContainerStarted","Data":"3f15c22d5a8d9d4a11c57933124d9a57ae0340f866f55fd8ddbf3459a757d780"} Nov 03 13:51:27.320256 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:27.320124 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-d969013e-85c7895599-g4z9h" event={"ID":"3ed4d963-93ff-4553-a493-c8426d121216","Type":"ContainerStarted","Data":"46cc23b17641a3dc98200faf3ec4184e104613ad0d4213468fcaee44249b0c61"} Nov 03 13:51:27.320256 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:27.320177 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-d969013e-85c7895599-g4z9h" Nov 03 13:51:27.356108 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:27.356048 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d969013e-575cbbfb6-6wf8s" podStartSLOduration=1.879723394 podStartE2EDuration="2.356036407s" totalImagesPullingTime="476.313013ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:51:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:51:27.353215303 +0000 UTC m=+2567.361406579" watchObservedRunningTime="2026-11-03 13:51:27.356036407 +0000 UTC m=+2567.364227684" Nov 03 13:51:27.380084 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:27.380019 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-d969013e-69795cd9c-fpcj9" podStartSLOduration=1.380000732 podStartE2EDuration="1.380000732s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:51:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:51:27.375927587 +0000 UTC m=+2567.384118865" watchObservedRunningTime="2026-11-03 13:51:27.380000732 +0000 UTC m=+2567.388192007" Nov 03 13:51:27.399451 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:27.399398 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-d969013e-85c7895599-g4z9h" podStartSLOduration=1.399383849 podStartE2EDuration="1.399383849s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:51:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:51:27.39655732 +0000 UTC m=+2567.404748597" watchObservedRunningTime="2026-11-03 13:51:27.399383849 +0000 UTC m=+2567.407575126" Nov 03 13:51:33.331118 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:33.331085 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d969013e-575cbbfb6-6wf8s" Nov 03 13:51:36.023335 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:36.023279 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-sv6fn"] Nov 03 13:51:36.025796 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:36.025768 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-sv6fn"] Nov 03 13:51:36.108495 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:36.108450 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a71e9bee-8090-49bd-ad64-30fd30aabe7a" path="/var/lib/kubelet/pods/a71e9bee-8090-49bd-ad64-30fd30aabe7a/volumes" Nov 03 13:51:38.329105 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:38.329069 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-d969013e-85c7895599-g4z9h" Nov 03 13:51:38.331925 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:38.329271 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-d969013e-69795cd9c-fpcj9" Nov 03 13:51:38.386391 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:38.386355 5024 generic.go:356] "Generic (PLEG): container finished" podID="11fc0ab3-5590-4613-8f68-cf4c75791d3a" containerID="4feb3e110f9f7aa45cacffefb63d1feb4143a18cb54e83cb1ad195d5a4ccbbb8" exitCode=0 Nov 03 13:51:38.386391 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:38.386380 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d969013e-openshift-default-6ff77f8c6c-cl26n" event={"ID":"11fc0ab3-5590-4613-8f68-cf4c75791d3a","Type":"ContainerDied","Data":"4feb3e110f9f7aa45cacffefb63d1feb4143a18cb54e83cb1ad195d5a4ccbbb8"} Nov 03 13:51:38.386880 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:38.386856 5024 scope.go:122] "RemoveContainer" containerID="4feb3e110f9f7aa45cacffefb63d1feb4143a18cb54e83cb1ad195d5a4ccbbb8" Nov 03 13:51:38.388386 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:38.388355 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:51:38.388514 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:38.388424 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:51:38.388514 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:38.388461 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:51:39.299996 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:39.299947 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-d969013e-openshift-default-6ff77f8c6c-cl26n" Nov 03 13:51:39.404941 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:39.404899 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d969013e-openshift-default-6ff77f8c6c-cl26n" event={"ID":"11fc0ab3-5590-4613-8f68-cf4c75791d3a","Type":"ContainerStarted","Data":"8c805f759fa0d59911d836642fde047320749103af3871e052f9b72082306c7a"} Nov 03 13:51:39.457078 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:39.457044 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-d969013e-openshift-default-6ff77f8c6c-cl26n" Nov 03 13:51:39.457905 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:39.457883 5024 patch_prober.go:29] interesting pod/e2e-ait-d969013e-openshift-default-6ff77f8c6c-cl26n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" start-of-body= Nov 03 13:51:39.457991 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:39.457929 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-d969013e-openshift-default-6ff77f8c6c-cl26n" podUID="11fc0ab3-5590-4613-8f68-cf4c75791d3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" Nov 03 13:51:39.523004 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:51:39.522967 5024 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d7mwz" containerName="extractor" Nov 03 13:51:40.457629 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:40.457597 5024 patch_prober.go:29] interesting pod/e2e-ait-d969013e-openshift-default-6ff77f8c6c-cl26n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" start-of-body= Nov 03 13:51:40.458022 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:40.457654 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-d969013e-openshift-default-6ff77f8c6c-cl26n" podUID="11fc0ab3-5590-4613-8f68-cf4c75791d3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" Nov 03 13:51:41.456865 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:41.456830 5024 patch_prober.go:29] interesting pod/e2e-ait-d969013e-openshift-default-6ff77f8c6c-cl26n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" start-of-body= Nov 03 13:51:41.457051 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:41.456882 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-d969013e-openshift-default-6ff77f8c6c-cl26n" podUID="11fc0ab3-5590-4613-8f68-cf4c75791d3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" Nov 03 13:51:42.457143 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:42.457103 5024 patch_prober.go:29] interesting pod/e2e-ait-d969013e-openshift-default-6ff77f8c6c-cl26n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" start-of-body= Nov 03 13:51:42.457558 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:42.457169 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-d969013e-openshift-default-6ff77f8c6c-cl26n" podUID="11fc0ab3-5590-4613-8f68-cf4c75791d3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" Nov 03 13:51:43.457180 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:43.457147 5024 patch_prober.go:29] interesting pod/e2e-ait-d969013e-openshift-default-6ff77f8c6c-cl26n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" start-of-body= Nov 03 13:51:43.457608 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:43.457202 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-d969013e-openshift-default-6ff77f8c6c-cl26n" podUID="11fc0ab3-5590-4613-8f68-cf4c75791d3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" Nov 03 13:51:44.460867 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:44.460831 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-d969013e-openshift-default-6ff77f8c6c-cl26n" Nov 03 13:51:44.461376 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:44.461224 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-d969013e-openshift-default-6ff77f8c6c-cl26n" Nov 03 13:51:44.461837 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:44.461817 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-d969013e-openshift-default-6ff77f8c6c-cl26n" Nov 03 13:51:45.674996 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:45.674958 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-d969013e-69795cd9c-fpcj9"] Nov 03 13:51:45.689851 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:45.689817 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-d969013e-85c7895599-g4z9h"] Nov 03 13:51:46.023396 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:46.023297 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d969013e-575cbbfb6-6wf8s"] Nov 03 13:51:46.023586 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:46.023557 5024 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d969013e-575cbbfb6-6wf8s" podUID="f9338282-ec5b-4805-9d2d-f8be31525f8a" containerName="maas-api" containerID="cri-o://d7bfdf84d52dfa814c670a9af1e89a635b31528098f9638f198aa8742bf60035" gracePeriod=30 Nov 03 13:51:46.326280 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:46.326252 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d969013e-575cbbfb6-6wf8s" Nov 03 13:51:46.432894 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:46.432857 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f9338282-ec5b-4805-9d2d-f8be31525f8a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9338282-ec5b-4805-9d2d-f8be31525f8a-metrics-certs\") pod \"f9338282-ec5b-4805-9d2d-f8be31525f8a\" (UID: \"f9338282-ec5b-4805-9d2d-f8be31525f8a\") " Nov 03 13:51:46.433071 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:46.432998 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f9338282-ec5b-4805-9d2d-f8be31525f8a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f9338282-ec5b-4805-9d2d-f8be31525f8a-maas-api-tls\") pod \"f9338282-ec5b-4805-9d2d-f8be31525f8a\" (UID: \"f9338282-ec5b-4805-9d2d-f8be31525f8a\") " Nov 03 13:51:46.433071 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:46.433046 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9338282-ec5b-4805-9d2d-f8be31525f8a-kube-api-access-k98rq\" (UniqueName: \"kubernetes.io/projected/f9338282-ec5b-4805-9d2d-f8be31525f8a-kube-api-access-k98rq\") pod \"f9338282-ec5b-4805-9d2d-f8be31525f8a\" (UID: \"f9338282-ec5b-4805-9d2d-f8be31525f8a\") " Nov 03 13:51:46.433180 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:46.433078 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f9338282-ec5b-4805-9d2d-f8be31525f8a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9338282-ec5b-4805-9d2d-f8be31525f8a-openshift-service-ca\") pod \"f9338282-ec5b-4805-9d2d-f8be31525f8a\" (UID: \"f9338282-ec5b-4805-9d2d-f8be31525f8a\") " Nov 03 13:51:46.433569 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:46.433536 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9338282-ec5b-4805-9d2d-f8be31525f8a-openshift-service-ca" pod "f9338282-ec5b-4805-9d2d-f8be31525f8a" (UID: "f9338282-ec5b-4805-9d2d-f8be31525f8a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:51:46.435082 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:46.435054 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f9338282-ec5b-4805-9d2d-f8be31525f8a-maas-api-tls" pod "f9338282-ec5b-4805-9d2d-f8be31525f8a" (UID: "f9338282-ec5b-4805-9d2d-f8be31525f8a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 13:51:46.435530 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:46.435507 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f9338282-ec5b-4805-9d2d-f8be31525f8a-metrics-certs" pod "f9338282-ec5b-4805-9d2d-f8be31525f8a" (UID: "f9338282-ec5b-4805-9d2d-f8be31525f8a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 13:51:46.435530 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:46.435518 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9338282-ec5b-4805-9d2d-f8be31525f8a-kube-api-access-k98rq" pod "f9338282-ec5b-4805-9d2d-f8be31525f8a" (UID: "f9338282-ec5b-4805-9d2d-f8be31525f8a"). InnerVolumeSpecName "kube-api-access-k98rq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:51:46.453051 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:46.453015 5024 generic.go:356] "Generic (PLEG): container finished" podID="f9338282-ec5b-4805-9d2d-f8be31525f8a" containerID="d7bfdf84d52dfa814c670a9af1e89a635b31528098f9638f198aa8742bf60035" exitCode=0 Nov 03 13:51:46.453949 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:46.453877 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d969013e-575cbbfb6-6wf8s" Nov 03 13:51:46.453949 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:46.453911 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d969013e-575cbbfb6-6wf8s" event={"ID":"f9338282-ec5b-4805-9d2d-f8be31525f8a","Type":"ContainerDied","Data":"d7bfdf84d52dfa814c670a9af1e89a635b31528098f9638f198aa8742bf60035"} Nov 03 13:51:46.454270 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:46.453966 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d969013e-575cbbfb6-6wf8s" event={"ID":"f9338282-ec5b-4805-9d2d-f8be31525f8a","Type":"ContainerDied","Data":"5342f866598a6d3d5808583eb7d2baf72b52ea1c67e83e575013600fc0fc572c"} Nov 03 13:51:46.454270 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:46.453998 5024 scope.go:122] "RemoveContainer" containerID="d7bfdf84d52dfa814c670a9af1e89a635b31528098f9638f198aa8742bf60035" Nov 03 13:51:46.465052 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:46.465033 5024 scope.go:122] "RemoveContainer" containerID="d7bfdf84d52dfa814c670a9af1e89a635b31528098f9638f198aa8742bf60035" Nov 03 13:51:46.465355 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:51:46.465319 5024 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d7bfdf84d52dfa814c670a9af1e89a635b31528098f9638f198aa8742bf60035\": container with ID starting with d7bfdf84d52dfa814c670a9af1e89a635b31528098f9638f198aa8742bf60035 not found: ID does not exist" containerID="d7bfdf84d52dfa814c670a9af1e89a635b31528098f9638f198aa8742bf60035" Nov 03 13:51:46.465449 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:46.465363 5024 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7bfdf84d52dfa814c670a9af1e89a635b31528098f9638f198aa8742bf60035"} err="failed to get container status \"d7bfdf84d52dfa814c670a9af1e89a635b31528098f9638f198aa8742bf60035\": rpc error: code = NotFound desc = could not find container \"d7bfdf84d52dfa814c670a9af1e89a635b31528098f9638f198aa8742bf60035\": container with ID starting with d7bfdf84d52dfa814c670a9af1e89a635b31528098f9638f198aa8742bf60035 not found: ID does not exist" Nov 03 13:51:46.480650 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:46.480621 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d969013e-575cbbfb6-6wf8s"] Nov 03 13:51:46.487795 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:46.487773 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d969013e-575cbbfb6-6wf8s"] Nov 03 13:51:46.534879 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:46.534841 5024 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f9338282-ec5b-4805-9d2d-f8be31525f8a-maas-api-tls\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:51:46.534879 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:46.534877 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k98rq\" (UniqueName: \"kubernetes.io/projected/f9338282-ec5b-4805-9d2d-f8be31525f8a-kube-api-access-k98rq\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:51:46.535089 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:46.534894 5024 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9338282-ec5b-4805-9d2d-f8be31525f8a-openshift-service-ca\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:51:46.535089 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:46.534953 5024 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9338282-ec5b-4805-9d2d-f8be31525f8a-metrics-certs\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:51:48.109226 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:48.109187 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f9338282-ec5b-4805-9d2d-f8be31525f8a" path="/var/lib/kubelet/pods/f9338282-ec5b-4805-9d2d-f8be31525f8a/volumes" Nov 03 13:51:50.676134 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:50.676065 5024 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-d969013e-69795cd9c-fpcj9" podUID="951922e7-2b2b-496a-9edb-31af5071228b" containerName="payload-processing" containerID="cri-o://2902a6e485e289cf1eeda922cf4f4c2067d3a2bcbfd748040ae382c586f488cf" gracePeriod=25 Nov 03 13:51:50.690310 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:50.690260 5024 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-d969013e-85c7895599-g4z9h" podUID="3ed4d963-93ff-4553-a493-c8426d121216" containerName="payload-pre-processing" containerID="cri-o://3f15c22d5a8d9d4a11c57933124d9a57ae0340f866f55fd8ddbf3459a757d780" gracePeriod=25 Nov 03 13:51:50.945332 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:50.945301 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-d969013e-69795cd9c-fpcj9" Nov 03 13:51:50.982831 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:50.982807 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-d969013e-85c7895599-g4z9h" Nov 03 13:51:51.084138 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:51.084095 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/951922e7-2b2b-496a-9edb-31af5071228b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/951922e7-2b2b-496a-9edb-31af5071228b-plugins-config-volume\") pod \"951922e7-2b2b-496a-9edb-31af5071228b\" (UID: \"951922e7-2b2b-496a-9edb-31af5071228b\") " Nov 03 13:51:51.084353 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:51.084155 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/951922e7-2b2b-496a-9edb-31af5071228b-kube-api-access-rbwwx\" (UniqueName: \"kubernetes.io/projected/951922e7-2b2b-496a-9edb-31af5071228b-kube-api-access-rbwwx\") pod \"951922e7-2b2b-496a-9edb-31af5071228b\" (UID: \"951922e7-2b2b-496a-9edb-31af5071228b\") " Nov 03 13:51:51.084353 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:51.084255 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ed4d963-93ff-4553-a493-c8426d121216-kube-api-access-b54gh\" (UniqueName: \"kubernetes.io/projected/3ed4d963-93ff-4553-a493-c8426d121216-kube-api-access-b54gh\") pod \"3ed4d963-93ff-4553-a493-c8426d121216\" (UID: \"3ed4d963-93ff-4553-a493-c8426d121216\") " Nov 03 13:51:51.084353 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:51.084312 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3ed4d963-93ff-4553-a493-c8426d121216-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ed4d963-93ff-4553-a493-c8426d121216-plugins-config-volume\") pod \"3ed4d963-93ff-4553-a493-c8426d121216\" (UID: \"3ed4d963-93ff-4553-a493-c8426d121216\") " Nov 03 13:51:51.084560 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:51.084501 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/951922e7-2b2b-496a-9edb-31af5071228b-plugins-config-volume" pod "951922e7-2b2b-496a-9edb-31af5071228b" (UID: "951922e7-2b2b-496a-9edb-31af5071228b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:51:51.084699 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:51.084670 5024 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/951922e7-2b2b-496a-9edb-31af5071228b-plugins-config-volume\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:51:51.084825 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:51.084731 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ed4d963-93ff-4553-a493-c8426d121216-plugins-config-volume" pod "3ed4d963-93ff-4553-a493-c8426d121216" (UID: "3ed4d963-93ff-4553-a493-c8426d121216"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:51:51.086348 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:51.086324 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/951922e7-2b2b-496a-9edb-31af5071228b-kube-api-access-rbwwx" pod "951922e7-2b2b-496a-9edb-31af5071228b" (UID: "951922e7-2b2b-496a-9edb-31af5071228b"). InnerVolumeSpecName "kube-api-access-rbwwx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:51:51.086420 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:51.086347 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ed4d963-93ff-4553-a493-c8426d121216-kube-api-access-b54gh" pod "3ed4d963-93ff-4553-a493-c8426d121216" (UID: "3ed4d963-93ff-4553-a493-c8426d121216"). InnerVolumeSpecName "kube-api-access-b54gh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:51:51.185925 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:51.185846 5024 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ed4d963-93ff-4553-a493-c8426d121216-plugins-config-volume\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:51:51.185925 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:51.185876 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rbwwx\" (UniqueName: \"kubernetes.io/projected/951922e7-2b2b-496a-9edb-31af5071228b-kube-api-access-rbwwx\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:51:51.185925 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:51.185887 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b54gh\" (UniqueName: \"kubernetes.io/projected/3ed4d963-93ff-4553-a493-c8426d121216-kube-api-access-b54gh\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:51:51.483698 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:51.483608 5024 generic.go:356] "Generic (PLEG): container finished" podID="951922e7-2b2b-496a-9edb-31af5071228b" containerID="2902a6e485e289cf1eeda922cf4f4c2067d3a2bcbfd748040ae382c586f488cf" exitCode=0 Nov 03 13:51:51.483698 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:51.483668 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-d969013e-69795cd9c-fpcj9" Nov 03 13:51:51.483942 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:51.483698 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-d969013e-69795cd9c-fpcj9" event={"ID":"951922e7-2b2b-496a-9edb-31af5071228b","Type":"ContainerDied","Data":"2902a6e485e289cf1eeda922cf4f4c2067d3a2bcbfd748040ae382c586f488cf"} Nov 03 13:51:51.483942 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:51.483751 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-d969013e-69795cd9c-fpcj9" event={"ID":"951922e7-2b2b-496a-9edb-31af5071228b","Type":"ContainerDied","Data":"f1a912a843f01b40e4c54032e46e83a6cb2902e9dc3103be7299af52c2a4693d"} Nov 03 13:51:51.483942 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:51.483774 5024 scope.go:122] "RemoveContainer" containerID="2902a6e485e289cf1eeda922cf4f4c2067d3a2bcbfd748040ae382c586f488cf" Nov 03 13:51:51.485058 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:51.485037 5024 generic.go:356] "Generic (PLEG): container finished" podID="3ed4d963-93ff-4553-a493-c8426d121216" containerID="3f15c22d5a8d9d4a11c57933124d9a57ae0340f866f55fd8ddbf3459a757d780" exitCode=0 Nov 03 13:51:51.485164 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:51.485108 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-d969013e-85c7895599-g4z9h" Nov 03 13:51:51.485164 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:51.485131 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-d969013e-85c7895599-g4z9h" event={"ID":"3ed4d963-93ff-4553-a493-c8426d121216","Type":"ContainerDied","Data":"3f15c22d5a8d9d4a11c57933124d9a57ae0340f866f55fd8ddbf3459a757d780"} Nov 03 13:51:51.485266 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:51.485182 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-d969013e-85c7895599-g4z9h" event={"ID":"3ed4d963-93ff-4553-a493-c8426d121216","Type":"ContainerDied","Data":"46cc23b17641a3dc98200faf3ec4184e104613ad0d4213468fcaee44249b0c61"} Nov 03 13:51:51.494509 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:51.494493 5024 scope.go:122] "RemoveContainer" containerID="2902a6e485e289cf1eeda922cf4f4c2067d3a2bcbfd748040ae382c586f488cf" Nov 03 13:51:51.494872 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:51:51.494849 5024 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2902a6e485e289cf1eeda922cf4f4c2067d3a2bcbfd748040ae382c586f488cf\": container with ID starting with 2902a6e485e289cf1eeda922cf4f4c2067d3a2bcbfd748040ae382c586f488cf not found: ID does not exist" containerID="2902a6e485e289cf1eeda922cf4f4c2067d3a2bcbfd748040ae382c586f488cf" Nov 03 13:51:51.494971 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:51.494877 5024 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2902a6e485e289cf1eeda922cf4f4c2067d3a2bcbfd748040ae382c586f488cf"} err="failed to get container status \"2902a6e485e289cf1eeda922cf4f4c2067d3a2bcbfd748040ae382c586f488cf\": rpc error: code = NotFound desc = could not find container \"2902a6e485e289cf1eeda922cf4f4c2067d3a2bcbfd748040ae382c586f488cf\": container with ID starting with 2902a6e485e289cf1eeda922cf4f4c2067d3a2bcbfd748040ae382c586f488cf not found: ID does not exist" Nov 03 13:51:51.494971 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:51.494897 5024 scope.go:122] "RemoveContainer" containerID="3f15c22d5a8d9d4a11c57933124d9a57ae0340f866f55fd8ddbf3459a757d780" Nov 03 13:51:51.503082 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:51.503067 5024 scope.go:122] "RemoveContainer" containerID="3f15c22d5a8d9d4a11c57933124d9a57ae0340f866f55fd8ddbf3459a757d780" Nov 03 13:51:51.503345 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:51:51.503327 5024 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f15c22d5a8d9d4a11c57933124d9a57ae0340f866f55fd8ddbf3459a757d780\": container with ID starting with 3f15c22d5a8d9d4a11c57933124d9a57ae0340f866f55fd8ddbf3459a757d780 not found: ID does not exist" containerID="3f15c22d5a8d9d4a11c57933124d9a57ae0340f866f55fd8ddbf3459a757d780" Nov 03 13:51:51.503398 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:51.503351 5024 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f15c22d5a8d9d4a11c57933124d9a57ae0340f866f55fd8ddbf3459a757d780"} err="failed to get container status \"3f15c22d5a8d9d4a11c57933124d9a57ae0340f866f55fd8ddbf3459a757d780\": rpc error: code = NotFound desc = could not find container \"3f15c22d5a8d9d4a11c57933124d9a57ae0340f866f55fd8ddbf3459a757d780\": container with ID starting with 3f15c22d5a8d9d4a11c57933124d9a57ae0340f866f55fd8ddbf3459a757d780 not found: ID does not exist" Nov 03 13:51:51.510040 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:51.510020 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-d969013e-69795cd9c-fpcj9"] Nov 03 13:51:51.515107 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:51.515085 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-d969013e-69795cd9c-fpcj9"] Nov 03 13:51:51.528794 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:51.528770 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-d969013e-85c7895599-g4z9h"] Nov 03 13:51:51.533325 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:51.533306 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-d969013e-85c7895599-g4z9h"] Nov 03 13:51:51.832278 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:51.832187 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-d969013e-openshift-default-6ff77f8c6c-cl26n"] Nov 03 13:51:51.832797 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:51.832455 5024 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-d969013e-openshift-default-6ff77f8c6c-cl26n" podUID="11fc0ab3-5590-4613-8f68-cf4c75791d3a" containerName="istio-proxy" containerID="cri-o://8c805f759fa0d59911d836642fde047320749103af3871e052f9b72082306c7a" gracePeriod=30 Nov 03 13:51:52.106956 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:52.106923 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3ed4d963-93ff-4553-a493-c8426d121216" path="/var/lib/kubelet/pods/3ed4d963-93ff-4553-a493-c8426d121216/volumes" Nov 03 13:51:52.107328 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:52.107310 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="951922e7-2b2b-496a-9edb-31af5071228b" path="/var/lib/kubelet/pods/951922e7-2b2b-496a-9edb-31af5071228b/volumes" Nov 03 13:51:56.286308 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:56.285442 5024 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-92vh9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.173:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 13:51:56.286308 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:56.285507 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-92vh9" podUID="825941b6-3ed3-47f2-a859-c891c4f29434" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.173:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 13:51:56.311312 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:56.310396 5024 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-zndxf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.172:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 13:51:56.311312 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:56.310476 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-zndxf" podUID="73f8f24f-b955-433b-95ed-5f441f7fceb2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.172:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 13:51:56.343427 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:56.343386 5024 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-k4c4j container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.174:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 13:51:56.343715 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:56.343686 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-k4c4j" podUID="4b70e366-6fba-4bfa-ac3e-0f0c7b0b64d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.174:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 13:51:57.148256 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:57.147922 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-d969013e-openshift-default-6ff77f8c6c-cl26n" Nov 03 13:51:57.218844 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:57.218811 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11fc0ab3-5590-4613-8f68-cf4c75791d3a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11fc0ab3-5590-4613-8f68-cf4c75791d3a-workload-certs\") pod \"11fc0ab3-5590-4613-8f68-cf4c75791d3a\" (UID: \"11fc0ab3-5590-4613-8f68-cf4c75791d3a\") " Nov 03 13:51:57.219012 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:57.218873 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11fc0ab3-5590-4613-8f68-cf4c75791d3a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11fc0ab3-5590-4613-8f68-cf4c75791d3a-workload-socket\") pod \"11fc0ab3-5590-4613-8f68-cf4c75791d3a\" (UID: \"11fc0ab3-5590-4613-8f68-cf4c75791d3a\") " Nov 03 13:51:57.219012 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:57.218894 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11fc0ab3-5590-4613-8f68-cf4c75791d3a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11fc0ab3-5590-4613-8f68-cf4c75791d3a-istio-data\") pod \"11fc0ab3-5590-4613-8f68-cf4c75791d3a\" (UID: \"11fc0ab3-5590-4613-8f68-cf4c75791d3a\") " Nov 03 13:51:57.219012 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:57.218932 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11fc0ab3-5590-4613-8f68-cf4c75791d3a-kube-api-access-xn625\" (UniqueName: \"kubernetes.io/projected/11fc0ab3-5590-4613-8f68-cf4c75791d3a-kube-api-access-xn625\") pod \"11fc0ab3-5590-4613-8f68-cf4c75791d3a\" (UID: \"11fc0ab3-5590-4613-8f68-cf4c75791d3a\") " Nov 03 13:51:57.219012 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:57.218995 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/11fc0ab3-5590-4613-8f68-cf4c75791d3a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11fc0ab3-5590-4613-8f68-cf4c75791d3a-istio-podinfo\") pod \"11fc0ab3-5590-4613-8f68-cf4c75791d3a\" (UID: \"11fc0ab3-5590-4613-8f68-cf4c75791d3a\") " Nov 03 13:51:57.219242 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:57.219027 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11fc0ab3-5590-4613-8f68-cf4c75791d3a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11fc0ab3-5590-4613-8f68-cf4c75791d3a-credential-socket\") pod \"11fc0ab3-5590-4613-8f68-cf4c75791d3a\" (UID: \"11fc0ab3-5590-4613-8f68-cf4c75791d3a\") " Nov 03 13:51:57.219242 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:57.219056 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11fc0ab3-5590-4613-8f68-cf4c75791d3a-istio-token\" (UniqueName: \"kubernetes.io/projected/11fc0ab3-5590-4613-8f68-cf4c75791d3a-istio-token\") pod \"11fc0ab3-5590-4613-8f68-cf4c75791d3a\" (UID: \"11fc0ab3-5590-4613-8f68-cf4c75791d3a\") " Nov 03 13:51:57.219242 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:57.219113 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11fc0ab3-5590-4613-8f68-cf4c75791d3a-workload-certs" pod "11fc0ab3-5590-4613-8f68-cf4c75791d3a" (UID: "11fc0ab3-5590-4613-8f68-cf4c75791d3a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:51:57.219242 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:57.219154 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11fc0ab3-5590-4613-8f68-cf4c75791d3a-workload-socket" pod "11fc0ab3-5590-4613-8f68-cf4c75791d3a" (UID: "11fc0ab3-5590-4613-8f68-cf4c75791d3a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:51:57.219242 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:57.219203 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11fc0ab3-5590-4613-8f68-cf4c75791d3a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11fc0ab3-5590-4613-8f68-cf4c75791d3a-istio-envoy\") pod \"11fc0ab3-5590-4613-8f68-cf4c75791d3a\" (UID: \"11fc0ab3-5590-4613-8f68-cf4c75791d3a\") " Nov 03 13:51:57.219242 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:57.219233 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11fc0ab3-5590-4613-8f68-cf4c75791d3a-istio-data" pod "11fc0ab3-5590-4613-8f68-cf4c75791d3a" (UID: "11fc0ab3-5590-4613-8f68-cf4c75791d3a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:51:57.219610 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:57.219244 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/11fc0ab3-5590-4613-8f68-cf4c75791d3a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11fc0ab3-5590-4613-8f68-cf4c75791d3a-istiod-ca-cert\") pod \"11fc0ab3-5590-4613-8f68-cf4c75791d3a\" (UID: \"11fc0ab3-5590-4613-8f68-cf4c75791d3a\") " Nov 03 13:51:57.219610 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:57.219364 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11fc0ab3-5590-4613-8f68-cf4c75791d3a-credential-socket" pod "11fc0ab3-5590-4613-8f68-cf4c75791d3a" (UID: "11fc0ab3-5590-4613-8f68-cf4c75791d3a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:51:57.219610 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:57.219583 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11fc0ab3-5590-4613-8f68-cf4c75791d3a-istiod-ca-cert" pod "11fc0ab3-5590-4613-8f68-cf4c75791d3a" (UID: "11fc0ab3-5590-4613-8f68-cf4c75791d3a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:51:57.219784 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:57.219717 5024 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11fc0ab3-5590-4613-8f68-cf4c75791d3a-credential-socket\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:51:57.219784 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:57.219743 5024 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11fc0ab3-5590-4613-8f68-cf4c75791d3a-istiod-ca-cert\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:51:57.219784 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:57.219758 5024 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11fc0ab3-5590-4613-8f68-cf4c75791d3a-workload-certs\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:51:57.219784 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:57.219772 5024 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11fc0ab3-5590-4613-8f68-cf4c75791d3a-workload-socket\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:51:57.220023 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:57.219786 5024 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11fc0ab3-5590-4613-8f68-cf4c75791d3a-istio-data\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:51:57.221469 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:57.221434 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/11fc0ab3-5590-4613-8f68-cf4c75791d3a-istio-podinfo" pod "11fc0ab3-5590-4613-8f68-cf4c75791d3a" (UID: "11fc0ab3-5590-4613-8f68-cf4c75791d3a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 13:51:57.221616 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:57.221593 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11fc0ab3-5590-4613-8f68-cf4c75791d3a-istio-token" pod "11fc0ab3-5590-4613-8f68-cf4c75791d3a" (UID: "11fc0ab3-5590-4613-8f68-cf4c75791d3a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:51:57.221728 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:57.221709 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11fc0ab3-5590-4613-8f68-cf4c75791d3a-istio-envoy" pod "11fc0ab3-5590-4613-8f68-cf4c75791d3a" (UID: "11fc0ab3-5590-4613-8f68-cf4c75791d3a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:51:57.221775 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:57.221741 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11fc0ab3-5590-4613-8f68-cf4c75791d3a-kube-api-access-xn625" pod "11fc0ab3-5590-4613-8f68-cf4c75791d3a" (UID: "11fc0ab3-5590-4613-8f68-cf4c75791d3a"). InnerVolumeSpecName "kube-api-access-xn625". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:51:57.323952 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:57.323877 5024 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11fc0ab3-5590-4613-8f68-cf4c75791d3a-istio-token\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:51:57.323952 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:57.323931 5024 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11fc0ab3-5590-4613-8f68-cf4c75791d3a-istio-envoy\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:51:57.323952 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:57.323958 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xn625\" (UniqueName: \"kubernetes.io/projected/11fc0ab3-5590-4613-8f68-cf4c75791d3a-kube-api-access-xn625\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:51:57.324595 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:57.323974 5024 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11fc0ab3-5590-4613-8f68-cf4c75791d3a-istio-podinfo\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:51:57.566641 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:57.566555 5024 generic.go:356] "Generic (PLEG): container finished" podID="11fc0ab3-5590-4613-8f68-cf4c75791d3a" containerID="8c805f759fa0d59911d836642fde047320749103af3871e052f9b72082306c7a" exitCode=0 Nov 03 13:51:57.566641 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:57.566605 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-d969013e-openshift-default-6ff77f8c6c-cl26n" Nov 03 13:51:57.566843 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:57.566644 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d969013e-openshift-default-6ff77f8c6c-cl26n" event={"ID":"11fc0ab3-5590-4613-8f68-cf4c75791d3a","Type":"ContainerDied","Data":"8c805f759fa0d59911d836642fde047320749103af3871e052f9b72082306c7a"} Nov 03 13:51:57.566843 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:57.566706 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d969013e-openshift-default-6ff77f8c6c-cl26n" event={"ID":"11fc0ab3-5590-4613-8f68-cf4c75791d3a","Type":"ContainerDied","Data":"e56a0bd7600423294476803d7dd74c020d80a436a789a21bdbdb82f391c27450"} Nov 03 13:51:57.566843 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:57.566731 5024 scope.go:122] "RemoveContainer" containerID="8c805f759fa0d59911d836642fde047320749103af3871e052f9b72082306c7a" Nov 03 13:51:57.577682 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:57.577663 5024 scope.go:122] "RemoveContainer" containerID="4feb3e110f9f7aa45cacffefb63d1feb4143a18cb54e83cb1ad195d5a4ccbbb8" Nov 03 13:51:57.586270 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:57.586249 5024 scope.go:122] "RemoveContainer" containerID="8c805f759fa0d59911d836642fde047320749103af3871e052f9b72082306c7a" Nov 03 13:51:57.586593 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:51:57.586573 5024 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c805f759fa0d59911d836642fde047320749103af3871e052f9b72082306c7a\": container with ID starting with 8c805f759fa0d59911d836642fde047320749103af3871e052f9b72082306c7a not found: ID does not exist" containerID="8c805f759fa0d59911d836642fde047320749103af3871e052f9b72082306c7a" Nov 03 13:51:57.586658 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:57.586602 5024 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c805f759fa0d59911d836642fde047320749103af3871e052f9b72082306c7a"} err="failed to get container status \"8c805f759fa0d59911d836642fde047320749103af3871e052f9b72082306c7a\": rpc error: code = NotFound desc = could not find container \"8c805f759fa0d59911d836642fde047320749103af3871e052f9b72082306c7a\": container with ID starting with 8c805f759fa0d59911d836642fde047320749103af3871e052f9b72082306c7a not found: ID does not exist" Nov 03 13:51:57.586658 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:57.586619 5024 scope.go:122] "RemoveContainer" containerID="4feb3e110f9f7aa45cacffefb63d1feb4143a18cb54e83cb1ad195d5a4ccbbb8" Nov 03 13:51:57.586853 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:51:57.586837 5024 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4feb3e110f9f7aa45cacffefb63d1feb4143a18cb54e83cb1ad195d5a4ccbbb8\": container with ID starting with 4feb3e110f9f7aa45cacffefb63d1feb4143a18cb54e83cb1ad195d5a4ccbbb8 not found: ID does not exist" containerID="4feb3e110f9f7aa45cacffefb63d1feb4143a18cb54e83cb1ad195d5a4ccbbb8" Nov 03 13:51:57.586903 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:57.586859 5024 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4feb3e110f9f7aa45cacffefb63d1feb4143a18cb54e83cb1ad195d5a4ccbbb8"} err="failed to get container status \"4feb3e110f9f7aa45cacffefb63d1feb4143a18cb54e83cb1ad195d5a4ccbbb8\": rpc error: code = NotFound desc = could not find container \"4feb3e110f9f7aa45cacffefb63d1feb4143a18cb54e83cb1ad195d5a4ccbbb8\": container with ID starting with 4feb3e110f9f7aa45cacffefb63d1feb4143a18cb54e83cb1ad195d5a4ccbbb8 not found: ID does not exist" Nov 03 13:51:57.595561 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:57.595541 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-d969013e-openshift-default-6ff77f8c6c-cl26n"] Nov 03 13:51:57.601625 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:57.601606 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-d969013e-openshift-default-6ff77f8c6c-cl26n"] Nov 03 13:51:58.109939 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:58.109507 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="11fc0ab3-5590-4613-8f68-cf4c75791d3a" path="/var/lib/kubelet/pods/11fc0ab3-5590-4613-8f68-cf4c75791d3a/volumes" Nov 03 13:51:58.725814 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:58.725777 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-b5plm"] Nov 03 13:51:58.726205 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:58.726173 5024 cpu_manager.go:414] "containerMap: removing container" podUID="11fc0ab3-5590-4613-8f68-cf4c75791d3a" containerName="istio-proxy" Nov 03 13:51:58.726205 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:58.726184 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11fc0ab3-5590-4613-8f68-cf4c75791d3a" containerName="istio-proxy" Nov 03 13:51:58.726205 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:58.726203 5024 cpu_manager.go:414] "containerMap: removing container" podUID="3ed4d963-93ff-4553-a493-c8426d121216" containerName="payload-pre-processing" Nov 03 13:51:58.726351 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:58.726209 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ed4d963-93ff-4553-a493-c8426d121216" containerName="payload-pre-processing" Nov 03 13:51:58.726351 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:58.726218 5024 cpu_manager.go:414] "containerMap: removing container" podUID="951922e7-2b2b-496a-9edb-31af5071228b" containerName="payload-processing" Nov 03 13:51:58.726351 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:58.726223 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="951922e7-2b2b-496a-9edb-31af5071228b" containerName="payload-processing" Nov 03 13:51:58.726351 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:58.726240 5024 cpu_manager.go:414] "containerMap: removing container" podUID="f9338282-ec5b-4805-9d2d-f8be31525f8a" containerName="maas-api" Nov 03 13:51:58.726351 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:58.726245 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9338282-ec5b-4805-9d2d-f8be31525f8a" containerName="maas-api" Nov 03 13:51:58.726351 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:58.726333 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ed4d963-93ff-4553-a493-c8426d121216" containerName="payload-pre-processing" Nov 03 13:51:58.726351 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:58.726342 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="951922e7-2b2b-496a-9edb-31af5071228b" containerName="payload-processing" Nov 03 13:51:58.726351 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:58.726350 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9338282-ec5b-4805-9d2d-f8be31525f8a" containerName="maas-api" Nov 03 13:51:58.726623 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:58.726359 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="11fc0ab3-5590-4613-8f68-cf4c75791d3a" containerName="istio-proxy" Nov 03 13:51:58.726623 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:58.726365 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="11fc0ab3-5590-4613-8f68-cf4c75791d3a" containerName="istio-proxy" Nov 03 13:51:58.730989 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:58.730970 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-b5plm" Nov 03 13:51:58.737184 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:58.737109 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 13:51:58.737184 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:58.737107 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53e3e37b-99af-4a19-baa8-0a1d954a220d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f904bff9-b5plm\" (UID: \"53e3e37b-99af-4a19-baa8-0a1d954a220d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-b5plm" Nov 03 13:51:58.737425 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:58.737193 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c2zbl\" (UniqueName: \"kubernetes.io/projected/53e3e37b-99af-4a19-baa8-0a1d954a220d-kube-api-access-c2zbl\") pod \"maas-api-revoke-sub-f904bff9-b5plm\" (UID: \"53e3e37b-99af-4a19-baa8-0a1d954a220d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-b5plm" Nov 03 13:51:58.737425 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:58.737245 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 13:51:58.737425 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:58.737253 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53e3e37b-99af-4a19-baa8-0a1d954a220d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f904bff9-b5plm\" (UID: \"53e3e37b-99af-4a19-baa8-0a1d954a220d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-b5plm" Nov 03 13:51:58.737425 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:58.737339 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 13:51:58.737691 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:58.737668 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-b5plm"] Nov 03 13:51:58.738295 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:58.738265 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w6vqv\"" Nov 03 13:51:58.837780 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:58.837752 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53e3e37b-99af-4a19-baa8-0a1d954a220d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f904bff9-b5plm\" (UID: \"53e3e37b-99af-4a19-baa8-0a1d954a220d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-b5plm" Nov 03 13:51:58.837955 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:58.837795 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c2zbl\" (UniqueName: \"kubernetes.io/projected/53e3e37b-99af-4a19-baa8-0a1d954a220d-kube-api-access-c2zbl\") pod \"maas-api-revoke-sub-f904bff9-b5plm\" (UID: \"53e3e37b-99af-4a19-baa8-0a1d954a220d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-b5plm" Nov 03 13:51:58.837955 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:58.837835 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53e3e37b-99af-4a19-baa8-0a1d954a220d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f904bff9-b5plm\" (UID: \"53e3e37b-99af-4a19-baa8-0a1d954a220d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-b5plm" Nov 03 13:51:58.838544 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:58.838515 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53e3e37b-99af-4a19-baa8-0a1d954a220d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f904bff9-b5plm\" (UID: \"53e3e37b-99af-4a19-baa8-0a1d954a220d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-b5plm" Nov 03 13:51:58.840110 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:58.840088 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53e3e37b-99af-4a19-baa8-0a1d954a220d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f904bff9-b5plm\" (UID: \"53e3e37b-99af-4a19-baa8-0a1d954a220d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-b5plm" Nov 03 13:51:58.848596 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:58.848576 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c2zbl\" (UniqueName: \"kubernetes.io/projected/53e3e37b-99af-4a19-baa8-0a1d954a220d-kube-api-access-c2zbl\") pod \"maas-api-revoke-sub-f904bff9-b5plm\" (UID: \"53e3e37b-99af-4a19-baa8-0a1d954a220d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-b5plm" Nov 03 13:51:59.046586 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:59.046502 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-b5plm" Nov 03 13:51:59.319402 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:59.319374 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-b5plm"] Nov 03 13:51:59.320829 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:51:59.320800 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53e3e37b_99af_4a19_baa8_0a1d954a220d.slice/crio-5616396efaf8acd1b8d56e107d539348dc37b83540dd0e38600f718a685c6570 WatchSource:0}: Error finding container 5616396efaf8acd1b8d56e107d539348dc37b83540dd0e38600f718a685c6570: Status 404 returned error can't find the container with id 5616396efaf8acd1b8d56e107d539348dc37b83540dd0e38600f718a685c6570 Nov 03 13:51:59.583663 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:51:59.583625 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-b5plm" event={"ID":"53e3e37b-99af-4a19-baa8-0a1d954a220d","Type":"ContainerStarted","Data":"5616396efaf8acd1b8d56e107d539348dc37b83540dd0e38600f718a685c6570"} Nov 03 13:52:00.590849 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:52:00.590814 5024 generic.go:356] "Generic (PLEG): container finished" podID="53e3e37b-99af-4a19-baa8-0a1d954a220d" containerID="6c8ecbb00ccb09b6bed08c6595f5893fa4ca1353392e265f4034117058103c42" exitCode=0 Nov 03 13:52:00.591220 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:52:00.590892 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-b5plm" event={"ID":"53e3e37b-99af-4a19-baa8-0a1d954a220d","Type":"ContainerDied","Data":"6c8ecbb00ccb09b6bed08c6595f5893fa4ca1353392e265f4034117058103c42"} Nov 03 13:52:01.749482 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:52:01.749449 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-b5plm" Nov 03 13:52:01.766692 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:52:01.766661 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53e3e37b-99af-4a19-baa8-0a1d954a220d-kube-api-access-c2zbl\" (UniqueName: \"kubernetes.io/projected/53e3e37b-99af-4a19-baa8-0a1d954a220d-kube-api-access-c2zbl\") pod \"53e3e37b-99af-4a19-baa8-0a1d954a220d\" (UID: \"53e3e37b-99af-4a19-baa8-0a1d954a220d\") " Nov 03 13:52:01.766858 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:52:01.766744 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/53e3e37b-99af-4a19-baa8-0a1d954a220d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53e3e37b-99af-4a19-baa8-0a1d954a220d-maas-api-xks-ca\") pod \"53e3e37b-99af-4a19-baa8-0a1d954a220d\" (UID: \"53e3e37b-99af-4a19-baa8-0a1d954a220d\") " Nov 03 13:52:01.766858 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:52:01.766771 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/53e3e37b-99af-4a19-baa8-0a1d954a220d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53e3e37b-99af-4a19-baa8-0a1d954a220d-maas-api-openshift-ca\") pod \"53e3e37b-99af-4a19-baa8-0a1d954a220d\" (UID: \"53e3e37b-99af-4a19-baa8-0a1d954a220d\") " Nov 03 13:52:01.767658 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:52:01.767230 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53e3e37b-99af-4a19-baa8-0a1d954a220d-maas-api-openshift-ca" pod "53e3e37b-99af-4a19-baa8-0a1d954a220d" (UID: "53e3e37b-99af-4a19-baa8-0a1d954a220d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:52:01.769279 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:52:01.769251 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53e3e37b-99af-4a19-baa8-0a1d954a220d-kube-api-access-c2zbl" pod "53e3e37b-99af-4a19-baa8-0a1d954a220d" (UID: "53e3e37b-99af-4a19-baa8-0a1d954a220d"). InnerVolumeSpecName "kube-api-access-c2zbl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:52:01.769943 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:52:01.769918 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/53e3e37b-99af-4a19-baa8-0a1d954a220d-maas-api-xks-ca" pod "53e3e37b-99af-4a19-baa8-0a1d954a220d" (UID: "53e3e37b-99af-4a19-baa8-0a1d954a220d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 13:52:01.868280 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:52:01.868244 5024 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53e3e37b-99af-4a19-baa8-0a1d954a220d-maas-api-xks-ca\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:52:01.868280 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:52:01.868279 5024 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53e3e37b-99af-4a19-baa8-0a1d954a220d-maas-api-openshift-ca\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:52:01.868506 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:52:01.868315 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c2zbl\" (UniqueName: \"kubernetes.io/projected/53e3e37b-99af-4a19-baa8-0a1d954a220d-kube-api-access-c2zbl\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:52:02.604595 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:52:02.604557 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-b5plm" Nov 03 13:52:02.604784 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:52:02.604565 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-b5plm" event={"ID":"53e3e37b-99af-4a19-baa8-0a1d954a220d","Type":"ContainerDied","Data":"5616396efaf8acd1b8d56e107d539348dc37b83540dd0e38600f718a685c6570"} Nov 03 13:52:02.604784 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:52:02.604693 5024 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5616396efaf8acd1b8d56e107d539348dc37b83540dd0e38600f718a685c6570" Nov 03 13:52:09.513432 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:52:09.513379 5024 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d7mwz" containerName="extractor" Nov 03 13:52:21.694452 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:52:21.694412 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-w7kkm"] Nov 03 13:52:21.695066 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:52:21.694985 5024 cpu_manager.go:414] "containerMap: removing container" podUID="53e3e37b-99af-4a19-baa8-0a1d954a220d" containerName="revoke-keys" Nov 03 13:52:21.695066 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:52:21.695004 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53e3e37b-99af-4a19-baa8-0a1d954a220d" containerName="revoke-keys" Nov 03 13:52:21.695066 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:52:21.695030 5024 cpu_manager.go:414] "containerMap: removing container" podUID="11fc0ab3-5590-4613-8f68-cf4c75791d3a" containerName="istio-proxy" Nov 03 13:52:21.695066 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:52:21.695037 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11fc0ab3-5590-4613-8f68-cf4c75791d3a" containerName="istio-proxy" Nov 03 13:52:21.695225 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:52:21.695118 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="53e3e37b-99af-4a19-baa8-0a1d954a220d" containerName="revoke-keys" Nov 03 13:52:21.698634 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:52:21.698614 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-w7kkm" Nov 03 13:52:21.704561 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:52:21.704537 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 13:52:21.704732 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:52:21.704575 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 13:52:21.704820 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:52:21.704793 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 13:52:21.704883 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:52:21.704865 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w6vqv\"" Nov 03 13:52:21.705976 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:52:21.705948 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-w7kkm"] Nov 03 13:52:21.774452 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:52:21.774415 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dk6w2\" (UniqueName: \"kubernetes.io/projected/1c854eda-d4f0-47a1-bde8-8ccd7f36907a-kube-api-access-dk6w2\") pod \"maas-api-revoke-sub-f03629db-w7kkm\" (UID: \"1c854eda-d4f0-47a1-bde8-8ccd7f36907a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-w7kkm" Nov 03 13:52:21.774610 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:52:21.774464 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c854eda-d4f0-47a1-bde8-8ccd7f36907a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-w7kkm\" (UID: \"1c854eda-d4f0-47a1-bde8-8ccd7f36907a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-w7kkm" Nov 03 13:52:21.774654 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:52:21.774628 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c854eda-d4f0-47a1-bde8-8ccd7f36907a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-w7kkm\" (UID: \"1c854eda-d4f0-47a1-bde8-8ccd7f36907a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-w7kkm" Nov 03 13:52:21.875130 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:52:21.875092 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dk6w2\" (UniqueName: \"kubernetes.io/projected/1c854eda-d4f0-47a1-bde8-8ccd7f36907a-kube-api-access-dk6w2\") pod \"maas-api-revoke-sub-f03629db-w7kkm\" (UID: \"1c854eda-d4f0-47a1-bde8-8ccd7f36907a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-w7kkm" Nov 03 13:52:21.875343 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:52:21.875142 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c854eda-d4f0-47a1-bde8-8ccd7f36907a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-w7kkm\" (UID: \"1c854eda-d4f0-47a1-bde8-8ccd7f36907a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-w7kkm" Nov 03 13:52:21.875343 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:52:21.875202 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c854eda-d4f0-47a1-bde8-8ccd7f36907a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-w7kkm\" (UID: \"1c854eda-d4f0-47a1-bde8-8ccd7f36907a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-w7kkm" Nov 03 13:52:21.875811 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:52:21.875788 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c854eda-d4f0-47a1-bde8-8ccd7f36907a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-w7kkm\" (UID: \"1c854eda-d4f0-47a1-bde8-8ccd7f36907a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-w7kkm" Nov 03 13:52:21.877505 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:52:21.877482 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c854eda-d4f0-47a1-bde8-8ccd7f36907a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-w7kkm\" (UID: \"1c854eda-d4f0-47a1-bde8-8ccd7f36907a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-w7kkm" Nov 03 13:52:21.884557 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:52:21.884533 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dk6w2\" (UniqueName: \"kubernetes.io/projected/1c854eda-d4f0-47a1-bde8-8ccd7f36907a-kube-api-access-dk6w2\") pod \"maas-api-revoke-sub-f03629db-w7kkm\" (UID: \"1c854eda-d4f0-47a1-bde8-8ccd7f36907a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-w7kkm" Nov 03 13:52:22.014229 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:52:22.014147 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-w7kkm" Nov 03 13:52:22.297876 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:52:22.297845 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-w7kkm"] Nov 03 13:52:22.299369 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:52:22.299341 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c854eda_d4f0_47a1_bde8_8ccd7f36907a.slice/crio-c4fe17c27f59285f13161ecba70f61bcb3f2f046c44d2ca7175a8eababb66178 WatchSource:0}: Error finding container c4fe17c27f59285f13161ecba70f61bcb3f2f046c44d2ca7175a8eababb66178: Status 404 returned error can't find the container with id c4fe17c27f59285f13161ecba70f61bcb3f2f046c44d2ca7175a8eababb66178 Nov 03 13:52:22.824738 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:52:22.824698 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-w7kkm" event={"ID":"1c854eda-d4f0-47a1-bde8-8ccd7f36907a","Type":"ContainerStarted","Data":"47b5206e2d1011ae80bcef3e59082a8bad65f13f10145ad5ddcb13726e29f378"} Nov 03 13:52:22.825135 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:52:22.824764 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-w7kkm" event={"ID":"1c854eda-d4f0-47a1-bde8-8ccd7f36907a","Type":"ContainerStarted","Data":"c4fe17c27f59285f13161ecba70f61bcb3f2f046c44d2ca7175a8eababb66178"} Nov 03 13:52:22.844953 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:52:22.844869 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-w7kkm" podStartSLOduration=1.844857478 podStartE2EDuration="1.844857478s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:52:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:52:22.842422792 +0000 UTC m=+2622.850614070" watchObservedRunningTime="2026-11-03 13:52:22.844857478 +0000 UTC m=+2622.853048755" Nov 03 13:52:23.831673 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:52:23.831638 5024 generic.go:356] "Generic (PLEG): container finished" podID="1c854eda-d4f0-47a1-bde8-8ccd7f36907a" containerID="47b5206e2d1011ae80bcef3e59082a8bad65f13f10145ad5ddcb13726e29f378" exitCode=0 Nov 03 13:52:23.832077 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:52:23.831731 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-w7kkm" event={"ID":"1c854eda-d4f0-47a1-bde8-8ccd7f36907a","Type":"ContainerDied","Data":"47b5206e2d1011ae80bcef3e59082a8bad65f13f10145ad5ddcb13726e29f378"} Nov 03 13:52:24.971867 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:52:24.971845 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-w7kkm" Nov 03 13:52:25.003420 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:52:25.003389 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c854eda-d4f0-47a1-bde8-8ccd7f36907a-kube-api-access-dk6w2\" (UniqueName: \"kubernetes.io/projected/1c854eda-d4f0-47a1-bde8-8ccd7f36907a-kube-api-access-dk6w2\") pod \"1c854eda-d4f0-47a1-bde8-8ccd7f36907a\" (UID: \"1c854eda-d4f0-47a1-bde8-8ccd7f36907a\") " Nov 03 13:52:25.003566 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:52:25.003432 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1c854eda-d4f0-47a1-bde8-8ccd7f36907a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c854eda-d4f0-47a1-bde8-8ccd7f36907a-maas-api-xks-ca\") pod \"1c854eda-d4f0-47a1-bde8-8ccd7f36907a\" (UID: \"1c854eda-d4f0-47a1-bde8-8ccd7f36907a\") " Nov 03 13:52:25.003642 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:52:25.003611 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c854eda-d4f0-47a1-bde8-8ccd7f36907a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c854eda-d4f0-47a1-bde8-8ccd7f36907a-maas-api-openshift-ca\") pod \"1c854eda-d4f0-47a1-bde8-8ccd7f36907a\" (UID: \"1c854eda-d4f0-47a1-bde8-8ccd7f36907a\") " Nov 03 13:52:25.004056 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:52:25.004021 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c854eda-d4f0-47a1-bde8-8ccd7f36907a-maas-api-openshift-ca" pod "1c854eda-d4f0-47a1-bde8-8ccd7f36907a" (UID: "1c854eda-d4f0-47a1-bde8-8ccd7f36907a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:52:25.005940 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:52:25.005909 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c854eda-d4f0-47a1-bde8-8ccd7f36907a-maas-api-xks-ca" pod "1c854eda-d4f0-47a1-bde8-8ccd7f36907a" (UID: "1c854eda-d4f0-47a1-bde8-8ccd7f36907a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 13:52:25.006023 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:52:25.005954 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c854eda-d4f0-47a1-bde8-8ccd7f36907a-kube-api-access-dk6w2" pod "1c854eda-d4f0-47a1-bde8-8ccd7f36907a" (UID: "1c854eda-d4f0-47a1-bde8-8ccd7f36907a"). InnerVolumeSpecName "kube-api-access-dk6w2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:52:25.104834 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:52:25.104803 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dk6w2\" (UniqueName: \"kubernetes.io/projected/1c854eda-d4f0-47a1-bde8-8ccd7f36907a-kube-api-access-dk6w2\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:52:25.104834 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:52:25.104833 5024 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c854eda-d4f0-47a1-bde8-8ccd7f36907a-maas-api-xks-ca\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:52:25.105021 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:52:25.104843 5024 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c854eda-d4f0-47a1-bde8-8ccd7f36907a-maas-api-openshift-ca\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:52:25.844728 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:52:25.844687 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-w7kkm" event={"ID":"1c854eda-d4f0-47a1-bde8-8ccd7f36907a","Type":"ContainerDied","Data":"c4fe17c27f59285f13161ecba70f61bcb3f2f046c44d2ca7175a8eababb66178"} Nov 03 13:52:25.844728 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:52:25.844728 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-w7kkm" Nov 03 13:52:25.844937 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:52:25.844743 5024 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c4fe17c27f59285f13161ecba70f61bcb3f2f046c44d2ca7175a8eababb66178" Nov 03 13:52:37.866994 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:52:37.866899 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-gpnsl"] Nov 03 13:52:37.867784 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:52:37.867399 5024 cpu_manager.go:414] "containerMap: removing container" podUID="1c854eda-d4f0-47a1-bde8-8ccd7f36907a" containerName="revoke-keys" Nov 03 13:52:37.867784 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:52:37.867413 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c854eda-d4f0-47a1-bde8-8ccd7f36907a" containerName="revoke-keys" Nov 03 13:52:37.867784 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:52:37.867519 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c854eda-d4f0-47a1-bde8-8ccd7f36907a" containerName="revoke-keys" Nov 03 13:52:37.871293 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:52:37.871241 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-gpnsl" Nov 03 13:52:37.876794 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:52:37.876663 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 13:52:37.877773 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:52:37.877743 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-gpnsl"] Nov 03 13:52:37.879549 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:52:37.879492 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 13:52:37.879549 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:52:37.879510 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 13:52:37.879549 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:52:37.879500 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w6vqv\"" Nov 03 13:52:37.991280 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:52:37.991238 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2535a241-4e8b-479f-ad62-2645a1b19874-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-gpnsl\" (UID: \"2535a241-4e8b-479f-ad62-2645a1b19874\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-gpnsl" Nov 03 13:52:37.991504 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:52:37.991389 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2535a241-4e8b-479f-ad62-2645a1b19874-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-gpnsl\" (UID: \"2535a241-4e8b-479f-ad62-2645a1b19874\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-gpnsl" Nov 03 13:52:37.991578 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:52:37.991520 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t7ftf\" (UniqueName: \"kubernetes.io/projected/2535a241-4e8b-479f-ad62-2645a1b19874-kube-api-access-t7ftf\") pod \"maas-api-revoke-sub-0165d7d2-gpnsl\" (UID: \"2535a241-4e8b-479f-ad62-2645a1b19874\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-gpnsl" Nov 03 13:52:38.092359 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:52:38.092324 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7ftf\" (UniqueName: \"kubernetes.io/projected/2535a241-4e8b-479f-ad62-2645a1b19874-kube-api-access-t7ftf\") pod \"maas-api-revoke-sub-0165d7d2-gpnsl\" (UID: \"2535a241-4e8b-479f-ad62-2645a1b19874\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-gpnsl" Nov 03 13:52:38.092529 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:52:38.092415 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2535a241-4e8b-479f-ad62-2645a1b19874-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-gpnsl\" (UID: \"2535a241-4e8b-479f-ad62-2645a1b19874\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-gpnsl" Nov 03 13:52:38.092529 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:52:38.092475 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2535a241-4e8b-479f-ad62-2645a1b19874-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-gpnsl\" (UID: \"2535a241-4e8b-479f-ad62-2645a1b19874\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-gpnsl" Nov 03 13:52:38.093105 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:52:38.093080 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2535a241-4e8b-479f-ad62-2645a1b19874-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-gpnsl\" (UID: \"2535a241-4e8b-479f-ad62-2645a1b19874\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-gpnsl" Nov 03 13:52:38.094788 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:52:38.094767 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2535a241-4e8b-479f-ad62-2645a1b19874-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-gpnsl\" (UID: \"2535a241-4e8b-479f-ad62-2645a1b19874\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-gpnsl" Nov 03 13:52:38.101480 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:52:38.101457 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t7ftf\" (UniqueName: \"kubernetes.io/projected/2535a241-4e8b-479f-ad62-2645a1b19874-kube-api-access-t7ftf\") pod \"maas-api-revoke-sub-0165d7d2-gpnsl\" (UID: \"2535a241-4e8b-479f-ad62-2645a1b19874\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-gpnsl" Nov 03 13:52:38.187251 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:52:38.187172 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-gpnsl" Nov 03 13:52:38.460217 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:52:38.460183 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-gpnsl"] Nov 03 13:52:38.461649 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:52:38.461620 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2535a241_4e8b_479f_ad62_2645a1b19874.slice/crio-65307d5f33f0ffae36c8f6186837421b1bb4fc48f9818e6967351d1053899976 WatchSource:0}: Error finding container 65307d5f33f0ffae36c8f6186837421b1bb4fc48f9818e6967351d1053899976: Status 404 returned error can't find the container with id 65307d5f33f0ffae36c8f6186837421b1bb4fc48f9818e6967351d1053899976 Nov 03 13:52:38.959986 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:52:38.959946 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-gpnsl" event={"ID":"2535a241-4e8b-479f-ad62-2645a1b19874","Type":"ContainerStarted","Data":"64505a03a78de5fb06c4527e226b4bcdda292424e4a82b5b9625c9f85212587c"} Nov 03 13:52:38.960387 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:52:38.960011 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-gpnsl" event={"ID":"2535a241-4e8b-479f-ad62-2645a1b19874","Type":"ContainerStarted","Data":"65307d5f33f0ffae36c8f6186837421b1bb4fc48f9818e6967351d1053899976"} Nov 03 13:52:38.980114 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:52:38.980061 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-gpnsl" podStartSLOduration=1.980048008 podStartE2EDuration="1.980048008s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:52:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:52:38.978405703 +0000 UTC m=+2638.986596979" watchObservedRunningTime="2026-11-03 13:52:38.980048008 +0000 UTC m=+2638.988239284" Nov 03 13:52:39.507128 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:52:39.507086 5024 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d7mwz" containerName="extractor" Nov 03 13:52:39.966821 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:52:39.966783 5024 generic.go:356] "Generic (PLEG): container finished" podID="2535a241-4e8b-479f-ad62-2645a1b19874" containerID="64505a03a78de5fb06c4527e226b4bcdda292424e4a82b5b9625c9f85212587c" exitCode=0 Nov 03 13:52:39.967222 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:52:39.966870 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-gpnsl" event={"ID":"2535a241-4e8b-479f-ad62-2645a1b19874","Type":"ContainerDied","Data":"64505a03a78de5fb06c4527e226b4bcdda292424e4a82b5b9625c9f85212587c"} Nov 03 13:52:41.101176 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:52:41.101145 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-gpnsl" Nov 03 13:52:41.222436 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:52:41.222353 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2535a241-4e8b-479f-ad62-2645a1b19874-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2535a241-4e8b-479f-ad62-2645a1b19874-maas-api-openshift-ca\") pod \"2535a241-4e8b-479f-ad62-2645a1b19874\" (UID: \"2535a241-4e8b-479f-ad62-2645a1b19874\") " Nov 03 13:52:41.222436 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:52:41.222432 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2535a241-4e8b-479f-ad62-2645a1b19874-kube-api-access-t7ftf\" (UniqueName: \"kubernetes.io/projected/2535a241-4e8b-479f-ad62-2645a1b19874-kube-api-access-t7ftf\") pod \"2535a241-4e8b-479f-ad62-2645a1b19874\" (UID: \"2535a241-4e8b-479f-ad62-2645a1b19874\") " Nov 03 13:52:41.222681 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:52:41.222465 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2535a241-4e8b-479f-ad62-2645a1b19874-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2535a241-4e8b-479f-ad62-2645a1b19874-maas-api-xks-ca\") pod \"2535a241-4e8b-479f-ad62-2645a1b19874\" (UID: \"2535a241-4e8b-479f-ad62-2645a1b19874\") " Nov 03 13:52:41.222746 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:52:41.222726 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2535a241-4e8b-479f-ad62-2645a1b19874-maas-api-openshift-ca" pod "2535a241-4e8b-479f-ad62-2645a1b19874" (UID: "2535a241-4e8b-479f-ad62-2645a1b19874"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:52:41.223141 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:52:41.223116 5024 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2535a241-4e8b-479f-ad62-2645a1b19874-maas-api-openshift-ca\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:52:41.224690 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:52:41.224664 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2535a241-4e8b-479f-ad62-2645a1b19874-maas-api-xks-ca" pod "2535a241-4e8b-479f-ad62-2645a1b19874" (UID: "2535a241-4e8b-479f-ad62-2645a1b19874"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 13:52:41.224786 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:52:41.224721 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2535a241-4e8b-479f-ad62-2645a1b19874-kube-api-access-t7ftf" pod "2535a241-4e8b-479f-ad62-2645a1b19874" (UID: "2535a241-4e8b-479f-ad62-2645a1b19874"). InnerVolumeSpecName "kube-api-access-t7ftf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:52:41.324418 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:52:41.324385 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t7ftf\" (UniqueName: \"kubernetes.io/projected/2535a241-4e8b-479f-ad62-2645a1b19874-kube-api-access-t7ftf\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:52:41.324418 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:52:41.324415 5024 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2535a241-4e8b-479f-ad62-2645a1b19874-maas-api-xks-ca\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:52:41.978758 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:52:41.978728 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-gpnsl" Nov 03 13:52:41.978959 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:52:41.978760 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-gpnsl" event={"ID":"2535a241-4e8b-479f-ad62-2645a1b19874","Type":"ContainerDied","Data":"65307d5f33f0ffae36c8f6186837421b1bb4fc48f9818e6967351d1053899976"} Nov 03 13:52:41.978959 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:52:41.978812 5024 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="65307d5f33f0ffae36c8f6186837421b1bb4fc48f9818e6967351d1053899976" Nov 03 13:52:45.806960 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:52:45.806926 5024 scope.go:122] "RemoveContainer" containerID="87612cd87efdc500c002e0b19137253dc1e77e1d2629ffebb951c20aa6887506" Nov 03 13:52:48.686900 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:52:48.686856 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-cwxvr"] Nov 03 13:52:48.687598 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:52:48.687577 5024 cpu_manager.go:414] "containerMap: removing container" podUID="2535a241-4e8b-479f-ad62-2645a1b19874" containerName="revoke-keys" Nov 03 13:52:48.687684 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:52:48.687601 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2535a241-4e8b-479f-ad62-2645a1b19874" containerName="revoke-keys" Nov 03 13:52:48.687725 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:52:48.687696 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="2535a241-4e8b-479f-ad62-2645a1b19874" containerName="revoke-keys" Nov 03 13:52:48.692951 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:52:48.692918 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-cwxvr" Nov 03 13:52:48.697103 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:52:48.697073 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-cwxvr"] Nov 03 13:52:48.697559 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:52:48.697526 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 13:52:48.699522 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:52:48.699483 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 13:52:48.699636 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:52:48.699516 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 13:52:48.699636 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:52:48.699535 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w6vqv\"" Nov 03 13:52:48.804274 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:52:48.804237 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91828ac5-9a2d-4704-95dc-e8b054e891ec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-cwxvr\" (UID: \"91828ac5-9a2d-4704-95dc-e8b054e891ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-cwxvr" Nov 03 13:52:48.804445 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:52:48.804318 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91828ac5-9a2d-4704-95dc-e8b054e891ec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-cwxvr\" (UID: \"91828ac5-9a2d-4704-95dc-e8b054e891ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-cwxvr" Nov 03 13:52:48.804445 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:52:48.804432 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f9gsq\" (UniqueName: \"kubernetes.io/projected/91828ac5-9a2d-4704-95dc-e8b054e891ec-kube-api-access-f9gsq\") pod \"maas-api-revoke-sub-6b19338f-cwxvr\" (UID: \"91828ac5-9a2d-4704-95dc-e8b054e891ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-cwxvr" Nov 03 13:52:48.905126 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:52:48.905089 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91828ac5-9a2d-4704-95dc-e8b054e891ec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-cwxvr\" (UID: \"91828ac5-9a2d-4704-95dc-e8b054e891ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-cwxvr" Nov 03 13:52:48.905322 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:52:48.905150 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91828ac5-9a2d-4704-95dc-e8b054e891ec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-cwxvr\" (UID: \"91828ac5-9a2d-4704-95dc-e8b054e891ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-cwxvr" Nov 03 13:52:48.905322 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:52:48.905204 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9gsq\" (UniqueName: \"kubernetes.io/projected/91828ac5-9a2d-4704-95dc-e8b054e891ec-kube-api-access-f9gsq\") pod \"maas-api-revoke-sub-6b19338f-cwxvr\" (UID: \"91828ac5-9a2d-4704-95dc-e8b054e891ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-cwxvr" Nov 03 13:52:48.905897 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:52:48.905873 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91828ac5-9a2d-4704-95dc-e8b054e891ec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-cwxvr\" (UID: \"91828ac5-9a2d-4704-95dc-e8b054e891ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-cwxvr" Nov 03 13:52:48.907767 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:52:48.907740 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91828ac5-9a2d-4704-95dc-e8b054e891ec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-cwxvr\" (UID: \"91828ac5-9a2d-4704-95dc-e8b054e891ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-cwxvr" Nov 03 13:52:48.914299 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:52:48.914264 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9gsq\" (UniqueName: \"kubernetes.io/projected/91828ac5-9a2d-4704-95dc-e8b054e891ec-kube-api-access-f9gsq\") pod \"maas-api-revoke-sub-6b19338f-cwxvr\" (UID: \"91828ac5-9a2d-4704-95dc-e8b054e891ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-cwxvr" Nov 03 13:52:49.008568 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:52:49.008506 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-cwxvr" Nov 03 13:52:49.286510 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:52:49.286479 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-cwxvr"] Nov 03 13:52:49.287516 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:52:49.287488 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91828ac5_9a2d_4704_95dc_e8b054e891ec.slice/crio-b1e86b6a018b8220fd8f24d7490947ecab0c70a9484a1089a14cbedbc66e6e8f WatchSource:0}: Error finding container b1e86b6a018b8220fd8f24d7490947ecab0c70a9484a1089a14cbedbc66e6e8f: Status 404 returned error can't find the container with id b1e86b6a018b8220fd8f24d7490947ecab0c70a9484a1089a14cbedbc66e6e8f Nov 03 13:52:50.022030 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:52:50.021988 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-cwxvr" event={"ID":"91828ac5-9a2d-4704-95dc-e8b054e891ec","Type":"ContainerStarted","Data":"ddd893680d229e89241431680ca35705de2e44615da723d4ca40d5d9450b5556"} Nov 03 13:52:50.022477 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:52:50.022055 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-cwxvr" event={"ID":"91828ac5-9a2d-4704-95dc-e8b054e891ec","Type":"ContainerStarted","Data":"b1e86b6a018b8220fd8f24d7490947ecab0c70a9484a1089a14cbedbc66e6e8f"} Nov 03 13:52:50.040486 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:52:50.040436 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-cwxvr" podStartSLOduration=2.04042189 podStartE2EDuration="2.04042189s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:52:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:52:50.039144089 +0000 UTC m=+2650.047335366" watchObservedRunningTime="2026-11-03 13:52:50.04042189 +0000 UTC m=+2650.048613168" Nov 03 13:52:51.028812 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:52:51.028777 5024 generic.go:356] "Generic (PLEG): container finished" podID="91828ac5-9a2d-4704-95dc-e8b054e891ec" containerID="ddd893680d229e89241431680ca35705de2e44615da723d4ca40d5d9450b5556" exitCode=0 Nov 03 13:52:51.029210 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:52:51.028863 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-cwxvr" event={"ID":"91828ac5-9a2d-4704-95dc-e8b054e891ec","Type":"ContainerDied","Data":"ddd893680d229e89241431680ca35705de2e44615da723d4ca40d5d9450b5556"} Nov 03 13:52:52.175099 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:52:52.175073 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-cwxvr" Nov 03 13:52:52.241335 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:52:52.241277 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/91828ac5-9a2d-4704-95dc-e8b054e891ec-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91828ac5-9a2d-4704-95dc-e8b054e891ec-maas-api-openshift-ca\") pod \"91828ac5-9a2d-4704-95dc-e8b054e891ec\" (UID: \"91828ac5-9a2d-4704-95dc-e8b054e891ec\") " Nov 03 13:52:52.241485 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:52:52.241423 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/91828ac5-9a2d-4704-95dc-e8b054e891ec-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91828ac5-9a2d-4704-95dc-e8b054e891ec-maas-api-xks-ca\") pod \"91828ac5-9a2d-4704-95dc-e8b054e891ec\" (UID: \"91828ac5-9a2d-4704-95dc-e8b054e891ec\") " Nov 03 13:52:52.241485 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:52:52.241462 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91828ac5-9a2d-4704-95dc-e8b054e891ec-kube-api-access-f9gsq\" (UniqueName: \"kubernetes.io/projected/91828ac5-9a2d-4704-95dc-e8b054e891ec-kube-api-access-f9gsq\") pod \"91828ac5-9a2d-4704-95dc-e8b054e891ec\" (UID: \"91828ac5-9a2d-4704-95dc-e8b054e891ec\") " Nov 03 13:52:52.241653 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:52:52.241628 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91828ac5-9a2d-4704-95dc-e8b054e891ec-maas-api-openshift-ca" pod "91828ac5-9a2d-4704-95dc-e8b054e891ec" (UID: "91828ac5-9a2d-4704-95dc-e8b054e891ec"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:52:52.241787 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:52:52.241772 5024 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91828ac5-9a2d-4704-95dc-e8b054e891ec-maas-api-openshift-ca\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:52:52.243596 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:52:52.243571 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/91828ac5-9a2d-4704-95dc-e8b054e891ec-maas-api-xks-ca" pod "91828ac5-9a2d-4704-95dc-e8b054e891ec" (UID: "91828ac5-9a2d-4704-95dc-e8b054e891ec"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 13:52:52.243705 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:52:52.243636 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91828ac5-9a2d-4704-95dc-e8b054e891ec-kube-api-access-f9gsq" pod "91828ac5-9a2d-4704-95dc-e8b054e891ec" (UID: "91828ac5-9a2d-4704-95dc-e8b054e891ec"). InnerVolumeSpecName "kube-api-access-f9gsq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:52:52.342405 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:52:52.342324 5024 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91828ac5-9a2d-4704-95dc-e8b054e891ec-maas-api-xks-ca\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:52:52.342405 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:52:52.342358 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f9gsq\" (UniqueName: \"kubernetes.io/projected/91828ac5-9a2d-4704-95dc-e8b054e891ec-kube-api-access-f9gsq\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:52:53.040454 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:52:53.040424 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-cwxvr" Nov 03 13:52:53.040454 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:52:53.040450 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-cwxvr" event={"ID":"91828ac5-9a2d-4704-95dc-e8b054e891ec","Type":"ContainerDied","Data":"b1e86b6a018b8220fd8f24d7490947ecab0c70a9484a1089a14cbedbc66e6e8f"} Nov 03 13:52:53.040667 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:52:53.040494 5024 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b1e86b6a018b8220fd8f24d7490947ecab0c70a9484a1089a14cbedbc66e6e8f" Nov 03 13:53:00.160020 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:53:00.159982 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-prio-pool-d2dab122-kserve-58bf7b4f87-jjs6b"] Nov 03 13:53:00.161221 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:53:00.161194 5024 cpu_manager.go:414] "containerMap: removing container" podUID="91828ac5-9a2d-4704-95dc-e8b054e891ec" containerName="revoke-keys" Nov 03 13:53:00.161414 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:53:00.161396 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91828ac5-9a2d-4704-95dc-e8b054e891ec" containerName="revoke-keys" Nov 03 13:53:00.161943 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:53:00.161922 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="91828ac5-9a2d-4704-95dc-e8b054e891ec" containerName="revoke-keys" Nov 03 13:53:00.218714 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:53:00.218681 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-prio-pool-d2dab122-kserve-58bf7b4f87-jjs6b"] Nov 03 13:53:00.219020 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:53:00.218999 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-d2dab122-kserve-58bf7b4f87-jjs6b" Nov 03 13:53:00.248594 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:53:00.233436 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-prio-pool-d2dab122-kserve-self-signed-certs\"" Nov 03 13:53:00.339308 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:53:00.335842 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xmjhc\" (UniqueName: \"kubernetes.io/projected/c2d91fcc-f535-463e-8f6d-88d7892578b8-kube-api-access-xmjhc\") pod \"e2e-prio-pool-d2dab122-kserve-58bf7b4f87-jjs6b\" (UID: \"c2d91fcc-f535-463e-8f6d-88d7892578b8\") " pod="llm/e2e-prio-pool-d2dab122-kserve-58bf7b4f87-jjs6b" Nov 03 13:53:00.355413 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:53:00.355375 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c2d91fcc-f535-463e-8f6d-88d7892578b8-tls-certs\") pod \"e2e-prio-pool-d2dab122-kserve-58bf7b4f87-jjs6b\" (UID: \"c2d91fcc-f535-463e-8f6d-88d7892578b8\") " pod="llm/e2e-prio-pool-d2dab122-kserve-58bf7b4f87-jjs6b" Nov 03 13:53:00.355692 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:53:00.355665 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c2d91fcc-f535-463e-8f6d-88d7892578b8-tmp-dir\") pod \"e2e-prio-pool-d2dab122-kserve-58bf7b4f87-jjs6b\" (UID: \"c2d91fcc-f535-463e-8f6d-88d7892578b8\") " pod="llm/e2e-prio-pool-d2dab122-kserve-58bf7b4f87-jjs6b" Nov 03 13:53:00.360305 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:53:00.355851 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c2d91fcc-f535-463e-8f6d-88d7892578b8-model-cache\") pod \"e2e-prio-pool-d2dab122-kserve-58bf7b4f87-jjs6b\" (UID: \"c2d91fcc-f535-463e-8f6d-88d7892578b8\") " pod="llm/e2e-prio-pool-d2dab122-kserve-58bf7b4f87-jjs6b" Nov 03 13:53:00.360531 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:53:00.360507 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c2d91fcc-f535-463e-8f6d-88d7892578b8-dshm\") pod \"e2e-prio-pool-d2dab122-kserve-58bf7b4f87-jjs6b\" (UID: \"c2d91fcc-f535-463e-8f6d-88d7892578b8\") " pod="llm/e2e-prio-pool-d2dab122-kserve-58bf7b4f87-jjs6b" Nov 03 13:53:00.365310 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:53:00.360919 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c2d91fcc-f535-463e-8f6d-88d7892578b8-home\") pod \"e2e-prio-pool-d2dab122-kserve-58bf7b4f87-jjs6b\" (UID: \"c2d91fcc-f535-463e-8f6d-88d7892578b8\") " pod="llm/e2e-prio-pool-d2dab122-kserve-58bf7b4f87-jjs6b" Nov 03 13:53:00.491359 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:53:00.468815 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xmjhc\" (UniqueName: \"kubernetes.io/projected/c2d91fcc-f535-463e-8f6d-88d7892578b8-kube-api-access-xmjhc\") pod \"e2e-prio-pool-d2dab122-kserve-58bf7b4f87-jjs6b\" (UID: \"c2d91fcc-f535-463e-8f6d-88d7892578b8\") " pod="llm/e2e-prio-pool-d2dab122-kserve-58bf7b4f87-jjs6b" Nov 03 13:53:00.491662 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:53:00.491628 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c2d91fcc-f535-463e-8f6d-88d7892578b8-tls-certs\") pod \"e2e-prio-pool-d2dab122-kserve-58bf7b4f87-jjs6b\" (UID: \"c2d91fcc-f535-463e-8f6d-88d7892578b8\") " pod="llm/e2e-prio-pool-d2dab122-kserve-58bf7b4f87-jjs6b" Nov 03 13:53:00.491861 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:53:00.491843 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c2d91fcc-f535-463e-8f6d-88d7892578b8-tmp-dir\") pod \"e2e-prio-pool-d2dab122-kserve-58bf7b4f87-jjs6b\" (UID: \"c2d91fcc-f535-463e-8f6d-88d7892578b8\") " pod="llm/e2e-prio-pool-d2dab122-kserve-58bf7b4f87-jjs6b" Nov 03 13:53:00.492007 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:53:00.491988 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c2d91fcc-f535-463e-8f6d-88d7892578b8-model-cache\") pod \"e2e-prio-pool-d2dab122-kserve-58bf7b4f87-jjs6b\" (UID: \"c2d91fcc-f535-463e-8f6d-88d7892578b8\") " pod="llm/e2e-prio-pool-d2dab122-kserve-58bf7b4f87-jjs6b" Nov 03 13:53:00.492150 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:53:00.492131 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c2d91fcc-f535-463e-8f6d-88d7892578b8-dshm\") pod \"e2e-prio-pool-d2dab122-kserve-58bf7b4f87-jjs6b\" (UID: \"c2d91fcc-f535-463e-8f6d-88d7892578b8\") " pod="llm/e2e-prio-pool-d2dab122-kserve-58bf7b4f87-jjs6b" Nov 03 13:53:00.492407 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:53:00.492383 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c2d91fcc-f535-463e-8f6d-88d7892578b8-home\") pod \"e2e-prio-pool-d2dab122-kserve-58bf7b4f87-jjs6b\" (UID: \"c2d91fcc-f535-463e-8f6d-88d7892578b8\") " pod="llm/e2e-prio-pool-d2dab122-kserve-58bf7b4f87-jjs6b" Nov 03 13:53:00.500305 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:53:00.492991 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c2d91fcc-f535-463e-8f6d-88d7892578b8-home\") pod \"e2e-prio-pool-d2dab122-kserve-58bf7b4f87-jjs6b\" (UID: \"c2d91fcc-f535-463e-8f6d-88d7892578b8\") " pod="llm/e2e-prio-pool-d2dab122-kserve-58bf7b4f87-jjs6b" Nov 03 13:53:00.573445 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:53:00.540015 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c2d91fcc-f535-463e-8f6d-88d7892578b8-model-cache\") pod \"e2e-prio-pool-d2dab122-kserve-58bf7b4f87-jjs6b\" (UID: \"c2d91fcc-f535-463e-8f6d-88d7892578b8\") " pod="llm/e2e-prio-pool-d2dab122-kserve-58bf7b4f87-jjs6b" Nov 03 13:53:00.573445 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:53:00.540325 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c2d91fcc-f535-463e-8f6d-88d7892578b8-tmp-dir\") pod \"e2e-prio-pool-d2dab122-kserve-58bf7b4f87-jjs6b\" (UID: \"c2d91fcc-f535-463e-8f6d-88d7892578b8\") " pod="llm/e2e-prio-pool-d2dab122-kserve-58bf7b4f87-jjs6b" Nov 03 13:53:00.573820 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:53:00.573773 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-prio-pool-d2dab122-kserve-router-scheduler-f5cfb7bb-b225h"] Nov 03 13:53:00.639772 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:53:00.639728 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-prio-pool-d2dab122-kserve-router-scheduler-f5cfb7bb-b225h"] Nov 03 13:53:00.652304 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:53:00.651457 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-d2dab122-kserve-router-scheduler-f5cfb7bb-b225h" Nov 03 13:53:00.676381 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:53:00.668585 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c2d91fcc-f535-463e-8f6d-88d7892578b8-dshm\") pod \"e2e-prio-pool-d2dab122-kserve-58bf7b4f87-jjs6b\" (UID: \"c2d91fcc-f535-463e-8f6d-88d7892578b8\") " pod="llm/e2e-prio-pool-d2dab122-kserve-58bf7b4f87-jjs6b" Nov 03 13:53:00.676381 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:53:00.668815 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xmjhc\" (UniqueName: \"kubernetes.io/projected/c2d91fcc-f535-463e-8f6d-88d7892578b8-kube-api-access-xmjhc\") pod \"e2e-prio-pool-d2dab122-kserve-58bf7b4f87-jjs6b\" (UID: \"c2d91fcc-f535-463e-8f6d-88d7892578b8\") " pod="llm/e2e-prio-pool-d2dab122-kserve-58bf7b4f87-jjs6b" Nov 03 13:53:00.764355 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:53:00.753249 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-prio-pool-d2dab122-epp-sa-dockercfg-5rz66\"" Nov 03 13:53:00.799200 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:53:00.766950 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b075d14b-76a6-4425-90e7-556220654cba-tls-certs\") pod \"e2e-prio-pool-d2dab122-kserve-router-scheduler-f5cfb7bb-b225h\" (UID: \"b075d14b-76a6-4425-90e7-556220654cba\") " pod="llm/e2e-prio-pool-d2dab122-kserve-router-scheduler-f5cfb7bb-b225h" Nov 03 13:53:00.799200 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:53:00.767034 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8kd6\" (UniqueName: \"kubernetes.io/projected/b075d14b-76a6-4425-90e7-556220654cba-kube-api-access-z8kd6\") pod \"e2e-prio-pool-d2dab122-kserve-router-scheduler-f5cfb7bb-b225h\" (UID: \"b075d14b-76a6-4425-90e7-556220654cba\") " pod="llm/e2e-prio-pool-d2dab122-kserve-router-scheduler-f5cfb7bb-b225h" Nov 03 13:53:00.873316 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:53:00.864102 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c2d91fcc-f535-463e-8f6d-88d7892578b8-tls-certs\") pod \"e2e-prio-pool-d2dab122-kserve-58bf7b4f87-jjs6b\" (UID: \"c2d91fcc-f535-463e-8f6d-88d7892578b8\") " pod="llm/e2e-prio-pool-d2dab122-kserve-58bf7b4f87-jjs6b" Nov 03 13:53:00.883306 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:53:00.874124 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b075d14b-76a6-4425-90e7-556220654cba-tls-certs\") pod \"e2e-prio-pool-d2dab122-kserve-router-scheduler-f5cfb7bb-b225h\" (UID: \"b075d14b-76a6-4425-90e7-556220654cba\") " pod="llm/e2e-prio-pool-d2dab122-kserve-router-scheduler-f5cfb7bb-b225h" Nov 03 13:53:00.883306 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:53:00.874199 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8kd6\" (UniqueName: \"kubernetes.io/projected/b075d14b-76a6-4425-90e7-556220654cba-kube-api-access-z8kd6\") pod \"e2e-prio-pool-d2dab122-kserve-router-scheduler-f5cfb7bb-b225h\" (UID: \"b075d14b-76a6-4425-90e7-556220654cba\") " pod="llm/e2e-prio-pool-d2dab122-kserve-router-scheduler-f5cfb7bb-b225h" Nov 03 13:53:00.933326 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:53:00.933259 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b075d14b-76a6-4425-90e7-556220654cba-tls-certs\") pod \"e2e-prio-pool-d2dab122-kserve-router-scheduler-f5cfb7bb-b225h\" (UID: \"b075d14b-76a6-4425-90e7-556220654cba\") " pod="llm/e2e-prio-pool-d2dab122-kserve-router-scheduler-f5cfb7bb-b225h" Nov 03 13:53:00.944149 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:53:00.943768 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-d2dab122-kserve-58bf7b4f87-jjs6b" Nov 03 13:53:00.976374 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:53:00.972516 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8kd6\" (UniqueName: \"kubernetes.io/projected/b075d14b-76a6-4425-90e7-556220654cba-kube-api-access-z8kd6\") pod \"e2e-prio-pool-d2dab122-kserve-router-scheduler-f5cfb7bb-b225h\" (UID: \"b075d14b-76a6-4425-90e7-556220654cba\") " pod="llm/e2e-prio-pool-d2dab122-kserve-router-scheduler-f5cfb7bb-b225h" Nov 03 13:53:01.140759 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:53:01.140467 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-d2dab122-kserve-router-scheduler-f5cfb7bb-b225h" Nov 03 13:53:01.687367 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:53:01.687035 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-prio-pool-d2dab122-kserve-58bf7b4f87-jjs6b"] Nov 03 13:53:02.250062 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:53:02.250016 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-d2dab122-kserve-58bf7b4f87-jjs6b" event={"ID":"c2d91fcc-f535-463e-8f6d-88d7892578b8","Type":"ContainerStarted","Data":"863469f4eddcb8d596e304d2ef00511aa9bc391c1d5fc77822bf8bbb4072c969"} Nov 03 13:53:02.338834 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:53:02.338795 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-prio-pool-d2dab122-kserve-router-scheduler-f5cfb7bb-b225h"] Nov 03 13:53:03.260810 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:53:03.260770 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-d2dab122-kserve-router-scheduler-f5cfb7bb-b225h" event={"ID":"b075d14b-76a6-4425-90e7-556220654cba","Type":"ContainerStarted","Data":"15f3913720e1202195572b8e38f3c096b8b7b835bf59246f329fa0b0e6c02acd"} Nov 03 13:53:03.264557 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:53:03.264513 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-d2dab122-kserve-58bf7b4f87-jjs6b" event={"ID":"c2d91fcc-f535-463e-8f6d-88d7892578b8","Type":"ContainerStarted","Data":"832f094d54747701790b64af01117f68352990e8bbf2fdbcace24ac8279fd2a3"} Nov 03 13:53:03.957314 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:53:03.957262 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-prio-pool-d2dab122-kserve-58bf7b4f87-jjs6b" Nov 03 13:53:07.331051 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:53:07.331009 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-d2dab122-kserve-router-scheduler-f5cfb7bb-b225h" event={"ID":"b075d14b-76a6-4425-90e7-556220654cba","Type":"ContainerStarted","Data":"c0cb49c2c0617a26c5d15d3cbc63e31634d7dc68c64ca0144aee751337961ea2"} Nov 03 13:53:07.331474 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:53:07.331104 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-prio-pool-d2dab122-kserve-router-scheduler-f5cfb7bb-b225h" Nov 03 13:53:07.351419 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:53:07.351374 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-prio-pool-d2dab122-kserve-router-scheduler-f5cfb7bb-b225h" podStartSLOduration=3.102081052 podStartE2EDuration="7.351362831s" totalImagesPullingTime="4.249281779s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:53:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:53:07.350044103 +0000 UTC m=+2667.358235391" watchObservedRunningTime="2026-11-03 13:53:07.351362831 +0000 UTC m=+2667.359554108" Nov 03 13:53:07.353193 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:53:07.353159 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-prio-pool-d2dab122-kserve-58bf7b4f87-jjs6b" podStartSLOduration=7.353148862 podStartE2EDuration="7.353148862s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:53:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:53:03.291602203 +0000 UTC m=+2663.299793482" watchObservedRunningTime="2026-11-03 13:53:07.353148862 +0000 UTC m=+2667.361340138" Nov 03 13:53:09.507803 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:53:09.507749 5024 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d7mwz" containerName="extractor" Nov 03 13:53:11.014387 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:53:11.014347 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-prio-pool-d2dab122-kserve-58bf7b4f87-jjs6b" Nov 03 13:53:11.026907 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:53:11.026878 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-prio-pool-d2dab122-kserve-58bf7b4f87-jjs6b" Nov 03 13:53:11.364842 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:53:11.364811 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-prio-pool-d2dab122-kserve-58bf7b4f87-jjs6b" Nov 03 13:53:38.339001 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:53:38.338966 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-prio-pool-d2dab122-kserve-router-scheduler-f5cfb7bb-b225h" Nov 03 13:53:39.506167 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:53:39.506129 5024 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d7mwz" containerName="extractor" Nov 03 13:53:41.451762 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:53:41.451727 5024 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6htht_cd91de08-084e-4667-b13f-2e07b3cff624/console-operator/1.log" Nov 03 13:53:41.465685 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:53:41.465659 5024 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6htht_cd91de08-084e-4667-b13f-2e07b3cff624/console-operator/1.log" Nov 03 13:53:43.024419 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:53:43.024337 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-8jf5w"] Nov 03 13:53:43.027293 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:53:43.027249 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-8jf5w"] Nov 03 13:53:44.111073 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:53:44.111032 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="406b9b9c-d55b-416b-8a28-6f301e0339f8" path="/var/lib/kubelet/pods/406b9b9c-d55b-416b-8a28-6f301e0339f8/volumes" Nov 03 13:54:03.111171 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:54:03.111132 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-prio-pool-d2dab122-kserve-58bf7b4f87-jjs6b"] Nov 03 13:54:03.117167 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:54:03.117137 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-prio-pool-d2dab122-kserve-router-scheduler-f5cfb7bb-b225h"] Nov 03 13:54:08.339632 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:54:08.339514 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-prio-pool-d2dab122-kserve-router-scheduler-f5cfb7bb-b225h" podUID="b075d14b-76a6-4425-90e7-556220654cba" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 13:54:09.680306 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:54:09.679969 5024 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d7mwz" containerName="extractor" Nov 03 13:54:17.023511 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:54:17.023472 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-751e2187-sjv8j"] Nov 03 13:54:17.025949 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:54:17.025916 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-751e2187-sjv8j"] Nov 03 13:54:18.107210 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:54:18.107172 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d5686fa-f7d5-41ff-8763-58e45c26fc70" path="/var/lib/kubelet/pods/8d5686fa-f7d5-41ff-8763-58e45c26fc70/volumes" Nov 03 13:54:18.140688 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:54:18.140656 5024 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-prio-pool-d2dab122-kserve-router-scheduler-f5cfb7bb-b225h" podUID="b075d14b-76a6-4425-90e7-556220654cba" containerName="main" containerID="cri-o://c0cb49c2c0617a26c5d15d3cbc63e31634d7dc68c64ca0144aee751337961ea2" gracePeriod=45 Nov 03 13:54:18.140872 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:54:18.140656 5024 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-prio-pool-d2dab122-kserve-58bf7b4f87-jjs6b" podUID="c2d91fcc-f535-463e-8f6d-88d7892578b8" containerName="main" containerID="cri-o://832f094d54747701790b64af01117f68352990e8bbf2fdbcace24ac8279fd2a3" gracePeriod=45 Nov 03 13:54:18.339421 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:54:18.339380 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-prio-pool-d2dab122-kserve-router-scheduler-f5cfb7bb-b225h" podUID="b075d14b-76a6-4425-90e7-556220654cba" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 13:54:18.396054 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:54:18.396024 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-d2dab122-kserve-58bf7b4f87-jjs6b" Nov 03 13:54:18.522727 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:54:18.522680 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2d91fcc-f535-463e-8f6d-88d7892578b8-home\" (UniqueName: \"kubernetes.io/empty-dir/c2d91fcc-f535-463e-8f6d-88d7892578b8-home\") pod \"c2d91fcc-f535-463e-8f6d-88d7892578b8\" (UID: \"c2d91fcc-f535-463e-8f6d-88d7892578b8\") " Nov 03 13:54:18.522929 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:54:18.522746 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2d91fcc-f535-463e-8f6d-88d7892578b8-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c2d91fcc-f535-463e-8f6d-88d7892578b8-dshm\") pod \"c2d91fcc-f535-463e-8f6d-88d7892578b8\" (UID: \"c2d91fcc-f535-463e-8f6d-88d7892578b8\") " Nov 03 13:54:18.522929 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:54:18.522777 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c2d91fcc-f535-463e-8f6d-88d7892578b8-tls-certs\" (UniqueName: \"kubernetes.io/secret/c2d91fcc-f535-463e-8f6d-88d7892578b8-tls-certs\") pod \"c2d91fcc-f535-463e-8f6d-88d7892578b8\" (UID: \"c2d91fcc-f535-463e-8f6d-88d7892578b8\") " Nov 03 13:54:18.522929 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:54:18.522799 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2d91fcc-f535-463e-8f6d-88d7892578b8-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c2d91fcc-f535-463e-8f6d-88d7892578b8-model-cache\") pod \"c2d91fcc-f535-463e-8f6d-88d7892578b8\" (UID: \"c2d91fcc-f535-463e-8f6d-88d7892578b8\") " Nov 03 13:54:18.522929 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:54:18.522827 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2d91fcc-f535-463e-8f6d-88d7892578b8-kube-api-access-xmjhc\" (UniqueName: \"kubernetes.io/projected/c2d91fcc-f535-463e-8f6d-88d7892578b8-kube-api-access-xmjhc\") pod \"c2d91fcc-f535-463e-8f6d-88d7892578b8\" (UID: \"c2d91fcc-f535-463e-8f6d-88d7892578b8\") " Nov 03 13:54:18.522929 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:54:18.522854 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2d91fcc-f535-463e-8f6d-88d7892578b8-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c2d91fcc-f535-463e-8f6d-88d7892578b8-tmp-dir\") pod \"c2d91fcc-f535-463e-8f6d-88d7892578b8\" (UID: \"c2d91fcc-f535-463e-8f6d-88d7892578b8\") " Nov 03 13:54:18.523234 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:54:18.523045 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2d91fcc-f535-463e-8f6d-88d7892578b8-home" pod "c2d91fcc-f535-463e-8f6d-88d7892578b8" (UID: "c2d91fcc-f535-463e-8f6d-88d7892578b8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:54:18.523234 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:54:18.523090 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2d91fcc-f535-463e-8f6d-88d7892578b8-model-cache" pod "c2d91fcc-f535-463e-8f6d-88d7892578b8" (UID: "c2d91fcc-f535-463e-8f6d-88d7892578b8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:54:18.523234 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:54:18.523192 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2d91fcc-f535-463e-8f6d-88d7892578b8-tmp-dir" pod "c2d91fcc-f535-463e-8f6d-88d7892578b8" (UID: "c2d91fcc-f535-463e-8f6d-88d7892578b8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:54:18.523407 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:54:18.523359 5024 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c2d91fcc-f535-463e-8f6d-88d7892578b8-tmp-dir\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:54:18.523407 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:54:18.523374 5024 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c2d91fcc-f535-463e-8f6d-88d7892578b8-home\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:54:18.523407 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:54:18.523390 5024 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c2d91fcc-f535-463e-8f6d-88d7892578b8-model-cache\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:54:18.525135 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:54:18.525106 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2d91fcc-f535-463e-8f6d-88d7892578b8-kube-api-access-xmjhc" pod "c2d91fcc-f535-463e-8f6d-88d7892578b8" (UID: "c2d91fcc-f535-463e-8f6d-88d7892578b8"). InnerVolumeSpecName "kube-api-access-xmjhc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:54:18.525270 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:54:18.525255 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2d91fcc-f535-463e-8f6d-88d7892578b8-dshm" pod "c2d91fcc-f535-463e-8f6d-88d7892578b8" (UID: "c2d91fcc-f535-463e-8f6d-88d7892578b8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:54:18.525442 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:54:18.525423 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c2d91fcc-f535-463e-8f6d-88d7892578b8-tls-certs" pod "c2d91fcc-f535-463e-8f6d-88d7892578b8" (UID: "c2d91fcc-f535-463e-8f6d-88d7892578b8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 13:54:18.624721 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:54:18.624617 5024 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c2d91fcc-f535-463e-8f6d-88d7892578b8-dshm\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:54:18.624721 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:54:18.624664 5024 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c2d91fcc-f535-463e-8f6d-88d7892578b8-tls-certs\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:54:18.624721 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:54:18.624676 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xmjhc\" (UniqueName: \"kubernetes.io/projected/c2d91fcc-f535-463e-8f6d-88d7892578b8-kube-api-access-xmjhc\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:54:18.887475 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:54:18.887384 5024 generic.go:356] "Generic (PLEG): container finished" podID="c2d91fcc-f535-463e-8f6d-88d7892578b8" containerID="832f094d54747701790b64af01117f68352990e8bbf2fdbcace24ac8279fd2a3" exitCode=0 Nov 03 13:54:18.887475 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:54:18.887436 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-d2dab122-kserve-58bf7b4f87-jjs6b" Nov 03 13:54:18.887670 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:54:18.887474 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-d2dab122-kserve-58bf7b4f87-jjs6b" event={"ID":"c2d91fcc-f535-463e-8f6d-88d7892578b8","Type":"ContainerDied","Data":"832f094d54747701790b64af01117f68352990e8bbf2fdbcace24ac8279fd2a3"} Nov 03 13:54:18.887670 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:54:18.887522 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-d2dab122-kserve-58bf7b4f87-jjs6b" event={"ID":"c2d91fcc-f535-463e-8f6d-88d7892578b8","Type":"ContainerDied","Data":"863469f4eddcb8d596e304d2ef00511aa9bc391c1d5fc77822bf8bbb4072c969"} Nov 03 13:54:18.887670 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:54:18.887540 5024 scope.go:122] "RemoveContainer" containerID="832f094d54747701790b64af01117f68352990e8bbf2fdbcace24ac8279fd2a3" Nov 03 13:54:18.897839 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:54:18.897816 5024 scope.go:122] "RemoveContainer" containerID="832f094d54747701790b64af01117f68352990e8bbf2fdbcace24ac8279fd2a3" Nov 03 13:54:18.898099 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:54:18.898079 5024 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"832f094d54747701790b64af01117f68352990e8bbf2fdbcace24ac8279fd2a3\": container with ID starting with 832f094d54747701790b64af01117f68352990e8bbf2fdbcace24ac8279fd2a3 not found: ID does not exist" containerID="832f094d54747701790b64af01117f68352990e8bbf2fdbcace24ac8279fd2a3" Nov 03 13:54:18.898195 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:54:18.898108 5024 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"832f094d54747701790b64af01117f68352990e8bbf2fdbcace24ac8279fd2a3"} err="failed to get container status \"832f094d54747701790b64af01117f68352990e8bbf2fdbcace24ac8279fd2a3\": rpc error: code = NotFound desc = could not find container \"832f094d54747701790b64af01117f68352990e8bbf2fdbcace24ac8279fd2a3\": container with ID starting with 832f094d54747701790b64af01117f68352990e8bbf2fdbcace24ac8279fd2a3 not found: ID does not exist" Nov 03 13:54:18.911933 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:54:18.911909 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-prio-pool-d2dab122-kserve-58bf7b4f87-jjs6b"] Nov 03 13:54:18.915790 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:54:18.915766 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-prio-pool-d2dab122-kserve-58bf7b4f87-jjs6b"] Nov 03 13:54:19.355503 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:54:19.355445 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-prio-pool-d2dab122-kserve-58bf7b4f87-jjs6b" podUID="c2d91fcc-f535-463e-8f6d-88d7892578b8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.199:8000/ready\": context deadline exceeded" Nov 03 13:54:20.107109 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:54:20.107074 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c2d91fcc-f535-463e-8f6d-88d7892578b8" path="/var/lib/kubelet/pods/c2d91fcc-f535-463e-8f6d-88d7892578b8/volumes" Nov 03 13:54:26.288309 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:54:26.284948 5024 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-92vh9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.173:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 13:54:26.288309 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:54:26.285045 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-92vh9" podUID="825941b6-3ed3-47f2-a859-c891c4f29434" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.173:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 13:54:26.318328 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:54:26.317376 5024 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-zndxf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.172:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 13:54:26.318328 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:54:26.317472 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-zndxf" podUID="73f8f24f-b955-433b-95ed-5f441f7fceb2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.172:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 13:54:26.359313 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:54:26.355551 5024 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-k4c4j container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.174:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 13:54:26.359313 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:54:26.355624 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-k4c4j" podUID="4b70e366-6fba-4bfa-ac3e-0f0c7b0b64d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.174:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 13:54:27.077309 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:54:27.076482 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cb53c5df-v52x2"] Nov 03 13:54:27.086308 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:54:27.083815 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cb53c5df-v52x2"] Nov 03 13:54:28.109591 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:54:28.109547 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="413b4679-30fe-421c-a1ef-4b463cc1603e" path="/var/lib/kubelet/pods/413b4679-30fe-421c-a1ef-4b463cc1603e/volumes" Nov 03 13:54:28.338893 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:54:28.338851 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-prio-pool-d2dab122-kserve-router-scheduler-f5cfb7bb-b225h" podUID="b075d14b-76a6-4425-90e7-556220654cba" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 13:54:28.339056 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:54:28.338955 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-prio-pool-d2dab122-kserve-router-scheduler-f5cfb7bb-b225h" Nov 03 13:54:37.020673 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:54:37.020634 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-788f6f66-8ttzx"] Nov 03 13:54:37.023224 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:54:37.023201 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-788f6f66-8ttzx"] Nov 03 13:54:38.107363 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:54:38.107326 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="58aa17f4-75c7-410b-ab1f-e05fdf6c626d" path="/var/lib/kubelet/pods/58aa17f4-75c7-410b-ab1f-e05fdf6c626d/volumes" Nov 03 13:54:38.339054 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:54:38.339018 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-prio-pool-d2dab122-kserve-router-scheduler-f5cfb7bb-b225h" podUID="b075d14b-76a6-4425-90e7-556220654cba" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 13:54:39.508057 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:54:39.508015 5024 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d7mwz" containerName="extractor" Nov 03 13:54:45.891219 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:54:45.891182 5024 scope.go:122] "RemoveContainer" containerID="d804da81ae33b107c6cd37730c5c3b22374cf3e854c004000b04ba74cb257fdc" Nov 03 13:54:47.028118 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:54:47.028083 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5076e05a-jgdkf"] Nov 03 13:54:47.030585 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:54:47.030560 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5076e05a-jgdkf"] Nov 03 13:54:48.107455 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:54:48.107420 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4774b3fb-9cf8-4dda-a12c-89290fcf0ea4" path="/var/lib/kubelet/pods/4774b3fb-9cf8-4dda-a12c-89290fcf0ea4/volumes" Nov 03 13:54:48.406977 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:54:48.406949 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-d2dab122-kserve-router-scheduler-f5cfb7bb-b225h" Nov 03 13:54:48.496032 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:54:48.495999 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b075d14b-76a6-4425-90e7-556220654cba-tls-certs\" (UniqueName: \"kubernetes.io/secret/b075d14b-76a6-4425-90e7-556220654cba-tls-certs\") pod \"b075d14b-76a6-4425-90e7-556220654cba\" (UID: \"b075d14b-76a6-4425-90e7-556220654cba\") " Nov 03 13:54:48.496213 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:54:48.496140 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b075d14b-76a6-4425-90e7-556220654cba-kube-api-access-z8kd6\" (UniqueName: \"kubernetes.io/projected/b075d14b-76a6-4425-90e7-556220654cba-kube-api-access-z8kd6\") pod \"b075d14b-76a6-4425-90e7-556220654cba\" (UID: \"b075d14b-76a6-4425-90e7-556220654cba\") " Nov 03 13:54:48.498101 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:54:48.498074 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b075d14b-76a6-4425-90e7-556220654cba-tls-certs" pod "b075d14b-76a6-4425-90e7-556220654cba" (UID: "b075d14b-76a6-4425-90e7-556220654cba"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 13:54:48.498242 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:54:48.498226 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b075d14b-76a6-4425-90e7-556220654cba-kube-api-access-z8kd6" pod "b075d14b-76a6-4425-90e7-556220654cba" (UID: "b075d14b-76a6-4425-90e7-556220654cba"). InnerVolumeSpecName "kube-api-access-z8kd6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:54:48.597332 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:54:48.597272 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z8kd6\" (UniqueName: \"kubernetes.io/projected/b075d14b-76a6-4425-90e7-556220654cba-kube-api-access-z8kd6\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:54:48.597332 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:54:48.597333 5024 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b075d14b-76a6-4425-90e7-556220654cba-tls-certs\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:54:49.175682 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:54:49.175648 5024 generic.go:356] "Generic (PLEG): container finished" podID="b075d14b-76a6-4425-90e7-556220654cba" containerID="c0cb49c2c0617a26c5d15d3cbc63e31634d7dc68c64ca0144aee751337961ea2" exitCode=0 Nov 03 13:54:49.176143 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:54:49.175687 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-d2dab122-kserve-router-scheduler-f5cfb7bb-b225h" Nov 03 13:54:49.176143 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:54:49.175725 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-d2dab122-kserve-router-scheduler-f5cfb7bb-b225h" event={"ID":"b075d14b-76a6-4425-90e7-556220654cba","Type":"ContainerDied","Data":"c0cb49c2c0617a26c5d15d3cbc63e31634d7dc68c64ca0144aee751337961ea2"} Nov 03 13:54:49.176143 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:54:49.175777 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-d2dab122-kserve-router-scheduler-f5cfb7bb-b225h" event={"ID":"b075d14b-76a6-4425-90e7-556220654cba","Type":"ContainerDied","Data":"15f3913720e1202195572b8e38f3c096b8b7b835bf59246f329fa0b0e6c02acd"} Nov 03 13:54:49.176143 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:54:49.175799 5024 scope.go:122] "RemoveContainer" containerID="c0cb49c2c0617a26c5d15d3cbc63e31634d7dc68c64ca0144aee751337961ea2" Nov 03 13:54:49.194608 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:54:49.194582 5024 scope.go:122] "RemoveContainer" containerID="c0cb49c2c0617a26c5d15d3cbc63e31634d7dc68c64ca0144aee751337961ea2" Nov 03 13:54:49.194929 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:54:49.194901 5024 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c0cb49c2c0617a26c5d15d3cbc63e31634d7dc68c64ca0144aee751337961ea2\": container with ID starting with c0cb49c2c0617a26c5d15d3cbc63e31634d7dc68c64ca0144aee751337961ea2 not found: ID does not exist" containerID="c0cb49c2c0617a26c5d15d3cbc63e31634d7dc68c64ca0144aee751337961ea2" Nov 03 13:54:49.195037 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:54:49.194933 5024 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0cb49c2c0617a26c5d15d3cbc63e31634d7dc68c64ca0144aee751337961ea2"} err="failed to get container status \"c0cb49c2c0617a26c5d15d3cbc63e31634d7dc68c64ca0144aee751337961ea2\": rpc error: code = NotFound desc = could not find container \"c0cb49c2c0617a26c5d15d3cbc63e31634d7dc68c64ca0144aee751337961ea2\": container with ID starting with c0cb49c2c0617a26c5d15d3cbc63e31634d7dc68c64ca0144aee751337961ea2 not found: ID does not exist" Nov 03 13:54:49.203127 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:54:49.203094 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-prio-pool-d2dab122-kserve-router-scheduler-f5cfb7bb-b225h"] Nov 03 13:54:49.206223 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:54:49.206197 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-prio-pool-d2dab122-kserve-router-scheduler-f5cfb7bb-b225h"] Nov 03 13:54:49.338518 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:54:49.338467 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-prio-pool-d2dab122-kserve-router-scheduler-f5cfb7bb-b225h" podUID="b075d14b-76a6-4425-90e7-556220654cba" containerName="main" probeResult="failure" output="timeout: failed to connect service \"10.132.0.200:9003\" within 1s: context deadline exceeded" Nov 03 13:54:49.338709 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:54:49.338562 5024 logging.go:55] [core] [Channel #5162 SubChannel #5163]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.200:9003", ServerName: "10.132.0.200:9003", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.200:9003: operation was canceled" Nov 03 13:54:50.107333 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:54:50.107280 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b075d14b-76a6-4425-90e7-556220654cba" path="/var/lib/kubelet/pods/b075d14b-76a6-4425-90e7-556220654cba/volumes" Nov 03 13:54:55.662767 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:54:55.662733 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-fg58r"] Nov 03 13:54:55.663251 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:54:55.663147 5024 cpu_manager.go:414] "containerMap: removing container" podUID="c2d91fcc-f535-463e-8f6d-88d7892578b8" containerName="main" Nov 03 13:54:55.663251 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:54:55.663160 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2d91fcc-f535-463e-8f6d-88d7892578b8" containerName="main" Nov 03 13:54:55.663251 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:54:55.663170 5024 cpu_manager.go:414] "containerMap: removing container" podUID="b075d14b-76a6-4425-90e7-556220654cba" containerName="main" Nov 03 13:54:55.663251 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:54:55.663193 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b075d14b-76a6-4425-90e7-556220654cba" containerName="main" Nov 03 13:54:55.663428 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:54:55.663274 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2d91fcc-f535-463e-8f6d-88d7892578b8" containerName="main" Nov 03 13:54:55.663428 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:54:55.663298 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="b075d14b-76a6-4425-90e7-556220654cba" containerName="main" Nov 03 13:54:55.666435 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:54:55.666413 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-fg58r" Nov 03 13:54:55.673348 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:54:55.673324 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 13:54:55.673506 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:54:55.673322 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 13:54:55.673580 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:54:55.673441 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 13:54:55.673580 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:54:55.673529 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w6vqv\"" Nov 03 13:54:55.675181 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:54:55.675157 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-fg58r"] Nov 03 13:54:55.761774 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:54:55.761736 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/796ad0b0-d02e-46b7-8018-c252134da079-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-fg58r\" (UID: \"796ad0b0-d02e-46b7-8018-c252134da079\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-fg58r" Nov 03 13:54:55.761942 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:54:55.761785 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ks46q\" (UniqueName: \"kubernetes.io/projected/796ad0b0-d02e-46b7-8018-c252134da079-kube-api-access-ks46q\") pod \"maas-api-revoke-sub-452b6d7b-fg58r\" (UID: \"796ad0b0-d02e-46b7-8018-c252134da079\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-fg58r" Nov 03 13:54:55.761992 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:54:55.761959 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/796ad0b0-d02e-46b7-8018-c252134da079-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-fg58r\" (UID: \"796ad0b0-d02e-46b7-8018-c252134da079\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-fg58r" Nov 03 13:54:55.862767 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:54:55.862725 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/796ad0b0-d02e-46b7-8018-c252134da079-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-fg58r\" (UID: \"796ad0b0-d02e-46b7-8018-c252134da079\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-fg58r" Nov 03 13:54:55.862945 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:54:55.862789 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/796ad0b0-d02e-46b7-8018-c252134da079-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-fg58r\" (UID: \"796ad0b0-d02e-46b7-8018-c252134da079\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-fg58r" Nov 03 13:54:55.862945 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:54:55.862810 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ks46q\" (UniqueName: \"kubernetes.io/projected/796ad0b0-d02e-46b7-8018-c252134da079-kube-api-access-ks46q\") pod \"maas-api-revoke-sub-452b6d7b-fg58r\" (UID: \"796ad0b0-d02e-46b7-8018-c252134da079\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-fg58r" Nov 03 13:54:55.863503 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:54:55.863478 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/796ad0b0-d02e-46b7-8018-c252134da079-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-fg58r\" (UID: \"796ad0b0-d02e-46b7-8018-c252134da079\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-fg58r" Nov 03 13:54:55.865143 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:54:55.865115 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/796ad0b0-d02e-46b7-8018-c252134da079-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-fg58r\" (UID: \"796ad0b0-d02e-46b7-8018-c252134da079\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-fg58r" Nov 03 13:54:55.873230 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:54:55.873203 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ks46q\" (UniqueName: \"kubernetes.io/projected/796ad0b0-d02e-46b7-8018-c252134da079-kube-api-access-ks46q\") pod \"maas-api-revoke-sub-452b6d7b-fg58r\" (UID: \"796ad0b0-d02e-46b7-8018-c252134da079\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-fg58r" Nov 03 13:54:55.981218 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:54:55.981135 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-fg58r" Nov 03 13:54:56.261779 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:54:56.261747 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-fg58r"] Nov 03 13:54:56.262888 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:54:56.262862 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod796ad0b0_d02e_46b7_8018_c252134da079.slice/crio-9f08657cbfbe3fc324adb800b7517af64a3b18070487a9e5bbafe11b6e698929 WatchSource:0}: Error finding container 9f08657cbfbe3fc324adb800b7517af64a3b18070487a9e5bbafe11b6e698929: Status 404 returned error can't find the container with id 9f08657cbfbe3fc324adb800b7517af64a3b18070487a9e5bbafe11b6e698929 Nov 03 13:54:57.224045 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:54:57.223999 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-fg58r" event={"ID":"796ad0b0-d02e-46b7-8018-c252134da079","Type":"ContainerStarted","Data":"feeb92acc04af50276d3081cea3f29cb25e9911fc682a9b78cec8021952e4947"} Nov 03 13:54:57.224625 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:54:57.224072 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-fg58r" event={"ID":"796ad0b0-d02e-46b7-8018-c252134da079","Type":"ContainerStarted","Data":"9f08657cbfbe3fc324adb800b7517af64a3b18070487a9e5bbafe11b6e698929"} Nov 03 13:54:57.243658 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:54:57.243564 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-fg58r" podStartSLOduration=2.243544989 podStartE2EDuration="2.243544989s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:54:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:54:57.243169563 +0000 UTC m=+2777.251360841" watchObservedRunningTime="2026-11-03 13:54:57.243544989 +0000 UTC m=+2777.251736267" Nov 03 13:54:58.029645 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:54:58.029602 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fbfb876e-gv9v5"] Nov 03 13:54:58.052256 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:54:58.052217 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fbfb876e-gv9v5"] Nov 03 13:54:58.141075 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:54:58.140637 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af1113d7-cd0a-439b-a323-7f9ca5eb6723" path="/var/lib/kubelet/pods/af1113d7-cd0a-439b-a323-7f9ca5eb6723/volumes" Nov 03 13:54:58.255308 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:54:58.252593 5024 generic.go:356] "Generic (PLEG): container finished" podID="796ad0b0-d02e-46b7-8018-c252134da079" containerID="feeb92acc04af50276d3081cea3f29cb25e9911fc682a9b78cec8021952e4947" exitCode=0 Nov 03 13:54:58.255308 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:54:58.252751 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-fg58r" event={"ID":"796ad0b0-d02e-46b7-8018-c252134da079","Type":"ContainerDied","Data":"feeb92acc04af50276d3081cea3f29cb25e9911fc682a9b78cec8021952e4947"} Nov 03 13:55:00.062240 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:55:00.061933 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-fg58r" Nov 03 13:55:00.142664 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:55:00.142623 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/796ad0b0-d02e-46b7-8018-c252134da079-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/796ad0b0-d02e-46b7-8018-c252134da079-maas-api-xks-ca\") pod \"796ad0b0-d02e-46b7-8018-c252134da079\" (UID: \"796ad0b0-d02e-46b7-8018-c252134da079\") " Nov 03 13:55:00.240958 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:55:00.240670 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/796ad0b0-d02e-46b7-8018-c252134da079-maas-api-xks-ca" pod "796ad0b0-d02e-46b7-8018-c252134da079" (UID: "796ad0b0-d02e-46b7-8018-c252134da079"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 13:55:00.241345 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:55:00.241322 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/796ad0b0-d02e-46b7-8018-c252134da079-kube-api-access-ks46q\" (UniqueName: \"kubernetes.io/projected/796ad0b0-d02e-46b7-8018-c252134da079-kube-api-access-ks46q\") pod \"796ad0b0-d02e-46b7-8018-c252134da079\" (UID: \"796ad0b0-d02e-46b7-8018-c252134da079\") " Nov 03 13:55:00.242025 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:55:00.242003 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/796ad0b0-d02e-46b7-8018-c252134da079-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/796ad0b0-d02e-46b7-8018-c252134da079-maas-api-openshift-ca\") pod \"796ad0b0-d02e-46b7-8018-c252134da079\" (UID: \"796ad0b0-d02e-46b7-8018-c252134da079\") " Nov 03 13:55:00.243726 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:55:00.243699 5024 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/796ad0b0-d02e-46b7-8018-c252134da079-maas-api-xks-ca\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:55:00.243877 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:55:00.242546 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/796ad0b0-d02e-46b7-8018-c252134da079-maas-api-openshift-ca" pod "796ad0b0-d02e-46b7-8018-c252134da079" (UID: "796ad0b0-d02e-46b7-8018-c252134da079"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:55:00.244695 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:55:00.244672 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/796ad0b0-d02e-46b7-8018-c252134da079-kube-api-access-ks46q" pod "796ad0b0-d02e-46b7-8018-c252134da079" (UID: "796ad0b0-d02e-46b7-8018-c252134da079"). InnerVolumeSpecName "kube-api-access-ks46q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:55:00.347106 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:55:00.347027 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ks46q\" (UniqueName: \"kubernetes.io/projected/796ad0b0-d02e-46b7-8018-c252134da079-kube-api-access-ks46q\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:55:00.347520 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:55:00.347498 5024 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/796ad0b0-d02e-46b7-8018-c252134da079-maas-api-openshift-ca\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:55:00.378708 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:55:00.378671 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-fg58r" event={"ID":"796ad0b0-d02e-46b7-8018-c252134da079","Type":"ContainerDied","Data":"9f08657cbfbe3fc324adb800b7517af64a3b18070487a9e5bbafe11b6e698929"} Nov 03 13:55:00.378945 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:55:00.378931 5024 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9f08657cbfbe3fc324adb800b7517af64a3b18070487a9e5bbafe11b6e698929" Nov 03 13:55:00.379091 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:55:00.379080 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-fg58r" Nov 03 13:55:08.023158 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:55:08.023119 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e0f27240-9pkw2"] Nov 03 13:55:08.026482 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:55:08.026456 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e0f27240-9pkw2"] Nov 03 13:55:08.107760 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:55:08.107728 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c3ae5602-f8f2-42a7-bbb9-a878e9a984ab" path="/var/lib/kubelet/pods/c3ae5602-f8f2-42a7-bbb9-a878e9a984ab/volumes" Nov 03 13:55:09.077266 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:55:09.077208 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-xqg8w"] Nov 03 13:55:09.078208 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:55:09.078188 5024 cpu_manager.go:414] "containerMap: removing container" podUID="796ad0b0-d02e-46b7-8018-c252134da079" containerName="revoke-keys" Nov 03 13:55:09.078325 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:55:09.078312 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="796ad0b0-d02e-46b7-8018-c252134da079" containerName="revoke-keys" Nov 03 13:55:09.078522 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:55:09.078513 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="796ad0b0-d02e-46b7-8018-c252134da079" containerName="revoke-keys" Nov 03 13:55:09.082435 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:55:09.082417 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-xqg8w" Nov 03 13:55:09.088114 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:55:09.088091 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-xqg8w"] Nov 03 13:55:09.088327 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:55:09.088128 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 13:55:09.088421 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:55:09.088166 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 13:55:09.088477 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:55:09.088217 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 13:55:09.106310 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:55:09.103489 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w6vqv\"" Nov 03 13:55:09.236738 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:55:09.236683 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b7mzk\" (UniqueName: \"kubernetes.io/projected/2a6ad3d1-e739-4ace-b7f6-13d3fefdb87d-kube-api-access-b7mzk\") pod \"maas-api-revoke-sub-3e31a506-xqg8w\" (UID: \"2a6ad3d1-e739-4ace-b7f6-13d3fefdb87d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-xqg8w" Nov 03 13:55:09.236947 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:55:09.236779 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a6ad3d1-e739-4ace-b7f6-13d3fefdb87d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-xqg8w\" (UID: \"2a6ad3d1-e739-4ace-b7f6-13d3fefdb87d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-xqg8w" Nov 03 13:55:09.236947 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:55:09.236852 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a6ad3d1-e739-4ace-b7f6-13d3fefdb87d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-xqg8w\" (UID: \"2a6ad3d1-e739-4ace-b7f6-13d3fefdb87d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-xqg8w" Nov 03 13:55:09.338066 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:55:09.337967 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7mzk\" (UniqueName: \"kubernetes.io/projected/2a6ad3d1-e739-4ace-b7f6-13d3fefdb87d-kube-api-access-b7mzk\") pod \"maas-api-revoke-sub-3e31a506-xqg8w\" (UID: \"2a6ad3d1-e739-4ace-b7f6-13d3fefdb87d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-xqg8w" Nov 03 13:55:09.338066 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:55:09.338036 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a6ad3d1-e739-4ace-b7f6-13d3fefdb87d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-xqg8w\" (UID: \"2a6ad3d1-e739-4ace-b7f6-13d3fefdb87d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-xqg8w" Nov 03 13:55:09.338369 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:55:09.338074 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a6ad3d1-e739-4ace-b7f6-13d3fefdb87d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-xqg8w\" (UID: \"2a6ad3d1-e739-4ace-b7f6-13d3fefdb87d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-xqg8w" Nov 03 13:55:09.339154 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:55:09.339121 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a6ad3d1-e739-4ace-b7f6-13d3fefdb87d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-xqg8w\" (UID: \"2a6ad3d1-e739-4ace-b7f6-13d3fefdb87d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-xqg8w" Nov 03 13:55:09.340568 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:55:09.340543 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a6ad3d1-e739-4ace-b7f6-13d3fefdb87d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-xqg8w\" (UID: \"2a6ad3d1-e739-4ace-b7f6-13d3fefdb87d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-xqg8w" Nov 03 13:55:09.347273 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:55:09.347249 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7mzk\" (UniqueName: \"kubernetes.io/projected/2a6ad3d1-e739-4ace-b7f6-13d3fefdb87d-kube-api-access-b7mzk\") pod \"maas-api-revoke-sub-3e31a506-xqg8w\" (UID: \"2a6ad3d1-e739-4ace-b7f6-13d3fefdb87d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-xqg8w" Nov 03 13:55:09.425577 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:55:09.425302 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-xqg8w" Nov 03 13:55:09.583684 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:55:09.583589 5024 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d7mwz" containerName="extractor" Nov 03 13:55:10.052310 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:55:10.033244 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-xqg8w"] Nov 03 13:55:10.479144 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:55:10.479047 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-xqg8w" event={"ID":"2a6ad3d1-e739-4ace-b7f6-13d3fefdb87d","Type":"ContainerStarted","Data":"b8c9b87433a0a6ae40541c965dd3261769de67b62c6d9db6aad7261b2ae23178"} Nov 03 13:55:11.307778 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:55:11.307384 5024 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-zndxf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.172:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 13:55:11.307778 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:55:11.307454 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-zndxf" podUID="73f8f24f-b955-433b-95ed-5f441f7fceb2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.172:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 13:55:11.343328 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:55:11.340401 5024 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-k4c4j container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.174:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 13:55:11.343328 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:55:11.340467 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-k4c4j" podUID="4b70e366-6fba-4bfa-ac3e-0f0c7b0b64d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.174:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 13:55:11.505305 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:55:11.505186 5024 generic.go:356] "Generic (PLEG): container finished" podID="2a6ad3d1-e739-4ace-b7f6-13d3fefdb87d" containerID="7f259cd499cb40c776274d9bdd4cad3413a65faa7492b4049015b27fb551e0ea" exitCode=0 Nov 03 13:55:11.518310 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:55:11.514895 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-xqg8w" event={"ID":"2a6ad3d1-e739-4ace-b7f6-13d3fefdb87d","Type":"ContainerDied","Data":"7f259cd499cb40c776274d9bdd4cad3413a65faa7492b4049015b27fb551e0ea"} Nov 03 13:55:12.727716 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:55:12.727673 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-xqg8w" Nov 03 13:55:12.782098 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:55:12.782001 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2a6ad3d1-e739-4ace-b7f6-13d3fefdb87d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a6ad3d1-e739-4ace-b7f6-13d3fefdb87d-maas-api-xks-ca\") pod \"2a6ad3d1-e739-4ace-b7f6-13d3fefdb87d\" (UID: \"2a6ad3d1-e739-4ace-b7f6-13d3fefdb87d\") " Nov 03 13:55:12.782098 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:55:12.782084 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a6ad3d1-e739-4ace-b7f6-13d3fefdb87d-kube-api-access-b7mzk\" (UniqueName: \"kubernetes.io/projected/2a6ad3d1-e739-4ace-b7f6-13d3fefdb87d-kube-api-access-b7mzk\") pod \"2a6ad3d1-e739-4ace-b7f6-13d3fefdb87d\" (UID: \"2a6ad3d1-e739-4ace-b7f6-13d3fefdb87d\") " Nov 03 13:55:12.782393 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:55:12.782142 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a6ad3d1-e739-4ace-b7f6-13d3fefdb87d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a6ad3d1-e739-4ace-b7f6-13d3fefdb87d-maas-api-openshift-ca\") pod \"2a6ad3d1-e739-4ace-b7f6-13d3fefdb87d\" (UID: \"2a6ad3d1-e739-4ace-b7f6-13d3fefdb87d\") " Nov 03 13:55:12.783267 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:55:12.783238 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a6ad3d1-e739-4ace-b7f6-13d3fefdb87d-maas-api-openshift-ca" pod "2a6ad3d1-e739-4ace-b7f6-13d3fefdb87d" (UID: "2a6ad3d1-e739-4ace-b7f6-13d3fefdb87d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:55:12.785398 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:55:12.785247 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a6ad3d1-e739-4ace-b7f6-13d3fefdb87d-kube-api-access-b7mzk" pod "2a6ad3d1-e739-4ace-b7f6-13d3fefdb87d" (UID: "2a6ad3d1-e739-4ace-b7f6-13d3fefdb87d"). InnerVolumeSpecName "kube-api-access-b7mzk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:55:12.785398 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:55:12.785370 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a6ad3d1-e739-4ace-b7f6-13d3fefdb87d-maas-api-xks-ca" pod "2a6ad3d1-e739-4ace-b7f6-13d3fefdb87d" (UID: "2a6ad3d1-e739-4ace-b7f6-13d3fefdb87d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 13:55:12.883318 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:55:12.883255 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b7mzk\" (UniqueName: \"kubernetes.io/projected/2a6ad3d1-e739-4ace-b7f6-13d3fefdb87d-kube-api-access-b7mzk\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:55:12.883318 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:55:12.883321 5024 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a6ad3d1-e739-4ace-b7f6-13d3fefdb87d-maas-api-openshift-ca\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:55:12.883567 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:55:12.883340 5024 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a6ad3d1-e739-4ace-b7f6-13d3fefdb87d-maas-api-xks-ca\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:55:13.587312 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:55:13.585573 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-xqg8w" event={"ID":"2a6ad3d1-e739-4ace-b7f6-13d3fefdb87d","Type":"ContainerDied","Data":"b8c9b87433a0a6ae40541c965dd3261769de67b62c6d9db6aad7261b2ae23178"} Nov 03 13:55:13.587312 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:55:13.585654 5024 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b8c9b87433a0a6ae40541c965dd3261769de67b62c6d9db6aad7261b2ae23178" Nov 03 13:55:13.587312 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:55:13.585743 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-xqg8w" Nov 03 13:55:18.024982 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:55:18.024938 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c4a5d20d-vtczd"] Nov 03 13:55:18.028672 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:55:18.028645 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c4a5d20d-vtczd"] Nov 03 13:55:18.107607 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:55:18.107571 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e0412dc2-841f-4005-a841-128786fce7be" path="/var/lib/kubelet/pods/e0412dc2-841f-4005-a841-128786fce7be/volumes" Nov 03 13:55:27.248690 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:55:27.248650 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3740ae17-khkcv"] Nov 03 13:55:27.249195 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:55:27.249063 5024 cpu_manager.go:414] "containerMap: removing container" podUID="2a6ad3d1-e739-4ace-b7f6-13d3fefdb87d" containerName="revoke-keys" Nov 03 13:55:27.249195 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:55:27.249074 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a6ad3d1-e739-4ace-b7f6-13d3fefdb87d" containerName="revoke-keys" Nov 03 13:55:27.249195 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:55:27.249145 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a6ad3d1-e739-4ace-b7f6-13d3fefdb87d" containerName="revoke-keys" Nov 03 13:55:27.252382 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:55:27.252361 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3740ae17-khkcv" Nov 03 13:55:27.259761 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:55:27.259656 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 13:55:27.259761 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:55:27.259656 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 13:55:27.259761 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:55:27.259680 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 13:55:27.260401 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:55:27.260381 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w6vqv\"" Nov 03 13:55:27.262308 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:55:27.262273 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3740ae17-khkcv"] Nov 03 13:55:27.350969 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:55:27.350935 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2231d96c-22b1-410c-b663-d5f015bf7b43-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3740ae17-khkcv\" (UID: \"2231d96c-22b1-410c-b663-d5f015bf7b43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3740ae17-khkcv" Nov 03 13:55:27.351138 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:55:27.351079 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rb88s\" (UniqueName: \"kubernetes.io/projected/2231d96c-22b1-410c-b663-d5f015bf7b43-kube-api-access-rb88s\") pod \"maas-api-revoke-sub-3740ae17-khkcv\" (UID: \"2231d96c-22b1-410c-b663-d5f015bf7b43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3740ae17-khkcv" Nov 03 13:55:27.351182 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:55:27.351135 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2231d96c-22b1-410c-b663-d5f015bf7b43-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3740ae17-khkcv\" (UID: \"2231d96c-22b1-410c-b663-d5f015bf7b43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3740ae17-khkcv" Nov 03 13:55:27.452155 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:55:27.452120 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2231d96c-22b1-410c-b663-d5f015bf7b43-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3740ae17-khkcv\" (UID: \"2231d96c-22b1-410c-b663-d5f015bf7b43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3740ae17-khkcv" Nov 03 13:55:27.452398 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:55:27.452202 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rb88s\" (UniqueName: \"kubernetes.io/projected/2231d96c-22b1-410c-b663-d5f015bf7b43-kube-api-access-rb88s\") pod \"maas-api-revoke-sub-3740ae17-khkcv\" (UID: \"2231d96c-22b1-410c-b663-d5f015bf7b43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3740ae17-khkcv" Nov 03 13:55:27.452398 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:55:27.452225 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2231d96c-22b1-410c-b663-d5f015bf7b43-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3740ae17-khkcv\" (UID: \"2231d96c-22b1-410c-b663-d5f015bf7b43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3740ae17-khkcv" Nov 03 13:55:27.452919 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:55:27.452893 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2231d96c-22b1-410c-b663-d5f015bf7b43-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3740ae17-khkcv\" (UID: \"2231d96c-22b1-410c-b663-d5f015bf7b43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3740ae17-khkcv" Nov 03 13:55:27.454686 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:55:27.454662 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2231d96c-22b1-410c-b663-d5f015bf7b43-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3740ae17-khkcv\" (UID: \"2231d96c-22b1-410c-b663-d5f015bf7b43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3740ae17-khkcv" Nov 03 13:55:27.461113 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:55:27.461091 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rb88s\" (UniqueName: \"kubernetes.io/projected/2231d96c-22b1-410c-b663-d5f015bf7b43-kube-api-access-rb88s\") pod \"maas-api-revoke-sub-3740ae17-khkcv\" (UID: \"2231d96c-22b1-410c-b663-d5f015bf7b43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3740ae17-khkcv" Nov 03 13:55:27.570034 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:55:27.569956 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3740ae17-khkcv" Nov 03 13:55:27.850227 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:55:27.850187 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3740ae17-khkcv"] Nov 03 13:55:27.851809 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:55:27.851779 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2231d96c_22b1_410c_b663_d5f015bf7b43.slice/crio-6a2e1497fdee0473aad216a9266b4787d7e11c47349be0e0765ae04c92a958a6 WatchSource:0}: Error finding container 6a2e1497fdee0473aad216a9266b4787d7e11c47349be0e0765ae04c92a958a6: Status 404 returned error can't find the container with id 6a2e1497fdee0473aad216a9266b4787d7e11c47349be0e0765ae04c92a958a6 Nov 03 13:55:28.026927 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:55:28.024917 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9bf1edb6-pdnkf"] Nov 03 13:55:28.031786 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:55:28.031748 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9bf1edb6-pdnkf"] Nov 03 13:55:28.107638 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:55:28.107607 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c179e407-4918-4bdb-9f6b-3ccdd219f40c" path="/var/lib/kubelet/pods/c179e407-4918-4bdb-9f6b-3ccdd219f40c/volumes" Nov 03 13:55:28.786209 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:55:28.786177 5024 generic.go:356] "Generic (PLEG): container finished" podID="2231d96c-22b1-410c-b663-d5f015bf7b43" containerID="b67487554ef8f62eff8374c3e2fba92a4f6f06bf233732917661e3989e571f06" exitCode=0 Nov 03 13:55:28.786613 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:55:28.786229 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3740ae17-khkcv" event={"ID":"2231d96c-22b1-410c-b663-d5f015bf7b43","Type":"ContainerDied","Data":"b67487554ef8f62eff8374c3e2fba92a4f6f06bf233732917661e3989e571f06"} Nov 03 13:55:28.786613 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:55:28.786266 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3740ae17-khkcv" event={"ID":"2231d96c-22b1-410c-b663-d5f015bf7b43","Type":"ContainerStarted","Data":"6a2e1497fdee0473aad216a9266b4787d7e11c47349be0e0765ae04c92a958a6"} Nov 03 13:55:29.923531 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:55:29.923506 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3740ae17-khkcv" Nov 03 13:55:29.975755 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:55:29.975714 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2231d96c-22b1-410c-b663-d5f015bf7b43-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2231d96c-22b1-410c-b663-d5f015bf7b43-maas-api-openshift-ca\") pod \"2231d96c-22b1-410c-b663-d5f015bf7b43\" (UID: \"2231d96c-22b1-410c-b663-d5f015bf7b43\") " Nov 03 13:55:29.975960 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:55:29.975857 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2231d96c-22b1-410c-b663-d5f015bf7b43-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2231d96c-22b1-410c-b663-d5f015bf7b43-maas-api-xks-ca\") pod \"2231d96c-22b1-410c-b663-d5f015bf7b43\" (UID: \"2231d96c-22b1-410c-b663-d5f015bf7b43\") " Nov 03 13:55:29.975960 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:55:29.975903 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2231d96c-22b1-410c-b663-d5f015bf7b43-kube-api-access-rb88s\" (UniqueName: \"kubernetes.io/projected/2231d96c-22b1-410c-b663-d5f015bf7b43-kube-api-access-rb88s\") pod \"2231d96c-22b1-410c-b663-d5f015bf7b43\" (UID: \"2231d96c-22b1-410c-b663-d5f015bf7b43\") " Nov 03 13:55:29.976104 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:55:29.976078 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2231d96c-22b1-410c-b663-d5f015bf7b43-maas-api-openshift-ca" pod "2231d96c-22b1-410c-b663-d5f015bf7b43" (UID: "2231d96c-22b1-410c-b663-d5f015bf7b43"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:55:29.976233 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:55:29.976219 5024 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2231d96c-22b1-410c-b663-d5f015bf7b43-maas-api-openshift-ca\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:55:29.978067 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:55:29.978034 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2231d96c-22b1-410c-b663-d5f015bf7b43-maas-api-xks-ca" pod "2231d96c-22b1-410c-b663-d5f015bf7b43" (UID: "2231d96c-22b1-410c-b663-d5f015bf7b43"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 13:55:29.978067 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:55:29.978044 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2231d96c-22b1-410c-b663-d5f015bf7b43-kube-api-access-rb88s" pod "2231d96c-22b1-410c-b663-d5f015bf7b43" (UID: "2231d96c-22b1-410c-b663-d5f015bf7b43"). InnerVolumeSpecName "kube-api-access-rb88s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:55:30.077029 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:55:30.076995 5024 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2231d96c-22b1-410c-b663-d5f015bf7b43-maas-api-xks-ca\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:55:30.077029 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:55:30.077025 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rb88s\" (UniqueName: \"kubernetes.io/projected/2231d96c-22b1-410c-b663-d5f015bf7b43-kube-api-access-rb88s\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:55:30.797927 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:55:30.797897 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3740ae17-khkcv" Nov 03 13:55:30.797927 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:55:30.797920 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3740ae17-khkcv" event={"ID":"2231d96c-22b1-410c-b663-d5f015bf7b43","Type":"ContainerDied","Data":"6a2e1497fdee0473aad216a9266b4787d7e11c47349be0e0765ae04c92a958a6"} Nov 03 13:55:30.798147 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:55:30.797963 5024 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6a2e1497fdee0473aad216a9266b4787d7e11c47349be0e0765ae04c92a958a6" Nov 03 13:55:38.022204 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:55:38.022098 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d31f7f2a-nv2ng"] Nov 03 13:55:38.025918 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:55:38.025889 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d31f7f2a-nv2ng"] Nov 03 13:55:38.107537 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:55:38.107504 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4983c732-1ead-4068-9bae-511cfd3ae7c3" path="/var/lib/kubelet/pods/4983c732-1ead-4068-9bae-511cfd3ae7c3/volumes" Nov 03 13:55:39.076917 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:55:39.076883 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-d0441c-openshift-default-d67bc796c-57tvj"] Nov 03 13:55:39.077413 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:55:39.077355 5024 cpu_manager.go:414] "containerMap: removing container" podUID="2231d96c-22b1-410c-b663-d5f015bf7b43" containerName="revoke-keys" Nov 03 13:55:39.077413 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:55:39.077367 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2231d96c-22b1-410c-b663-d5f015bf7b43" containerName="revoke-keys" Nov 03 13:55:39.077539 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:55:39.077437 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="2231d96c-22b1-410c-b663-d5f015bf7b43" containerName="revoke-keys" Nov 03 13:55:39.081519 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:55:39.081496 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-d0441c-openshift-default-d67bc796c-57tvj" Nov 03 13:55:39.091860 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:55:39.091834 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-d0441c-openshift-default-dockercfg-9q5rr\"" Nov 03 13:55:39.094336 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:55:39.094275 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-d0441c-openshift-default-d67bc796c-57tvj"] Nov 03 13:55:39.165714 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:55:39.165679 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2019d8e6-12bd-40f8-b387-06fb66b66a08-istio-data\") pod \"e2e-ipp-a-d0441c-openshift-default-d67bc796c-57tvj\" (UID: \"2019d8e6-12bd-40f8-b387-06fb66b66a08\") " pod="openshift-ingress/e2e-ipp-a-d0441c-openshift-default-d67bc796c-57tvj" Nov 03 13:55:39.165714 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:55:39.165713 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2019d8e6-12bd-40f8-b387-06fb66b66a08-credential-socket\") pod \"e2e-ipp-a-d0441c-openshift-default-d67bc796c-57tvj\" (UID: \"2019d8e6-12bd-40f8-b387-06fb66b66a08\") " pod="openshift-ingress/e2e-ipp-a-d0441c-openshift-default-d67bc796c-57tvj" Nov 03 13:55:39.165924 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:55:39.165740 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2019d8e6-12bd-40f8-b387-06fb66b66a08-workload-certs\") pod \"e2e-ipp-a-d0441c-openshift-default-d67bc796c-57tvj\" (UID: \"2019d8e6-12bd-40f8-b387-06fb66b66a08\") " pod="openshift-ingress/e2e-ipp-a-d0441c-openshift-default-d67bc796c-57tvj" Nov 03 13:55:39.165924 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:55:39.165830 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2019d8e6-12bd-40f8-b387-06fb66b66a08-istiod-ca-cert\") pod \"e2e-ipp-a-d0441c-openshift-default-d67bc796c-57tvj\" (UID: \"2019d8e6-12bd-40f8-b387-06fb66b66a08\") " pod="openshift-ingress/e2e-ipp-a-d0441c-openshift-default-d67bc796c-57tvj" Nov 03 13:55:39.166009 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:55:39.165991 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmsf6\" (UniqueName: \"kubernetes.io/projected/2019d8e6-12bd-40f8-b387-06fb66b66a08-kube-api-access-wmsf6\") pod \"e2e-ipp-a-d0441c-openshift-default-d67bc796c-57tvj\" (UID: \"2019d8e6-12bd-40f8-b387-06fb66b66a08\") " pod="openshift-ingress/e2e-ipp-a-d0441c-openshift-default-d67bc796c-57tvj" Nov 03 13:55:39.166571 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:55:39.166029 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2019d8e6-12bd-40f8-b387-06fb66b66a08-istio-token\") pod \"e2e-ipp-a-d0441c-openshift-default-d67bc796c-57tvj\" (UID: \"2019d8e6-12bd-40f8-b387-06fb66b66a08\") " pod="openshift-ingress/e2e-ipp-a-d0441c-openshift-default-d67bc796c-57tvj" Nov 03 13:55:39.166810 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:55:39.166781 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2019d8e6-12bd-40f8-b387-06fb66b66a08-istio-envoy\") pod \"e2e-ipp-a-d0441c-openshift-default-d67bc796c-57tvj\" (UID: \"2019d8e6-12bd-40f8-b387-06fb66b66a08\") " pod="openshift-ingress/e2e-ipp-a-d0441c-openshift-default-d67bc796c-57tvj" Nov 03 13:55:39.167830 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:55:39.166866 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2019d8e6-12bd-40f8-b387-06fb66b66a08-workload-socket\") pod \"e2e-ipp-a-d0441c-openshift-default-d67bc796c-57tvj\" (UID: \"2019d8e6-12bd-40f8-b387-06fb66b66a08\") " pod="openshift-ingress/e2e-ipp-a-d0441c-openshift-default-d67bc796c-57tvj" Nov 03 13:55:39.167830 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:55:39.166927 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2019d8e6-12bd-40f8-b387-06fb66b66a08-istio-podinfo\") pod \"e2e-ipp-a-d0441c-openshift-default-d67bc796c-57tvj\" (UID: \"2019d8e6-12bd-40f8-b387-06fb66b66a08\") " pod="openshift-ingress/e2e-ipp-a-d0441c-openshift-default-d67bc796c-57tvj" Nov 03 13:55:39.268072 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:55:39.268029 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2019d8e6-12bd-40f8-b387-06fb66b66a08-istio-data\") pod \"e2e-ipp-a-d0441c-openshift-default-d67bc796c-57tvj\" (UID: \"2019d8e6-12bd-40f8-b387-06fb66b66a08\") " pod="openshift-ingress/e2e-ipp-a-d0441c-openshift-default-d67bc796c-57tvj" Nov 03 13:55:39.268072 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:55:39.268073 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2019d8e6-12bd-40f8-b387-06fb66b66a08-credential-socket\") pod \"e2e-ipp-a-d0441c-openshift-default-d67bc796c-57tvj\" (UID: \"2019d8e6-12bd-40f8-b387-06fb66b66a08\") " pod="openshift-ingress/e2e-ipp-a-d0441c-openshift-default-d67bc796c-57tvj" Nov 03 13:55:39.268374 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:55:39.268100 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2019d8e6-12bd-40f8-b387-06fb66b66a08-workload-certs\") pod \"e2e-ipp-a-d0441c-openshift-default-d67bc796c-57tvj\" (UID: \"2019d8e6-12bd-40f8-b387-06fb66b66a08\") " pod="openshift-ingress/e2e-ipp-a-d0441c-openshift-default-d67bc796c-57tvj" Nov 03 13:55:39.268374 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:55:39.268132 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2019d8e6-12bd-40f8-b387-06fb66b66a08-istiod-ca-cert\") pod \"e2e-ipp-a-d0441c-openshift-default-d67bc796c-57tvj\" (UID: \"2019d8e6-12bd-40f8-b387-06fb66b66a08\") " pod="openshift-ingress/e2e-ipp-a-d0441c-openshift-default-d67bc796c-57tvj" Nov 03 13:55:39.268374 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:55:39.268174 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmsf6\" (UniqueName: \"kubernetes.io/projected/2019d8e6-12bd-40f8-b387-06fb66b66a08-kube-api-access-wmsf6\") pod \"e2e-ipp-a-d0441c-openshift-default-d67bc796c-57tvj\" (UID: \"2019d8e6-12bd-40f8-b387-06fb66b66a08\") " pod="openshift-ingress/e2e-ipp-a-d0441c-openshift-default-d67bc796c-57tvj" Nov 03 13:55:39.268374 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:55:39.268198 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2019d8e6-12bd-40f8-b387-06fb66b66a08-istio-token\") pod \"e2e-ipp-a-d0441c-openshift-default-d67bc796c-57tvj\" (UID: \"2019d8e6-12bd-40f8-b387-06fb66b66a08\") " pod="openshift-ingress/e2e-ipp-a-d0441c-openshift-default-d67bc796c-57tvj" Nov 03 13:55:39.268374 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:55:39.268216 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2019d8e6-12bd-40f8-b387-06fb66b66a08-istio-envoy\") pod \"e2e-ipp-a-d0441c-openshift-default-d67bc796c-57tvj\" (UID: \"2019d8e6-12bd-40f8-b387-06fb66b66a08\") " pod="openshift-ingress/e2e-ipp-a-d0441c-openshift-default-d67bc796c-57tvj" Nov 03 13:55:39.268374 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:55:39.268266 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2019d8e6-12bd-40f8-b387-06fb66b66a08-workload-socket\") pod \"e2e-ipp-a-d0441c-openshift-default-d67bc796c-57tvj\" (UID: \"2019d8e6-12bd-40f8-b387-06fb66b66a08\") " pod="openshift-ingress/e2e-ipp-a-d0441c-openshift-default-d67bc796c-57tvj" Nov 03 13:55:39.268374 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:55:39.268332 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2019d8e6-12bd-40f8-b387-06fb66b66a08-istio-podinfo\") pod \"e2e-ipp-a-d0441c-openshift-default-d67bc796c-57tvj\" (UID: \"2019d8e6-12bd-40f8-b387-06fb66b66a08\") " pod="openshift-ingress/e2e-ipp-a-d0441c-openshift-default-d67bc796c-57tvj" Nov 03 13:55:39.268751 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:55:39.268576 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2019d8e6-12bd-40f8-b387-06fb66b66a08-credential-socket\") pod \"e2e-ipp-a-d0441c-openshift-default-d67bc796c-57tvj\" (UID: \"2019d8e6-12bd-40f8-b387-06fb66b66a08\") " pod="openshift-ingress/e2e-ipp-a-d0441c-openshift-default-d67bc796c-57tvj" Nov 03 13:55:39.268751 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:55:39.268598 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2019d8e6-12bd-40f8-b387-06fb66b66a08-istio-data\") pod \"e2e-ipp-a-d0441c-openshift-default-d67bc796c-57tvj\" (UID: \"2019d8e6-12bd-40f8-b387-06fb66b66a08\") " pod="openshift-ingress/e2e-ipp-a-d0441c-openshift-default-d67bc796c-57tvj" Nov 03 13:55:39.268935 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:55:39.268899 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2019d8e6-12bd-40f8-b387-06fb66b66a08-workload-certs\") pod \"e2e-ipp-a-d0441c-openshift-default-d67bc796c-57tvj\" (UID: \"2019d8e6-12bd-40f8-b387-06fb66b66a08\") " pod="openshift-ingress/e2e-ipp-a-d0441c-openshift-default-d67bc796c-57tvj" Nov 03 13:55:39.269071 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:55:39.268950 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2019d8e6-12bd-40f8-b387-06fb66b66a08-workload-socket\") pod \"e2e-ipp-a-d0441c-openshift-default-d67bc796c-57tvj\" (UID: \"2019d8e6-12bd-40f8-b387-06fb66b66a08\") " pod="openshift-ingress/e2e-ipp-a-d0441c-openshift-default-d67bc796c-57tvj" Nov 03 13:55:39.269071 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:55:39.269057 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2019d8e6-12bd-40f8-b387-06fb66b66a08-istiod-ca-cert\") pod \"e2e-ipp-a-d0441c-openshift-default-d67bc796c-57tvj\" (UID: \"2019d8e6-12bd-40f8-b387-06fb66b66a08\") " pod="openshift-ingress/e2e-ipp-a-d0441c-openshift-default-d67bc796c-57tvj" Nov 03 13:55:39.270832 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:55:39.270808 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2019d8e6-12bd-40f8-b387-06fb66b66a08-istio-envoy\") pod \"e2e-ipp-a-d0441c-openshift-default-d67bc796c-57tvj\" (UID: \"2019d8e6-12bd-40f8-b387-06fb66b66a08\") " pod="openshift-ingress/e2e-ipp-a-d0441c-openshift-default-d67bc796c-57tvj" Nov 03 13:55:39.270987 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:55:39.270968 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2019d8e6-12bd-40f8-b387-06fb66b66a08-istio-podinfo\") pod \"e2e-ipp-a-d0441c-openshift-default-d67bc796c-57tvj\" (UID: \"2019d8e6-12bd-40f8-b387-06fb66b66a08\") " pod="openshift-ingress/e2e-ipp-a-d0441c-openshift-default-d67bc796c-57tvj" Nov 03 13:55:39.277498 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:55:39.277470 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2019d8e6-12bd-40f8-b387-06fb66b66a08-istio-token\") pod \"e2e-ipp-a-d0441c-openshift-default-d67bc796c-57tvj\" (UID: \"2019d8e6-12bd-40f8-b387-06fb66b66a08\") " pod="openshift-ingress/e2e-ipp-a-d0441c-openshift-default-d67bc796c-57tvj" Nov 03 13:55:39.277605 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:55:39.277582 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmsf6\" (UniqueName: \"kubernetes.io/projected/2019d8e6-12bd-40f8-b387-06fb66b66a08-kube-api-access-wmsf6\") pod \"e2e-ipp-a-d0441c-openshift-default-d67bc796c-57tvj\" (UID: \"2019d8e6-12bd-40f8-b387-06fb66b66a08\") " pod="openshift-ingress/e2e-ipp-a-d0441c-openshift-default-d67bc796c-57tvj" Nov 03 13:55:39.396415 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:55:39.396381 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-d0441c-openshift-default-d67bc796c-57tvj" Nov 03 13:55:39.515312 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:55:39.510779 5024 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d7mwz" containerName="extractor" Nov 03 13:55:39.687412 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:55:39.687376 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-d0441c-openshift-default-d67bc796c-57tvj"] Nov 03 13:55:39.687749 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:55:39.687718 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2019d8e6_12bd_40f8_b387_06fb66b66a08.slice/crio-ae17617a5a0e44332c88304a48e42e6ab12cea2d2e5e5ef74431df0a7aabaeeb WatchSource:0}: Error finding container ae17617a5a0e44332c88304a48e42e6ab12cea2d2e5e5ef74431df0a7aabaeeb: Status 404 returned error can't find the container with id ae17617a5a0e44332c88304a48e42e6ab12cea2d2e5e5ef74431df0a7aabaeeb Nov 03 13:55:39.690026 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:55:39.689987 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:55:39.690106 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:55:39.690077 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:55:39.690145 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:55:39.690120 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:55:39.855521 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:55:39.855476 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-d0441c-openshift-default-d67bc796c-57tvj" event={"ID":"2019d8e6-12bd-40f8-b387-06fb66b66a08","Type":"ContainerStarted","Data":"cdcd6a2408f99154fc7f340499e8c5fc27c66b2d0084f9a39776ccef66be55f6"} Nov 03 13:55:39.855724 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:55:39.855538 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-d0441c-openshift-default-d67bc796c-57tvj" event={"ID":"2019d8e6-12bd-40f8-b387-06fb66b66a08","Type":"ContainerStarted","Data":"ae17617a5a0e44332c88304a48e42e6ab12cea2d2e5e5ef74431df0a7aabaeeb"} Nov 03 13:55:39.878110 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:55:39.878059 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-d0441c-openshift-default-d67bc796c-57tvj" podStartSLOduration=0.878044816 podStartE2EDuration="878.044816ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:55:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:55:39.87701762 +0000 UTC m=+2819.885208898" watchObservedRunningTime="2026-11-03 13:55:39.878044816 +0000 UTC m=+2819.886236093" Nov 03 13:55:40.397026 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:55:40.396994 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-d0441c-openshift-default-d67bc796c-57tvj" Nov 03 13:55:40.402692 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:55:40.402659 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-d0441c-openshift-default-d67bc796c-57tvj" Nov 03 13:55:40.861132 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:55:40.861103 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-d0441c-openshift-default-d67bc796c-57tvj" Nov 03 13:55:40.862241 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:55:40.862217 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-d0441c-openshift-default-d67bc796c-57tvj" Nov 03 13:55:43.257783 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:55:43.257737 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-d0441c-7744d76cb6-pfrhs"] Nov 03 13:55:43.262123 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:55:43.262094 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d0441c-7744d76cb6-pfrhs" Nov 03 13:55:43.270604 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:55:43.270576 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 03 13:55:43.270790 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:55:43.270653 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-p5qd5\"" Nov 03 13:55:43.270790 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:55:43.270690 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-d0441c-serving-cert\"" Nov 03 13:55:43.270790 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:55:43.270576 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 13:55:43.271320 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:55:43.271278 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 13:55:43.288638 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:55:43.288611 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-d0441c-7744d76cb6-pfrhs"] Nov 03 13:55:43.411569 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:55:43.411530 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zcz9k\" (UniqueName: \"kubernetes.io/projected/adda3b0a-50a1-4bca-918c-a2f8dbe0f4ee-kube-api-access-zcz9k\") pod \"maas-api-e2e-ipp-a-d0441c-7744d76cb6-pfrhs\" (UID: \"adda3b0a-50a1-4bca-918c-a2f8dbe0f4ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d0441c-7744d76cb6-pfrhs" Nov 03 13:55:43.411771 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:55:43.411677 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/adda3b0a-50a1-4bca-918c-a2f8dbe0f4ee-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-d0441c-7744d76cb6-pfrhs\" (UID: \"adda3b0a-50a1-4bca-918c-a2f8dbe0f4ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d0441c-7744d76cb6-pfrhs" Nov 03 13:55:43.411771 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:55:43.411744 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/adda3b0a-50a1-4bca-918c-a2f8dbe0f4ee-metrics-certs\") pod \"maas-api-e2e-ipp-a-d0441c-7744d76cb6-pfrhs\" (UID: \"adda3b0a-50a1-4bca-918c-a2f8dbe0f4ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d0441c-7744d76cb6-pfrhs" Nov 03 13:55:43.411924 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:55:43.411895 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/adda3b0a-50a1-4bca-918c-a2f8dbe0f4ee-maas-api-tls\") pod \"maas-api-e2e-ipp-a-d0441c-7744d76cb6-pfrhs\" (UID: \"adda3b0a-50a1-4bca-918c-a2f8dbe0f4ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d0441c-7744d76cb6-pfrhs" Nov 03 13:55:43.457759 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:55:43.457726 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-d0441c-54cccc9767-t76cc"] Nov 03 13:55:43.461496 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:55:43.461479 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d0441c-54cccc9767-t76cc" Nov 03 13:55:43.469995 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:55:43.469968 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-d0441c-dockercfg-qtkkw\"" Nov 03 13:55:43.470132 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:55:43.470053 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-d0441c\"" Nov 03 13:55:43.472621 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:55:43.472588 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-d0441c-54cccc9767-t76cc"] Nov 03 13:55:43.492169 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:55:43.492142 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-d0441c-5678568b5-zwjch"] Nov 03 13:55:43.495790 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:55:43.495768 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-d0441c-5678568b5-zwjch" Nov 03 13:55:43.510054 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:55:43.510001 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-d0441c-5678568b5-zwjch"] Nov 03 13:55:43.512453 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:55:43.512422 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/adda3b0a-50a1-4bca-918c-a2f8dbe0f4ee-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-d0441c-7744d76cb6-pfrhs\" (UID: \"adda3b0a-50a1-4bca-918c-a2f8dbe0f4ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d0441c-7744d76cb6-pfrhs" Nov 03 13:55:43.512568 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:55:43.512500 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/adda3b0a-50a1-4bca-918c-a2f8dbe0f4ee-metrics-certs\") pod \"maas-api-e2e-ipp-a-d0441c-7744d76cb6-pfrhs\" (UID: \"adda3b0a-50a1-4bca-918c-a2f8dbe0f4ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d0441c-7744d76cb6-pfrhs" Nov 03 13:55:43.512640 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:55:43.512579 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/adda3b0a-50a1-4bca-918c-a2f8dbe0f4ee-maas-api-tls\") pod \"maas-api-e2e-ipp-a-d0441c-7744d76cb6-pfrhs\" (UID: \"adda3b0a-50a1-4bca-918c-a2f8dbe0f4ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d0441c-7744d76cb6-pfrhs" Nov 03 13:55:43.512706 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:55:43.512639 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zcz9k\" (UniqueName: \"kubernetes.io/projected/adda3b0a-50a1-4bca-918c-a2f8dbe0f4ee-kube-api-access-zcz9k\") pod \"maas-api-e2e-ipp-a-d0441c-7744d76cb6-pfrhs\" (UID: \"adda3b0a-50a1-4bca-918c-a2f8dbe0f4ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d0441c-7744d76cb6-pfrhs" Nov 03 13:55:43.513382 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:55:43.513349 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/adda3b0a-50a1-4bca-918c-a2f8dbe0f4ee-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-d0441c-7744d76cb6-pfrhs\" (UID: \"adda3b0a-50a1-4bca-918c-a2f8dbe0f4ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d0441c-7744d76cb6-pfrhs" Nov 03 13:55:43.515820 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:55:43.515796 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/adda3b0a-50a1-4bca-918c-a2f8dbe0f4ee-maas-api-tls\") pod \"maas-api-e2e-ipp-a-d0441c-7744d76cb6-pfrhs\" (UID: \"adda3b0a-50a1-4bca-918c-a2f8dbe0f4ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d0441c-7744d76cb6-pfrhs" Nov 03 13:55:43.515941 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:55:43.515796 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/adda3b0a-50a1-4bca-918c-a2f8dbe0f4ee-metrics-certs\") pod \"maas-api-e2e-ipp-a-d0441c-7744d76cb6-pfrhs\" (UID: \"adda3b0a-50a1-4bca-918c-a2f8dbe0f4ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d0441c-7744d76cb6-pfrhs" Nov 03 13:55:43.522062 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:55:43.522042 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zcz9k\" (UniqueName: \"kubernetes.io/projected/adda3b0a-50a1-4bca-918c-a2f8dbe0f4ee-kube-api-access-zcz9k\") pod \"maas-api-e2e-ipp-a-d0441c-7744d76cb6-pfrhs\" (UID: \"adda3b0a-50a1-4bca-918c-a2f8dbe0f4ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d0441c-7744d76cb6-pfrhs" Nov 03 13:55:43.592737 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:55:43.592695 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d0441c-7744d76cb6-pfrhs" Nov 03 13:55:43.613641 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:55:43.613605 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac3eda30-7046-4c7d-8b64-db922dc586a2-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-d0441c-5678568b5-zwjch\" (UID: \"ac3eda30-7046-4c7d-8b64-db922dc586a2\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-d0441c-5678568b5-zwjch" Nov 03 13:55:43.613793 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:55:43.613719 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6st8\" (UniqueName: \"kubernetes.io/projected/0b6a39ee-06ce-4f0b-8808-053cc47e48b6-kube-api-access-b6st8\") pod \"payload-pre-processing-e2e-ipp-a-d0441c-54cccc9767-t76cc\" (UID: \"0b6a39ee-06ce-4f0b-8808-053cc47e48b6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d0441c-54cccc9767-t76cc" Nov 03 13:55:43.613793 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:55:43.613768 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b6a39ee-06ce-4f0b-8808-053cc47e48b6-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-d0441c-54cccc9767-t76cc\" (UID: \"0b6a39ee-06ce-4f0b-8808-053cc47e48b6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d0441c-54cccc9767-t76cc" Nov 03 13:55:43.613919 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:55:43.613891 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kk99r\" (UniqueName: \"kubernetes.io/projected/ac3eda30-7046-4c7d-8b64-db922dc586a2-kube-api-access-kk99r\") pod \"payload-processing-e2e-ipp-a-d0441c-5678568b5-zwjch\" (UID: \"ac3eda30-7046-4c7d-8b64-db922dc586a2\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-d0441c-5678568b5-zwjch" Nov 03 13:55:43.715500 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:55:43.715460 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kk99r\" (UniqueName: \"kubernetes.io/projected/ac3eda30-7046-4c7d-8b64-db922dc586a2-kube-api-access-kk99r\") pod \"payload-processing-e2e-ipp-a-d0441c-5678568b5-zwjch\" (UID: \"ac3eda30-7046-4c7d-8b64-db922dc586a2\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-d0441c-5678568b5-zwjch" Nov 03 13:55:43.715670 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:55:43.715547 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac3eda30-7046-4c7d-8b64-db922dc586a2-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-d0441c-5678568b5-zwjch\" (UID: \"ac3eda30-7046-4c7d-8b64-db922dc586a2\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-d0441c-5678568b5-zwjch" Nov 03 13:55:43.715670 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:55:43.715600 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6st8\" (UniqueName: \"kubernetes.io/projected/0b6a39ee-06ce-4f0b-8808-053cc47e48b6-kube-api-access-b6st8\") pod \"payload-pre-processing-e2e-ipp-a-d0441c-54cccc9767-t76cc\" (UID: \"0b6a39ee-06ce-4f0b-8808-053cc47e48b6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d0441c-54cccc9767-t76cc" Nov 03 13:55:43.715670 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:55:43.715647 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b6a39ee-06ce-4f0b-8808-053cc47e48b6-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-d0441c-54cccc9767-t76cc\" (UID: \"0b6a39ee-06ce-4f0b-8808-053cc47e48b6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d0441c-54cccc9767-t76cc" Nov 03 13:55:43.716454 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:55:43.716426 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac3eda30-7046-4c7d-8b64-db922dc586a2-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-d0441c-5678568b5-zwjch\" (UID: \"ac3eda30-7046-4c7d-8b64-db922dc586a2\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-d0441c-5678568b5-zwjch" Nov 03 13:55:43.716599 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:55:43.716514 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b6a39ee-06ce-4f0b-8808-053cc47e48b6-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-d0441c-54cccc9767-t76cc\" (UID: \"0b6a39ee-06ce-4f0b-8808-053cc47e48b6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d0441c-54cccc9767-t76cc" Nov 03 13:55:43.733469 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:55:43.733444 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6st8\" (UniqueName: \"kubernetes.io/projected/0b6a39ee-06ce-4f0b-8808-053cc47e48b6-kube-api-access-b6st8\") pod \"payload-pre-processing-e2e-ipp-a-d0441c-54cccc9767-t76cc\" (UID: \"0b6a39ee-06ce-4f0b-8808-053cc47e48b6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d0441c-54cccc9767-t76cc" Nov 03 13:55:43.733587 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:55:43.733481 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kk99r\" (UniqueName: \"kubernetes.io/projected/ac3eda30-7046-4c7d-8b64-db922dc586a2-kube-api-access-kk99r\") pod \"payload-processing-e2e-ipp-a-d0441c-5678568b5-zwjch\" (UID: \"ac3eda30-7046-4c7d-8b64-db922dc586a2\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-d0441c-5678568b5-zwjch" Nov 03 13:55:43.775507 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:55:43.775427 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d0441c-54cccc9767-t76cc" Nov 03 13:55:43.811971 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:55:43.811940 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-d0441c-5678568b5-zwjch" Nov 03 13:55:43.887558 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:55:43.887528 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-d0441c-7744d76cb6-pfrhs"] Nov 03 13:55:43.920214 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:55:43.920166 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podadda3b0a_50a1_4bca_918c_a2f8dbe0f4ee.slice/crio-627d927a490a5687390838be09667f43688088061ce99ffe0d96bd31918ed4b2 WatchSource:0}: Error finding container 627d927a490a5687390838be09667f43688088061ce99ffe0d96bd31918ed4b2: Status 404 returned error can't find the container with id 627d927a490a5687390838be09667f43688088061ce99ffe0d96bd31918ed4b2 Nov 03 13:55:44.054162 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:55:44.054132 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-d0441c-54cccc9767-t76cc"] Nov 03 13:55:44.055703 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:55:44.055676 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b6a39ee_06ce_4f0b_8808_053cc47e48b6.slice/crio-09ceae80e6858dbdae9418c63cb80b90ad156903af71e936ae564b6b5f047227 WatchSource:0}: Error finding container 09ceae80e6858dbdae9418c63cb80b90ad156903af71e936ae564b6b5f047227: Status 404 returned error can't find the container with id 09ceae80e6858dbdae9418c63cb80b90ad156903af71e936ae564b6b5f047227 Nov 03 13:55:44.105790 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:55:44.105744 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac3eda30_7046_4c7d_8b64_db922dc586a2.slice/crio-28ea7870624af453686089941f9f0778e60d756a45a293639437d14ad10e5362 WatchSource:0}: Error finding container 28ea7870624af453686089941f9f0778e60d756a45a293639437d14ad10e5362: Status 404 returned error can't find the container with id 28ea7870624af453686089941f9f0778e60d756a45a293639437d14ad10e5362 Nov 03 13:55:44.110686 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:55:44.110660 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-d0441c-5678568b5-zwjch"] Nov 03 13:55:44.892219 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:55:44.892167 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-d0441c-5678568b5-zwjch" event={"ID":"ac3eda30-7046-4c7d-8b64-db922dc586a2","Type":"ContainerStarted","Data":"25eee8edb6616919fc7b7278bf3a8576e1cba95d2d99c29edbf4486a0bb885fa"} Nov 03 13:55:44.892809 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:55:44.892249 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-d0441c-5678568b5-zwjch" event={"ID":"ac3eda30-7046-4c7d-8b64-db922dc586a2","Type":"ContainerStarted","Data":"28ea7870624af453686089941f9f0778e60d756a45a293639437d14ad10e5362"} Nov 03 13:55:44.892809 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:55:44.892422 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-d0441c-5678568b5-zwjch" Nov 03 13:55:44.894336 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:55:44.894307 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d0441c-7744d76cb6-pfrhs" event={"ID":"adda3b0a-50a1-4bca-918c-a2f8dbe0f4ee","Type":"ContainerStarted","Data":"a4e4700e7fb8f1be599b736b177a6bb49533c15678fcb40c031c3eccf8d95cd6"} Nov 03 13:55:44.894475 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:55:44.894356 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d0441c-7744d76cb6-pfrhs" event={"ID":"adda3b0a-50a1-4bca-918c-a2f8dbe0f4ee","Type":"ContainerStarted","Data":"627d927a490a5687390838be09667f43688088061ce99ffe0d96bd31918ed4b2"} Nov 03 13:55:44.894475 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:55:44.894385 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d0441c-7744d76cb6-pfrhs" Nov 03 13:55:44.896213 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:55:44.896186 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d0441c-54cccc9767-t76cc" event={"ID":"0b6a39ee-06ce-4f0b-8808-053cc47e48b6","Type":"ContainerStarted","Data":"ed477160bd1932351c9937159829e87735da2574ae08b428b3b4d1613f94a77d"} Nov 03 13:55:44.896365 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:55:44.896234 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d0441c-54cccc9767-t76cc" event={"ID":"0b6a39ee-06ce-4f0b-8808-053cc47e48b6","Type":"ContainerStarted","Data":"09ceae80e6858dbdae9418c63cb80b90ad156903af71e936ae564b6b5f047227"} Nov 03 13:55:44.896365 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:55:44.896267 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d0441c-54cccc9767-t76cc" Nov 03 13:55:44.916384 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:55:44.916329 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-d0441c-5678568b5-zwjch" podStartSLOduration=1.9163130750000001 podStartE2EDuration="1.916313075s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:55:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:55:44.911070423 +0000 UTC m=+2824.919261701" watchObservedRunningTime="2026-11-03 13:55:44.916313075 +0000 UTC m=+2824.924504354" Nov 03 13:55:44.934944 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:55:44.934895 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d0441c-7744d76cb6-pfrhs" podStartSLOduration=1.12972191 podStartE2EDuration="1.9348807s" totalImagesPullingTime="805.15879ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:55:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:55:44.931505175 +0000 UTC m=+2824.939696453" watchObservedRunningTime="2026-11-03 13:55:44.9348807 +0000 UTC m=+2824.943071977" Nov 03 13:55:44.950691 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:55:44.950645 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d0441c-54cccc9767-t76cc" podStartSLOduration=1.950633216 podStartE2EDuration="1.950633216s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:55:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:55:44.949214813 +0000 UTC m=+2824.957406090" watchObservedRunningTime="2026-11-03 13:55:44.950633216 +0000 UTC m=+2824.958824521" Nov 03 13:55:45.918125 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:55:45.918089 5024 scope.go:122] "RemoveContainer" containerID="6ddb347c0c969acd75b79cda529a539475c46cf5acbf7b48bc83e5fd4e3e2554" Nov 03 13:55:45.928141 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:55:45.928109 5024 scope.go:122] "RemoveContainer" containerID="3faca8f151cfc8ce6630be326063f7f218d1bfe51bc6a3e812b14953373632ed" Nov 03 13:55:45.937266 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:55:45.937238 5024 scope.go:122] "RemoveContainer" containerID="4394f8bb87b2acae3239088442fafe4c488da85ea60f858dc21248a0c33de53a" Nov 03 13:55:45.945572 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:55:45.945546 5024 scope.go:122] "RemoveContainer" containerID="60baf7c2f3d913f953437449d73e1294f14168bd4ecb40323a483f704679fbfb" Nov 03 13:55:49.026483 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:55:49.026447 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a80c6bd9-qpdr2"] Nov 03 13:55:49.028597 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:55:49.028567 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a80c6bd9-qpdr2"] Nov 03 13:55:50.108407 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:55:50.108371 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dcffa56d-1f8e-4aab-9105-db5888e2bf4a" path="/var/lib/kubelet/pods/dcffa56d-1f8e-4aab-9105-db5888e2bf4a/volumes" Nov 03 13:55:50.910145 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:55:50.910111 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d0441c-7744d76cb6-pfrhs" Nov 03 13:55:55.862965 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:55:55.862925 5024 patch_prober.go:29] interesting pod/e2e-ipp-a-d0441c-openshift-default-d67bc796c-57tvj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" start-of-body= Nov 03 13:55:55.863447 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:55:55.862983 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-d0441c-openshift-default-d67bc796c-57tvj" podUID="2019d8e6-12bd-40f8-b387-06fb66b66a08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" Nov 03 13:55:55.906367 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:55:55.906336 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-d0441c-5678568b5-zwjch" Nov 03 13:55:55.906565 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:55:55.906419 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d0441c-54cccc9767-t76cc" Nov 03 13:55:55.966003 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:55:55.965963 5024 generic.go:356] "Generic (PLEG): container finished" podID="2019d8e6-12bd-40f8-b387-06fb66b66a08" containerID="cdcd6a2408f99154fc7f340499e8c5fc27c66b2d0084f9a39776ccef66be55f6" exitCode=0 Nov 03 13:55:55.966230 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:55:55.966028 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-d0441c-openshift-default-d67bc796c-57tvj" event={"ID":"2019d8e6-12bd-40f8-b387-06fb66b66a08","Type":"ContainerDied","Data":"cdcd6a2408f99154fc7f340499e8c5fc27c66b2d0084f9a39776ccef66be55f6"} Nov 03 13:55:55.966507 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:55:55.966472 5024 scope.go:122] "RemoveContainer" containerID="cdcd6a2408f99154fc7f340499e8c5fc27c66b2d0084f9a39776ccef66be55f6" Nov 03 13:55:55.967878 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:55:55.967850 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:55:55.967959 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:55:55.967919 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:55:55.968008 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:55:55.967957 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:55:56.997699 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:55:56.997663 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-d0441c-openshift-default-d67bc796c-57tvj" event={"ID":"2019d8e6-12bd-40f8-b387-06fb66b66a08","Type":"ContainerStarted","Data":"1cd9b458925c471109059b17a691630d8cce9d19b4c89d64b2600b7801a98e7a"} Nov 03 13:55:57.043339 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:55:57.043302 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-d0441c-5678568b5-zwjch"] Nov 03 13:55:57.059818 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:55:57.059786 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-d0441c-54cccc9767-t76cc"] Nov 03 13:55:57.401888 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:55:57.401848 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-d0441c-openshift-default-d67bc796c-57tvj" Nov 03 13:55:57.402788 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:55:57.402652 5024 patch_prober.go:29] interesting pod/e2e-ipp-a-d0441c-openshift-default-d67bc796c-57tvj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" start-of-body= Nov 03 13:55:57.402788 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:55:57.402710 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-d0441c-openshift-default-d67bc796c-57tvj" podUID="2019d8e6-12bd-40f8-b387-06fb66b66a08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" Nov 03 13:55:58.396757 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:55:58.396718 5024 patch_prober.go:29] interesting pod/e2e-ipp-a-d0441c-openshift-default-d67bc796c-57tvj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" start-of-body= Nov 03 13:55:58.397373 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:55:58.397342 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-d0441c-openshift-default-d67bc796c-57tvj" podUID="2019d8e6-12bd-40f8-b387-06fb66b66a08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" Nov 03 13:55:59.029867 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:55:59.029829 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c8af12ec-qvb2g"] Nov 03 13:55:59.035231 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:55:59.035199 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c8af12ec-qvb2g"] Nov 03 13:55:59.195488 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:55:59.195451 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-d0441c-66875c8bc9-tw57h"] Nov 03 13:55:59.200398 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:55:59.200366 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d0441c-66875c8bc9-tw57h" Nov 03 13:55:59.203826 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:55:59.203799 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-d0441c-66875c8bc9-tw57h"] Nov 03 13:55:59.204975 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:55:59.204950 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-d0441c-dockercfg-tj556\"" Nov 03 13:55:59.229160 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:55:59.229127 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9899225c-d0e5-4426-b09a-a1009b6b109e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-d0441c-66875c8bc9-tw57h\" (UID: \"9899225c-d0e5-4426-b09a-a1009b6b109e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d0441c-66875c8bc9-tw57h" Nov 03 13:55:59.229303 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:55:59.229237 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8mpc\" (UniqueName: \"kubernetes.io/projected/9899225c-d0e5-4426-b09a-a1009b6b109e-kube-api-access-k8mpc\") pod \"payload-pre-processing-e2e-ipp-a-d0441c-66875c8bc9-tw57h\" (UID: \"9899225c-d0e5-4426-b09a-a1009b6b109e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d0441c-66875c8bc9-tw57h" Nov 03 13:55:59.312079 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:55:59.311993 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-d0441c-7776b4999d-6kzsf"] Nov 03 13:55:59.316604 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:55:59.316577 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-d0441c-7776b4999d-6kzsf"] Nov 03 13:55:59.316788 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:55:59.316664 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-d0441c-7776b4999d-6kzsf" Nov 03 13:55:59.330211 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:55:59.330178 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9899225c-d0e5-4426-b09a-a1009b6b109e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-d0441c-66875c8bc9-tw57h\" (UID: \"9899225c-d0e5-4426-b09a-a1009b6b109e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d0441c-66875c8bc9-tw57h" Nov 03 13:55:59.330353 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:55:59.330335 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8mpc\" (UniqueName: \"kubernetes.io/projected/9899225c-d0e5-4426-b09a-a1009b6b109e-kube-api-access-k8mpc\") pod \"payload-pre-processing-e2e-ipp-a-d0441c-66875c8bc9-tw57h\" (UID: \"9899225c-d0e5-4426-b09a-a1009b6b109e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d0441c-66875c8bc9-tw57h" Nov 03 13:55:59.330402 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:55:59.330377 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2072f8f-fee3-45b5-acc9-f1be7a68943d-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-d0441c-7776b4999d-6kzsf\" (UID: \"b2072f8f-fee3-45b5-acc9-f1be7a68943d\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-d0441c-7776b4999d-6kzsf" Nov 03 13:55:59.330502 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:55:59.330486 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vx4m4\" (UniqueName: \"kubernetes.io/projected/b2072f8f-fee3-45b5-acc9-f1be7a68943d-kube-api-access-vx4m4\") pod \"payload-processing-e2e-ipp-a-d0441c-7776b4999d-6kzsf\" (UID: \"b2072f8f-fee3-45b5-acc9-f1be7a68943d\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-d0441c-7776b4999d-6kzsf" Nov 03 13:55:59.330897 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:55:59.330879 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9899225c-d0e5-4426-b09a-a1009b6b109e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-d0441c-66875c8bc9-tw57h\" (UID: \"9899225c-d0e5-4426-b09a-a1009b6b109e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d0441c-66875c8bc9-tw57h" Nov 03 13:55:59.340397 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:55:59.340370 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8mpc\" (UniqueName: \"kubernetes.io/projected/9899225c-d0e5-4426-b09a-a1009b6b109e-kube-api-access-k8mpc\") pod \"payload-pre-processing-e2e-ipp-a-d0441c-66875c8bc9-tw57h\" (UID: \"9899225c-d0e5-4426-b09a-a1009b6b109e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d0441c-66875c8bc9-tw57h" Nov 03 13:55:59.401757 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:55:59.401710 5024 patch_prober.go:29] interesting pod/e2e-ipp-a-d0441c-openshift-default-d67bc796c-57tvj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" start-of-body= Nov 03 13:55:59.402406 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:55:59.402373 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-d0441c-openshift-default-d67bc796c-57tvj" podUID="2019d8e6-12bd-40f8-b387-06fb66b66a08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" Nov 03 13:55:59.430948 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:55:59.430913 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vx4m4\" (UniqueName: \"kubernetes.io/projected/b2072f8f-fee3-45b5-acc9-f1be7a68943d-kube-api-access-vx4m4\") pod \"payload-processing-e2e-ipp-a-d0441c-7776b4999d-6kzsf\" (UID: \"b2072f8f-fee3-45b5-acc9-f1be7a68943d\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-d0441c-7776b4999d-6kzsf" Nov 03 13:55:59.431112 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:55:59.431012 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2072f8f-fee3-45b5-acc9-f1be7a68943d-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-d0441c-7776b4999d-6kzsf\" (UID: \"b2072f8f-fee3-45b5-acc9-f1be7a68943d\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-d0441c-7776b4999d-6kzsf" Nov 03 13:55:59.431782 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:55:59.431762 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2072f8f-fee3-45b5-acc9-f1be7a68943d-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-d0441c-7776b4999d-6kzsf\" (UID: \"b2072f8f-fee3-45b5-acc9-f1be7a68943d\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-d0441c-7776b4999d-6kzsf" Nov 03 13:55:59.440526 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:55:59.440505 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vx4m4\" (UniqueName: \"kubernetes.io/projected/b2072f8f-fee3-45b5-acc9-f1be7a68943d-kube-api-access-vx4m4\") pod \"payload-processing-e2e-ipp-a-d0441c-7776b4999d-6kzsf\" (UID: \"b2072f8f-fee3-45b5-acc9-f1be7a68943d\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-d0441c-7776b4999d-6kzsf" Nov 03 13:55:59.516414 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:55:59.516376 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d0441c-66875c8bc9-tw57h" Nov 03 13:55:59.630647 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:55:59.630419 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-d0441c-7776b4999d-6kzsf" Nov 03 13:55:59.808130 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:55:59.808102 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-d0441c-66875c8bc9-tw57h"] Nov 03 13:55:59.809004 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:55:59.808979 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9899225c_d0e5_4426_b09a_a1009b6b109e.slice/crio-2769b145ca0963ee4c568f25e3115da6982c95670475be1634b0314cfcb0a4e0 WatchSource:0}: Error finding container 2769b145ca0963ee4c568f25e3115da6982c95670475be1634b0314cfcb0a4e0: Status 404 returned error can't find the container with id 2769b145ca0963ee4c568f25e3115da6982c95670475be1634b0314cfcb0a4e0 Nov 03 13:55:59.941063 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:55:59.941025 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-d0441c-7776b4999d-6kzsf"] Nov 03 13:55:59.944398 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:55:59.944361 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2072f8f_fee3_45b5_acc9_f1be7a68943d.slice/crio-318fde18a330defc27ae04397c632df2498a1c3a9bae45bcdc3bfc3703d83d51 WatchSource:0}: Error finding container 318fde18a330defc27ae04397c632df2498a1c3a9bae45bcdc3bfc3703d83d51: Status 404 returned error can't find the container with id 318fde18a330defc27ae04397c632df2498a1c3a9bae45bcdc3bfc3703d83d51 Nov 03 13:56:00.017256 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:56:00.017213 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-d0441c-7776b4999d-6kzsf" event={"ID":"b2072f8f-fee3-45b5-acc9-f1be7a68943d","Type":"ContainerStarted","Data":"318fde18a330defc27ae04397c632df2498a1c3a9bae45bcdc3bfc3703d83d51"} Nov 03 13:56:00.018550 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:56:00.018513 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d0441c-66875c8bc9-tw57h" event={"ID":"9899225c-d0e5-4426-b09a-a1009b6b109e","Type":"ContainerStarted","Data":"2769b145ca0963ee4c568f25e3115da6982c95670475be1634b0314cfcb0a4e0"} Nov 03 13:56:00.108278 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:56:00.108243 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="57920e64-cc2d-476d-8ede-92077229ab21" path="/var/lib/kubelet/pods/57920e64-cc2d-476d-8ede-92077229ab21/volumes" Nov 03 13:56:00.400139 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:56:00.399691 5024 patch_prober.go:29] interesting pod/e2e-ipp-a-d0441c-openshift-default-d67bc796c-57tvj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" start-of-body= Nov 03 13:56:00.400139 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:56:00.399765 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-d0441c-openshift-default-d67bc796c-57tvj" podUID="2019d8e6-12bd-40f8-b387-06fb66b66a08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" Nov 03 13:56:01.402977 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:56:01.402553 5024 patch_prober.go:29] interesting pod/e2e-ipp-a-d0441c-openshift-default-d67bc796c-57tvj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" start-of-body= Nov 03 13:56:01.402977 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:56:01.402612 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-d0441c-openshift-default-d67bc796c-57tvj" podUID="2019d8e6-12bd-40f8-b387-06fb66b66a08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" Nov 03 13:56:02.043623 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:56:02.043576 5024 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-d0441c-5678568b5-zwjch" podUID="ac3eda30-7046-4c7d-8b64-db922dc586a2" containerName="payload-processing" containerID="cri-o://25eee8edb6616919fc7b7278bf3a8576e1cba95d2d99c29edbf4486a0bb885fa" gracePeriod=25 Nov 03 13:56:02.060230 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:56:02.060185 5024 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d0441c-54cccc9767-t76cc" podUID="0b6a39ee-06ce-4f0b-8808-053cc47e48b6" containerName="payload-pre-processing" containerID="cri-o://ed477160bd1932351c9937159829e87735da2574ae08b428b3b4d1613f94a77d" gracePeriod=25 Nov 03 13:56:02.402982 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:56:02.402945 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-d0441c-openshift-default-d67bc796c-57tvj" Nov 03 13:56:02.404016 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:56:02.403995 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-d0441c-openshift-default-d67bc796c-57tvj" Nov 03 13:56:02.404758 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:56:02.404732 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-d0441c-openshift-default-d67bc796c-57tvj" Nov 03 13:56:02.449693 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:56:02.449665 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d0441c-54cccc9767-t76cc" Nov 03 13:56:02.479830 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:56:02.479770 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b6a39ee-06ce-4f0b-8808-053cc47e48b6-kube-api-access-b6st8\" (UniqueName: \"kubernetes.io/projected/0b6a39ee-06ce-4f0b-8808-053cc47e48b6-kube-api-access-b6st8\") pod \"0b6a39ee-06ce-4f0b-8808-053cc47e48b6\" (UID: \"0b6a39ee-06ce-4f0b-8808-053cc47e48b6\") " Nov 03 13:56:02.479990 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:56:02.479932 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0b6a39ee-06ce-4f0b-8808-053cc47e48b6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b6a39ee-06ce-4f0b-8808-053cc47e48b6-plugins-config-volume\") pod \"0b6a39ee-06ce-4f0b-8808-053cc47e48b6\" (UID: \"0b6a39ee-06ce-4f0b-8808-053cc47e48b6\") " Nov 03 13:56:02.483488 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:56:02.483462 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b6a39ee-06ce-4f0b-8808-053cc47e48b6-plugins-config-volume" pod "0b6a39ee-06ce-4f0b-8808-053cc47e48b6" (UID: "0b6a39ee-06ce-4f0b-8808-053cc47e48b6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:56:02.490991 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:56:02.490953 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b6a39ee-06ce-4f0b-8808-053cc47e48b6-kube-api-access-b6st8" pod "0b6a39ee-06ce-4f0b-8808-053cc47e48b6" (UID: "0b6a39ee-06ce-4f0b-8808-053cc47e48b6"). InnerVolumeSpecName "kube-api-access-b6st8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:56:02.491764 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:56:02.491731 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-d0441c-5678568b5-zwjch" Nov 03 13:56:02.581159 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:56:02.580703 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ac3eda30-7046-4c7d-8b64-db922dc586a2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac3eda30-7046-4c7d-8b64-db922dc586a2-plugins-config-volume\") pod \"ac3eda30-7046-4c7d-8b64-db922dc586a2\" (UID: \"ac3eda30-7046-4c7d-8b64-db922dc586a2\") " Nov 03 13:56:02.581159 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:56:02.580766 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac3eda30-7046-4c7d-8b64-db922dc586a2-kube-api-access-kk99r\" (UniqueName: \"kubernetes.io/projected/ac3eda30-7046-4c7d-8b64-db922dc586a2-kube-api-access-kk99r\") pod \"ac3eda30-7046-4c7d-8b64-db922dc586a2\" (UID: \"ac3eda30-7046-4c7d-8b64-db922dc586a2\") " Nov 03 13:56:02.581159 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:56:02.580972 5024 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b6a39ee-06ce-4f0b-8808-053cc47e48b6-plugins-config-volume\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:56:02.581159 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:56:02.580984 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b6st8\" (UniqueName: \"kubernetes.io/projected/0b6a39ee-06ce-4f0b-8808-053cc47e48b6-kube-api-access-b6st8\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:56:02.581900 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:56:02.581870 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac3eda30-7046-4c7d-8b64-db922dc586a2-plugins-config-volume" pod "ac3eda30-7046-4c7d-8b64-db922dc586a2" (UID: "ac3eda30-7046-4c7d-8b64-db922dc586a2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:56:02.584095 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:56:02.584067 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac3eda30-7046-4c7d-8b64-db922dc586a2-kube-api-access-kk99r" pod "ac3eda30-7046-4c7d-8b64-db922dc586a2" (UID: "ac3eda30-7046-4c7d-8b64-db922dc586a2"). InnerVolumeSpecName "kube-api-access-kk99r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:56:02.682119 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:56:02.682079 5024 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac3eda30-7046-4c7d-8b64-db922dc586a2-plugins-config-volume\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:56:02.682119 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:56:02.682112 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kk99r\" (UniqueName: \"kubernetes.io/projected/ac3eda30-7046-4c7d-8b64-db922dc586a2-kube-api-access-kk99r\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:56:03.050886 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:56:03.050845 5024 generic.go:356] "Generic (PLEG): container finished" podID="0b6a39ee-06ce-4f0b-8808-053cc47e48b6" containerID="ed477160bd1932351c9937159829e87735da2574ae08b428b3b4d1613f94a77d" exitCode=0 Nov 03 13:56:03.051105 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:56:03.050908 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d0441c-54cccc9767-t76cc" Nov 03 13:56:03.051105 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:56:03.050938 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d0441c-54cccc9767-t76cc" event={"ID":"0b6a39ee-06ce-4f0b-8808-053cc47e48b6","Type":"ContainerDied","Data":"ed477160bd1932351c9937159829e87735da2574ae08b428b3b4d1613f94a77d"} Nov 03 13:56:03.051105 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:56:03.051058 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d0441c-54cccc9767-t76cc" event={"ID":"0b6a39ee-06ce-4f0b-8808-053cc47e48b6","Type":"ContainerDied","Data":"09ceae80e6858dbdae9418c63cb80b90ad156903af71e936ae564b6b5f047227"} Nov 03 13:56:03.051105 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:56:03.051092 5024 scope.go:122] "RemoveContainer" containerID="ed477160bd1932351c9937159829e87735da2574ae08b428b3b4d1613f94a77d" Nov 03 13:56:03.052856 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:56:03.052802 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d0441c-66875c8bc9-tw57h" event={"ID":"9899225c-d0e5-4426-b09a-a1009b6b109e","Type":"ContainerStarted","Data":"d8f7d982a67fa5ddfbcf411888f6fe32fd62044bd7900dcb418a06327203634c"} Nov 03 13:56:03.053062 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:56:03.053035 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d0441c-66875c8bc9-tw57h" Nov 03 13:56:03.054549 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:56:03.054516 5024 generic.go:356] "Generic (PLEG): container finished" podID="ac3eda30-7046-4c7d-8b64-db922dc586a2" containerID="25eee8edb6616919fc7b7278bf3a8576e1cba95d2d99c29edbf4486a0bb885fa" exitCode=0 Nov 03 13:56:03.054672 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:56:03.054592 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-d0441c-5678568b5-zwjch" Nov 03 13:56:03.054672 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:56:03.054602 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-d0441c-5678568b5-zwjch" event={"ID":"ac3eda30-7046-4c7d-8b64-db922dc586a2","Type":"ContainerDied","Data":"25eee8edb6616919fc7b7278bf3a8576e1cba95d2d99c29edbf4486a0bb885fa"} Nov 03 13:56:03.054672 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:56:03.054658 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-d0441c-5678568b5-zwjch" event={"ID":"ac3eda30-7046-4c7d-8b64-db922dc586a2","Type":"ContainerDied","Data":"28ea7870624af453686089941f9f0778e60d756a45a293639437d14ad10e5362"} Nov 03 13:56:03.056824 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:56:03.056801 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-d0441c-7776b4999d-6kzsf" event={"ID":"b2072f8f-fee3-45b5-acc9-f1be7a68943d","Type":"ContainerStarted","Data":"e9ac87ee8698e71dd056d97a0fb454f9abe5be2dc65235038b501f9e2fcd9251"} Nov 03 13:56:03.057016 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:56:03.056991 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-d0441c-7776b4999d-6kzsf" Nov 03 13:56:03.071295 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:56:03.071265 5024 scope.go:122] "RemoveContainer" containerID="ed477160bd1932351c9937159829e87735da2574ae08b428b3b4d1613f94a77d" Nov 03 13:56:03.071574 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:56:03.071546 5024 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed477160bd1932351c9937159829e87735da2574ae08b428b3b4d1613f94a77d\": container with ID starting with ed477160bd1932351c9937159829e87735da2574ae08b428b3b4d1613f94a77d not found: ID does not exist" containerID="ed477160bd1932351c9937159829e87735da2574ae08b428b3b4d1613f94a77d" Nov 03 13:56:03.071637 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:56:03.071584 5024 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed477160bd1932351c9937159829e87735da2574ae08b428b3b4d1613f94a77d"} err="failed to get container status \"ed477160bd1932351c9937159829e87735da2574ae08b428b3b4d1613f94a77d\": rpc error: code = NotFound desc = could not find container \"ed477160bd1932351c9937159829e87735da2574ae08b428b3b4d1613f94a77d\": container with ID starting with ed477160bd1932351c9937159829e87735da2574ae08b428b3b4d1613f94a77d not found: ID does not exist" Nov 03 13:56:03.071637 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:56:03.071608 5024 scope.go:122] "RemoveContainer" containerID="25eee8edb6616919fc7b7278bf3a8576e1cba95d2d99c29edbf4486a0bb885fa" Nov 03 13:56:03.082661 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:56:03.082644 5024 scope.go:122] "RemoveContainer" containerID="25eee8edb6616919fc7b7278bf3a8576e1cba95d2d99c29edbf4486a0bb885fa" Nov 03 13:56:03.082900 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:56:03.082882 5024 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"25eee8edb6616919fc7b7278bf3a8576e1cba95d2d99c29edbf4486a0bb885fa\": container with ID starting with 25eee8edb6616919fc7b7278bf3a8576e1cba95d2d99c29edbf4486a0bb885fa not found: ID does not exist" containerID="25eee8edb6616919fc7b7278bf3a8576e1cba95d2d99c29edbf4486a0bb885fa" Nov 03 13:56:03.082985 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:56:03.082908 5024 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25eee8edb6616919fc7b7278bf3a8576e1cba95d2d99c29edbf4486a0bb885fa"} err="failed to get container status \"25eee8edb6616919fc7b7278bf3a8576e1cba95d2d99c29edbf4486a0bb885fa\": rpc error: code = NotFound desc = could not find container \"25eee8edb6616919fc7b7278bf3a8576e1cba95d2d99c29edbf4486a0bb885fa\": container with ID starting with 25eee8edb6616919fc7b7278bf3a8576e1cba95d2d99c29edbf4486a0bb885fa not found: ID does not exist" Nov 03 13:56:03.097577 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:56:03.097528 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d0441c-66875c8bc9-tw57h" podStartSLOduration=1.575586488 podStartE2EDuration="4.097512983s" totalImagesPullingTime="2.521926495s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:55:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:56:03.082456294 +0000 UTC m=+2843.090647570" watchObservedRunningTime="2026-11-03 13:56:03.097512983 +0000 UTC m=+2843.105704260" Nov 03 13:56:03.099001 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:56:03.098983 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-d0441c-54cccc9767-t76cc"] Nov 03 13:56:03.104117 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:56:03.104097 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-d0441c-54cccc9767-t76cc"] Nov 03 13:56:03.115328 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:56:03.115306 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-d0441c-5678568b5-zwjch"] Nov 03 13:56:03.120069 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:56:03.120047 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-d0441c-5678568b5-zwjch"] Nov 03 13:56:03.135542 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:56:03.135501 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-d0441c-7776b4999d-6kzsf" podStartSLOduration=1.575908111 podStartE2EDuration="4.135490099s" totalImagesPullingTime="2.559581988s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:55:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:56:03.134248914 +0000 UTC m=+2843.142440192" watchObservedRunningTime="2026-11-03 13:56:03.135490099 +0000 UTC m=+2843.143681398" Nov 03 13:56:04.107057 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:56:04.107021 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b6a39ee-06ce-4f0b-8808-053cc47e48b6" path="/var/lib/kubelet/pods/0b6a39ee-06ce-4f0b-8808-053cc47e48b6/volumes" Nov 03 13:56:04.107546 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:56:04.107414 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac3eda30-7046-4c7d-8b64-db922dc586a2" path="/var/lib/kubelet/pods/ac3eda30-7046-4c7d-8b64-db922dc586a2/volumes" Nov 03 13:56:09.021750 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:56:09.021709 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a9c0c6e9-frrsm"] Nov 03 13:56:09.023992 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:56:09.023968 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a9c0c6e9-frrsm"] Nov 03 13:56:09.507211 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:56:09.507171 5024 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d7mwz" containerName="extractor" Nov 03 13:56:10.108655 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:56:10.108623 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0919e155-6632-4b77-bccd-de89cc4847ad" path="/var/lib/kubelet/pods/0919e155-6632-4b77-bccd-de89cc4847ad/volumes" Nov 03 13:56:14.067746 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:56:14.067709 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-d0441c-7776b4999d-6kzsf" Nov 03 13:56:14.068222 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:56:14.067931 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d0441c-66875c8bc9-tw57h" Nov 03 13:56:19.368429 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:56:19.368381 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-4d1115-openshift-default-59d4f94977-rmjd7"] Nov 03 13:56:19.369076 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:56:19.369052 5024 cpu_manager.go:414] "containerMap: removing container" podUID="ac3eda30-7046-4c7d-8b64-db922dc586a2" containerName="payload-processing" Nov 03 13:56:19.369175 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:56:19.369078 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac3eda30-7046-4c7d-8b64-db922dc586a2" containerName="payload-processing" Nov 03 13:56:19.369175 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:56:19.369105 5024 cpu_manager.go:414] "containerMap: removing container" podUID="0b6a39ee-06ce-4f0b-8808-053cc47e48b6" containerName="payload-pre-processing" Nov 03 13:56:19.369175 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:56:19.369114 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b6a39ee-06ce-4f0b-8808-053cc47e48b6" containerName="payload-pre-processing" Nov 03 13:56:19.369361 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:56:19.369272 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac3eda30-7046-4c7d-8b64-db922dc586a2" containerName="payload-processing" Nov 03 13:56:19.369361 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:56:19.369304 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b6a39ee-06ce-4f0b-8808-053cc47e48b6" containerName="payload-pre-processing" Nov 03 13:56:19.374300 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:56:19.374264 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-4d1115-openshift-default-59d4f94977-rmjd7" Nov 03 13:56:19.384002 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:56:19.383921 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-4d1115-openshift-default-59d4f94977-rmjd7"] Nov 03 13:56:19.384581 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:56:19.384555 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-4d1115-openshift-default-dockercfg-j9zwf\"" Nov 03 13:56:19.448624 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:56:19.448580 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76f44df8-f07c-4c0f-9405-18d2a3b576fb-workload-socket\") pod \"e2e-ipp-b-4d1115-openshift-default-59d4f94977-rmjd7\" (UID: \"76f44df8-f07c-4c0f-9405-18d2a3b576fb\") " pod="openshift-ingress/e2e-ipp-b-4d1115-openshift-default-59d4f94977-rmjd7" Nov 03 13:56:19.449127 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:56:19.449057 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76f44df8-f07c-4c0f-9405-18d2a3b576fb-istio-podinfo\") pod \"e2e-ipp-b-4d1115-openshift-default-59d4f94977-rmjd7\" (UID: \"76f44df8-f07c-4c0f-9405-18d2a3b576fb\") " pod="openshift-ingress/e2e-ipp-b-4d1115-openshift-default-59d4f94977-rmjd7" Nov 03 13:56:19.449317 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:56:19.449213 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76f44df8-f07c-4c0f-9405-18d2a3b576fb-credential-socket\") pod \"e2e-ipp-b-4d1115-openshift-default-59d4f94977-rmjd7\" (UID: \"76f44df8-f07c-4c0f-9405-18d2a3b576fb\") " pod="openshift-ingress/e2e-ipp-b-4d1115-openshift-default-59d4f94977-rmjd7" Nov 03 13:56:19.449449 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:56:19.449425 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76f44df8-f07c-4c0f-9405-18d2a3b576fb-istiod-ca-cert\") pod \"e2e-ipp-b-4d1115-openshift-default-59d4f94977-rmjd7\" (UID: \"76f44df8-f07c-4c0f-9405-18d2a3b576fb\") " pod="openshift-ingress/e2e-ipp-b-4d1115-openshift-default-59d4f94977-rmjd7" Nov 03 13:56:19.449545 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:56:19.449480 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76f44df8-f07c-4c0f-9405-18d2a3b576fb-workload-certs\") pod \"e2e-ipp-b-4d1115-openshift-default-59d4f94977-rmjd7\" (UID: \"76f44df8-f07c-4c0f-9405-18d2a3b576fb\") " pod="openshift-ingress/e2e-ipp-b-4d1115-openshift-default-59d4f94977-rmjd7" Nov 03 13:56:19.449635 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:56:19.449611 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4mpzf\" (UniqueName: \"kubernetes.io/projected/76f44df8-f07c-4c0f-9405-18d2a3b576fb-kube-api-access-4mpzf\") pod \"e2e-ipp-b-4d1115-openshift-default-59d4f94977-rmjd7\" (UID: \"76f44df8-f07c-4c0f-9405-18d2a3b576fb\") " pod="openshift-ingress/e2e-ipp-b-4d1115-openshift-default-59d4f94977-rmjd7" Nov 03 13:56:19.449699 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:56:19.449652 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76f44df8-f07c-4c0f-9405-18d2a3b576fb-istio-envoy\") pod \"e2e-ipp-b-4d1115-openshift-default-59d4f94977-rmjd7\" (UID: \"76f44df8-f07c-4c0f-9405-18d2a3b576fb\") " pod="openshift-ingress/e2e-ipp-b-4d1115-openshift-default-59d4f94977-rmjd7" Nov 03 13:56:19.449699 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:56:19.449687 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76f44df8-f07c-4c0f-9405-18d2a3b576fb-istio-data\") pod \"e2e-ipp-b-4d1115-openshift-default-59d4f94977-rmjd7\" (UID: \"76f44df8-f07c-4c0f-9405-18d2a3b576fb\") " pod="openshift-ingress/e2e-ipp-b-4d1115-openshift-default-59d4f94977-rmjd7" Nov 03 13:56:19.449820 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:56:19.449730 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76f44df8-f07c-4c0f-9405-18d2a3b576fb-istio-token\") pod \"e2e-ipp-b-4d1115-openshift-default-59d4f94977-rmjd7\" (UID: \"76f44df8-f07c-4c0f-9405-18d2a3b576fb\") " pod="openshift-ingress/e2e-ipp-b-4d1115-openshift-default-59d4f94977-rmjd7" Nov 03 13:56:19.551081 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:56:19.551043 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76f44df8-f07c-4c0f-9405-18d2a3b576fb-workload-certs\") pod \"e2e-ipp-b-4d1115-openshift-default-59d4f94977-rmjd7\" (UID: \"76f44df8-f07c-4c0f-9405-18d2a3b576fb\") " pod="openshift-ingress/e2e-ipp-b-4d1115-openshift-default-59d4f94977-rmjd7" Nov 03 13:56:19.551267 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:56:19.551097 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4mpzf\" (UniqueName: \"kubernetes.io/projected/76f44df8-f07c-4c0f-9405-18d2a3b576fb-kube-api-access-4mpzf\") pod \"e2e-ipp-b-4d1115-openshift-default-59d4f94977-rmjd7\" (UID: \"76f44df8-f07c-4c0f-9405-18d2a3b576fb\") " pod="openshift-ingress/e2e-ipp-b-4d1115-openshift-default-59d4f94977-rmjd7" Nov 03 13:56:19.551267 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:56:19.551212 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76f44df8-f07c-4c0f-9405-18d2a3b576fb-istio-envoy\") pod \"e2e-ipp-b-4d1115-openshift-default-59d4f94977-rmjd7\" (UID: \"76f44df8-f07c-4c0f-9405-18d2a3b576fb\") " pod="openshift-ingress/e2e-ipp-b-4d1115-openshift-default-59d4f94977-rmjd7" Nov 03 13:56:19.551267 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:56:19.551256 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76f44df8-f07c-4c0f-9405-18d2a3b576fb-istio-data\") pod \"e2e-ipp-b-4d1115-openshift-default-59d4f94977-rmjd7\" (UID: \"76f44df8-f07c-4c0f-9405-18d2a3b576fb\") " pod="openshift-ingress/e2e-ipp-b-4d1115-openshift-default-59d4f94977-rmjd7" Nov 03 13:56:19.551433 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:56:19.551321 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76f44df8-f07c-4c0f-9405-18d2a3b576fb-istio-token\") pod \"e2e-ipp-b-4d1115-openshift-default-59d4f94977-rmjd7\" (UID: \"76f44df8-f07c-4c0f-9405-18d2a3b576fb\") " pod="openshift-ingress/e2e-ipp-b-4d1115-openshift-default-59d4f94977-rmjd7" Nov 03 13:56:19.551433 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:56:19.551362 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76f44df8-f07c-4c0f-9405-18d2a3b576fb-workload-socket\") pod \"e2e-ipp-b-4d1115-openshift-default-59d4f94977-rmjd7\" (UID: \"76f44df8-f07c-4c0f-9405-18d2a3b576fb\") " pod="openshift-ingress/e2e-ipp-b-4d1115-openshift-default-59d4f94977-rmjd7" Nov 03 13:56:19.551516 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:56:19.551437 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76f44df8-f07c-4c0f-9405-18d2a3b576fb-istio-podinfo\") pod \"e2e-ipp-b-4d1115-openshift-default-59d4f94977-rmjd7\" (UID: \"76f44df8-f07c-4c0f-9405-18d2a3b576fb\") " pod="openshift-ingress/e2e-ipp-b-4d1115-openshift-default-59d4f94977-rmjd7" Nov 03 13:56:19.551579 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:56:19.551513 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76f44df8-f07c-4c0f-9405-18d2a3b576fb-credential-socket\") pod \"e2e-ipp-b-4d1115-openshift-default-59d4f94977-rmjd7\" (UID: \"76f44df8-f07c-4c0f-9405-18d2a3b576fb\") " pod="openshift-ingress/e2e-ipp-b-4d1115-openshift-default-59d4f94977-rmjd7" Nov 03 13:56:19.551579 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:56:19.551523 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76f44df8-f07c-4c0f-9405-18d2a3b576fb-workload-certs\") pod \"e2e-ipp-b-4d1115-openshift-default-59d4f94977-rmjd7\" (UID: \"76f44df8-f07c-4c0f-9405-18d2a3b576fb\") " pod="openshift-ingress/e2e-ipp-b-4d1115-openshift-default-59d4f94977-rmjd7" Nov 03 13:56:19.551703 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:56:19.551598 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76f44df8-f07c-4c0f-9405-18d2a3b576fb-istiod-ca-cert\") pod \"e2e-ipp-b-4d1115-openshift-default-59d4f94977-rmjd7\" (UID: \"76f44df8-f07c-4c0f-9405-18d2a3b576fb\") " pod="openshift-ingress/e2e-ipp-b-4d1115-openshift-default-59d4f94977-rmjd7" Nov 03 13:56:19.551703 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:56:19.551607 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76f44df8-f07c-4c0f-9405-18d2a3b576fb-istio-data\") pod \"e2e-ipp-b-4d1115-openshift-default-59d4f94977-rmjd7\" (UID: \"76f44df8-f07c-4c0f-9405-18d2a3b576fb\") " pod="openshift-ingress/e2e-ipp-b-4d1115-openshift-default-59d4f94977-rmjd7" Nov 03 13:56:19.551823 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:56:19.551796 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76f44df8-f07c-4c0f-9405-18d2a3b576fb-workload-socket\") pod \"e2e-ipp-b-4d1115-openshift-default-59d4f94977-rmjd7\" (UID: \"76f44df8-f07c-4c0f-9405-18d2a3b576fb\") " pod="openshift-ingress/e2e-ipp-b-4d1115-openshift-default-59d4f94977-rmjd7" Nov 03 13:56:19.551875 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:56:19.551852 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76f44df8-f07c-4c0f-9405-18d2a3b576fb-credential-socket\") pod \"e2e-ipp-b-4d1115-openshift-default-59d4f94977-rmjd7\" (UID: \"76f44df8-f07c-4c0f-9405-18d2a3b576fb\") " pod="openshift-ingress/e2e-ipp-b-4d1115-openshift-default-59d4f94977-rmjd7" Nov 03 13:56:19.552319 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:56:19.552277 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76f44df8-f07c-4c0f-9405-18d2a3b576fb-istiod-ca-cert\") pod \"e2e-ipp-b-4d1115-openshift-default-59d4f94977-rmjd7\" (UID: \"76f44df8-f07c-4c0f-9405-18d2a3b576fb\") " pod="openshift-ingress/e2e-ipp-b-4d1115-openshift-default-59d4f94977-rmjd7" Nov 03 13:56:19.553710 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:56:19.553687 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76f44df8-f07c-4c0f-9405-18d2a3b576fb-istio-envoy\") pod \"e2e-ipp-b-4d1115-openshift-default-59d4f94977-rmjd7\" (UID: \"76f44df8-f07c-4c0f-9405-18d2a3b576fb\") " pod="openshift-ingress/e2e-ipp-b-4d1115-openshift-default-59d4f94977-rmjd7" Nov 03 13:56:19.554108 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:56:19.554091 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76f44df8-f07c-4c0f-9405-18d2a3b576fb-istio-podinfo\") pod \"e2e-ipp-b-4d1115-openshift-default-59d4f94977-rmjd7\" (UID: \"76f44df8-f07c-4c0f-9405-18d2a3b576fb\") " pod="openshift-ingress/e2e-ipp-b-4d1115-openshift-default-59d4f94977-rmjd7" Nov 03 13:56:19.560732 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:56:19.560698 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76f44df8-f07c-4c0f-9405-18d2a3b576fb-istio-token\") pod \"e2e-ipp-b-4d1115-openshift-default-59d4f94977-rmjd7\" (UID: \"76f44df8-f07c-4c0f-9405-18d2a3b576fb\") " pod="openshift-ingress/e2e-ipp-b-4d1115-openshift-default-59d4f94977-rmjd7" Nov 03 13:56:19.560881 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:56:19.560862 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4mpzf\" (UniqueName: \"kubernetes.io/projected/76f44df8-f07c-4c0f-9405-18d2a3b576fb-kube-api-access-4mpzf\") pod \"e2e-ipp-b-4d1115-openshift-default-59d4f94977-rmjd7\" (UID: \"76f44df8-f07c-4c0f-9405-18d2a3b576fb\") " pod="openshift-ingress/e2e-ipp-b-4d1115-openshift-default-59d4f94977-rmjd7" Nov 03 13:56:19.692067 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:56:19.691964 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-4d1115-openshift-default-59d4f94977-rmjd7" Nov 03 13:56:19.974914 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:56:19.974880 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-4d1115-openshift-default-59d4f94977-rmjd7"] Nov 03 13:56:19.975913 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:56:19.975888 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76f44df8_f07c_4c0f_9405_18d2a3b576fb.slice/crio-a74231a166dca018e9b568ebcaea4d7595ef5239757bb7333ed81319c6d657cb WatchSource:0}: Error finding container a74231a166dca018e9b568ebcaea4d7595ef5239757bb7333ed81319c6d657cb: Status 404 returned error can't find the container with id a74231a166dca018e9b568ebcaea4d7595ef5239757bb7333ed81319c6d657cb Nov 03 13:56:19.978523 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:56:19.978489 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:56:19.978606 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:56:19.978562 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:56:19.978606 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:56:19.978592 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:56:20.166121 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:56:20.166067 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-4d1115-openshift-default-59d4f94977-rmjd7" event={"ID":"76f44df8-f07c-4c0f-9405-18d2a3b576fb","Type":"ContainerStarted","Data":"4e2f78a63cfeb387fde0d709b923a04821ac5151219d772b075f0ca1483c281d"} Nov 03 13:56:20.166342 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:56:20.166194 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-4d1115-openshift-default-59d4f94977-rmjd7" event={"ID":"76f44df8-f07c-4c0f-9405-18d2a3b576fb","Type":"ContainerStarted","Data":"a74231a166dca018e9b568ebcaea4d7595ef5239757bb7333ed81319c6d657cb"} Nov 03 13:56:20.192802 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:56:20.192734 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-4d1115-openshift-default-59d4f94977-rmjd7" podStartSLOduration=1.192716011 podStartE2EDuration="1.192716011s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:56:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:56:20.190961841 +0000 UTC m=+2860.199153118" watchObservedRunningTime="2026-11-03 13:56:20.192716011 +0000 UTC m=+2860.200907290" Nov 03 13:56:20.692389 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:56:20.692348 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-4d1115-openshift-default-59d4f94977-rmjd7" Nov 03 13:56:21.698153 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:56:21.698117 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-4d1115-openshift-default-59d4f94977-rmjd7" Nov 03 13:56:22.180059 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:56:22.180027 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-4d1115-openshift-default-59d4f94977-rmjd7" Nov 03 13:56:22.180951 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:56:22.180921 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-4d1115-openshift-default-59d4f94977-rmjd7" Nov 03 13:56:26.260576 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:56:26.260536 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-4d1115-6f8dd857bf-bgrsm"] Nov 03 13:56:26.265204 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:56:26.265176 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4d1115-6f8dd857bf-bgrsm" Nov 03 13:56:26.278208 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:56:26.273412 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-4d1115-serving-cert\"" Nov 03 13:56:26.278208 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:56:26.277152 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-4d1115-6f8dd857bf-bgrsm"] Nov 03 13:56:26.334730 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:56:26.334695 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c8793d34-41c6-4aaf-a675-19c9f080d516-maas-api-tls\") pod \"maas-api-e2e-ipp-b-4d1115-6f8dd857bf-bgrsm\" (UID: \"c8793d34-41c6-4aaf-a675-19c9f080d516\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4d1115-6f8dd857bf-bgrsm" Nov 03 13:56:26.334948 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:56:26.334735 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gc8b7\" (UniqueName: \"kubernetes.io/projected/c8793d34-41c6-4aaf-a675-19c9f080d516-kube-api-access-gc8b7\") pod \"maas-api-e2e-ipp-b-4d1115-6f8dd857bf-bgrsm\" (UID: \"c8793d34-41c6-4aaf-a675-19c9f080d516\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4d1115-6f8dd857bf-bgrsm" Nov 03 13:56:26.334948 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:56:26.334883 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c8793d34-41c6-4aaf-a675-19c9f080d516-metrics-certs\") pod \"maas-api-e2e-ipp-b-4d1115-6f8dd857bf-bgrsm\" (UID: \"c8793d34-41c6-4aaf-a675-19c9f080d516\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4d1115-6f8dd857bf-bgrsm" Nov 03 13:56:26.335055 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:56:26.335005 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8793d34-41c6-4aaf-a675-19c9f080d516-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-4d1115-6f8dd857bf-bgrsm\" (UID: \"c8793d34-41c6-4aaf-a675-19c9f080d516\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4d1115-6f8dd857bf-bgrsm" Nov 03 13:56:26.436516 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:56:26.436480 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8793d34-41c6-4aaf-a675-19c9f080d516-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-4d1115-6f8dd857bf-bgrsm\" (UID: \"c8793d34-41c6-4aaf-a675-19c9f080d516\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4d1115-6f8dd857bf-bgrsm" Nov 03 13:56:26.436703 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:56:26.436569 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c8793d34-41c6-4aaf-a675-19c9f080d516-maas-api-tls\") pod \"maas-api-e2e-ipp-b-4d1115-6f8dd857bf-bgrsm\" (UID: \"c8793d34-41c6-4aaf-a675-19c9f080d516\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4d1115-6f8dd857bf-bgrsm" Nov 03 13:56:26.436703 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:56:26.436610 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gc8b7\" (UniqueName: \"kubernetes.io/projected/c8793d34-41c6-4aaf-a675-19c9f080d516-kube-api-access-gc8b7\") pod \"maas-api-e2e-ipp-b-4d1115-6f8dd857bf-bgrsm\" (UID: \"c8793d34-41c6-4aaf-a675-19c9f080d516\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4d1115-6f8dd857bf-bgrsm" Nov 03 13:56:26.436824 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:56:26.436701 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c8793d34-41c6-4aaf-a675-19c9f080d516-metrics-certs\") pod \"maas-api-e2e-ipp-b-4d1115-6f8dd857bf-bgrsm\" (UID: \"c8793d34-41c6-4aaf-a675-19c9f080d516\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4d1115-6f8dd857bf-bgrsm" Nov 03 13:56:26.436824 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:56:26.436705 5024 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-4d1115-serving-cert: secret "maas-api-e2e-ipp-b-4d1115-serving-cert" not found Nov 03 13:56:26.436824 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:56:26.436776 5024 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c8793d34-41c6-4aaf-a675-19c9f080d516-maas-api-tls podName:c8793d34-41c6-4aaf-a675-19c9f080d516 nodeName:}" failed. No retries permitted until 2026-11-03 13:56:26.936758171 +0000 UTC m=+2866.944949452 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c8793d34-41c6-4aaf-a675-19c9f080d516-maas-api-tls") pod "maas-api-e2e-ipp-b-4d1115-6f8dd857bf-bgrsm" (UID: "c8793d34-41c6-4aaf-a675-19c9f080d516") : secret "maas-api-e2e-ipp-b-4d1115-serving-cert" not found Nov 03 13:56:26.437194 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:56:26.437171 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8793d34-41c6-4aaf-a675-19c9f080d516-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-4d1115-6f8dd857bf-bgrsm\" (UID: \"c8793d34-41c6-4aaf-a675-19c9f080d516\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4d1115-6f8dd857bf-bgrsm" Nov 03 13:56:26.439222 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:56:26.439203 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c8793d34-41c6-4aaf-a675-19c9f080d516-metrics-certs\") pod \"maas-api-e2e-ipp-b-4d1115-6f8dd857bf-bgrsm\" (UID: \"c8793d34-41c6-4aaf-a675-19c9f080d516\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4d1115-6f8dd857bf-bgrsm" Nov 03 13:56:26.447541 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:56:26.447517 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gc8b7\" (UniqueName: \"kubernetes.io/projected/c8793d34-41c6-4aaf-a675-19c9f080d516-kube-api-access-gc8b7\") pod \"maas-api-e2e-ipp-b-4d1115-6f8dd857bf-bgrsm\" (UID: \"c8793d34-41c6-4aaf-a675-19c9f080d516\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4d1115-6f8dd857bf-bgrsm" Nov 03 13:56:26.950021 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:56:26.949979 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c8793d34-41c6-4aaf-a675-19c9f080d516-maas-api-tls\") pod \"maas-api-e2e-ipp-b-4d1115-6f8dd857bf-bgrsm\" (UID: \"c8793d34-41c6-4aaf-a675-19c9f080d516\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4d1115-6f8dd857bf-bgrsm" Nov 03 13:56:26.952533 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:56:26.952504 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c8793d34-41c6-4aaf-a675-19c9f080d516-maas-api-tls\") pod \"maas-api-e2e-ipp-b-4d1115-6f8dd857bf-bgrsm\" (UID: \"c8793d34-41c6-4aaf-a675-19c9f080d516\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4d1115-6f8dd857bf-bgrsm" Nov 03 13:56:27.185130 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:56:27.185095 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4d1115-6f8dd857bf-bgrsm" Nov 03 13:56:27.464309 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:56:27.464262 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-4d1115-6f8dd857bf-bgrsm"] Nov 03 13:56:27.465790 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:56:27.465755 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8793d34_41c6_4aaf_a675_19c9f080d516.slice/crio-b80611fdc8ae0b710afd619350cfd937adfeee4f920b3addf09a1157084c07cb WatchSource:0}: Error finding container b80611fdc8ae0b710afd619350cfd937adfeee4f920b3addf09a1157084c07cb: Status 404 returned error can't find the container with id b80611fdc8ae0b710afd619350cfd937adfeee4f920b3addf09a1157084c07cb Nov 03 13:56:27.467237 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:56:27.467217 5024 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 13:56:28.216395 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:56:28.216299 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4d1115-6f8dd857bf-bgrsm" event={"ID":"c8793d34-41c6-4aaf-a675-19c9f080d516","Type":"ContainerStarted","Data":"ee134fdc65203ab4eb9624e663e6cab53d57d2bbba7654ee63d6887367e47a5b"} Nov 03 13:56:28.216395 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:56:28.216367 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4d1115-6f8dd857bf-bgrsm" event={"ID":"c8793d34-41c6-4aaf-a675-19c9f080d516","Type":"ContainerStarted","Data":"b80611fdc8ae0b710afd619350cfd937adfeee4f920b3addf09a1157084c07cb"} Nov 03 13:56:28.216841 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:56:28.216408 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4d1115-6f8dd857bf-bgrsm" Nov 03 13:56:28.237862 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:56:28.237795 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4d1115-6f8dd857bf-bgrsm" podStartSLOduration=1.786159437 podStartE2EDuration="2.237775905s" totalImagesPullingTime="451.616468ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:56:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:56:28.234882363 +0000 UTC m=+2868.243073642" watchObservedRunningTime="2026-11-03 13:56:28.237775905 +0000 UTC m=+2868.245967183" Nov 03 13:56:34.229516 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:56:34.229484 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4d1115-6f8dd857bf-bgrsm" Nov 03 13:56:38.144355 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:56:38.143663 5024 patch_prober.go:29] interesting pod/e2e-ipp-b-4d1115-openshift-default-59d4f94977-rmjd7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.210:15021/healthz/ready\": read tcp 10.132.0.2:47184->10.132.0.210:15021: read: connection reset by peer" start-of-body= Nov 03 13:56:38.144355 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:56:38.143731 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-4d1115-openshift-default-59d4f94977-rmjd7" podUID="76f44df8-f07c-4c0f-9405-18d2a3b576fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.210:15021/healthz/ready\": read tcp 10.132.0.2:47184->10.132.0.210:15021: read: connection reset by peer" Nov 03 13:56:38.295476 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:56:38.295441 5024 generic.go:356] "Generic (PLEG): container finished" podID="76f44df8-f07c-4c0f-9405-18d2a3b576fb" containerID="4e2f78a63cfeb387fde0d709b923a04821ac5151219d772b075f0ca1483c281d" exitCode=0 Nov 03 13:56:38.295646 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:56:38.295496 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-4d1115-openshift-default-59d4f94977-rmjd7" event={"ID":"76f44df8-f07c-4c0f-9405-18d2a3b576fb","Type":"ContainerDied","Data":"4e2f78a63cfeb387fde0d709b923a04821ac5151219d772b075f0ca1483c281d"} Nov 03 13:56:38.295862 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:56:38.295849 5024 scope.go:122] "RemoveContainer" containerID="4e2f78a63cfeb387fde0d709b923a04821ac5151219d772b075f0ca1483c281d" Nov 03 13:56:38.297107 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:56:38.297074 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:56:38.297171 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:56:38.297148 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:56:38.297208 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:56:38.297182 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:56:39.305494 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:56:39.305453 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-4d1115-openshift-default-59d4f94977-rmjd7" event={"ID":"76f44df8-f07c-4c0f-9405-18d2a3b576fb","Type":"ContainerStarted","Data":"9626726f9fd3389f8b8f00b102053b7d14b0981d55c95f508acea9eca57d7319"} Nov 03 13:56:39.522409 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:56:39.522364 5024 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d7mwz" containerName="extractor" Nov 03 13:56:39.695489 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:56:39.695445 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-4d1115-openshift-default-59d4f94977-rmjd7" Nov 03 13:56:39.695714 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:56:39.695683 5024 patch_prober.go:29] interesting pod/e2e-ipp-b-4d1115-openshift-default-59d4f94977-rmjd7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" start-of-body= Nov 03 13:56:39.695781 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:56:39.695746 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-4d1115-openshift-default-59d4f94977-rmjd7" podUID="76f44df8-f07c-4c0f-9405-18d2a3b576fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" Nov 03 13:56:40.504753 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:56:40.504711 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-4d1115-7fc4d545cf-8vqz6"] Nov 03 13:56:40.508602 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:56:40.508583 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4d1115-7fc4d545cf-8vqz6" Nov 03 13:56:40.517388 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:56:40.517354 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-4d1115\"" Nov 03 13:56:40.519249 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:56:40.519208 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-4d1115-dockercfg-jgwwh\"" Nov 03 13:56:40.522088 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:56:40.522064 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-4d1115-7fc4d545cf-8vqz6"] Nov 03 13:56:40.621776 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:56:40.621731 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8nmcx\" (UniqueName: \"kubernetes.io/projected/933767da-c7c7-4982-bddb-5629094464f1-kube-api-access-8nmcx\") pod \"payload-pre-processing-e2e-ipp-b-4d1115-7fc4d545cf-8vqz6\" (UID: \"933767da-c7c7-4982-bddb-5629094464f1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4d1115-7fc4d545cf-8vqz6" Nov 03 13:56:40.622135 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:56:40.622109 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/933767da-c7c7-4982-bddb-5629094464f1-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-4d1115-7fc4d545cf-8vqz6\" (UID: \"933767da-c7c7-4982-bddb-5629094464f1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4d1115-7fc4d545cf-8vqz6" Nov 03 13:56:40.623814 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:56:40.623763 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-4d1115-578bff4f88-fb7gg"] Nov 03 13:56:40.628327 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:56:40.628305 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-4d1115-578bff4f88-fb7gg"] Nov 03 13:56:40.628429 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:56:40.628401 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-4d1115-578bff4f88-fb7gg" Nov 03 13:56:40.695307 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:56:40.694730 5024 patch_prober.go:29] interesting pod/e2e-ipp-b-4d1115-openshift-default-59d4f94977-rmjd7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" start-of-body= Nov 03 13:56:40.695307 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:56:40.694798 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-4d1115-openshift-default-59d4f94977-rmjd7" podUID="76f44df8-f07c-4c0f-9405-18d2a3b576fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" Nov 03 13:56:40.723771 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:56:40.723723 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8nmcx\" (UniqueName: \"kubernetes.io/projected/933767da-c7c7-4982-bddb-5629094464f1-kube-api-access-8nmcx\") pod \"payload-pre-processing-e2e-ipp-b-4d1115-7fc4d545cf-8vqz6\" (UID: \"933767da-c7c7-4982-bddb-5629094464f1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4d1115-7fc4d545cf-8vqz6" Nov 03 13:56:40.723959 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:56:40.723798 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-psm76\" (UniqueName: \"kubernetes.io/projected/ef470bd5-cb90-49f7-a229-b7a4f64a0c3c-kube-api-access-psm76\") pod \"payload-processing-e2e-ipp-b-4d1115-578bff4f88-fb7gg\" (UID: \"ef470bd5-cb90-49f7-a229-b7a4f64a0c3c\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-4d1115-578bff4f88-fb7gg" Nov 03 13:56:40.723959 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:56:40.723862 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/933767da-c7c7-4982-bddb-5629094464f1-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-4d1115-7fc4d545cf-8vqz6\" (UID: \"933767da-c7c7-4982-bddb-5629094464f1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4d1115-7fc4d545cf-8vqz6" Nov 03 13:56:40.724051 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:56:40.723964 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef470bd5-cb90-49f7-a229-b7a4f64a0c3c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-4d1115-578bff4f88-fb7gg\" (UID: \"ef470bd5-cb90-49f7-a229-b7a4f64a0c3c\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-4d1115-578bff4f88-fb7gg" Nov 03 13:56:40.724600 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:56:40.724574 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/933767da-c7c7-4982-bddb-5629094464f1-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-4d1115-7fc4d545cf-8vqz6\" (UID: \"933767da-c7c7-4982-bddb-5629094464f1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4d1115-7fc4d545cf-8vqz6" Nov 03 13:56:40.733257 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:56:40.733231 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8nmcx\" (UniqueName: \"kubernetes.io/projected/933767da-c7c7-4982-bddb-5629094464f1-kube-api-access-8nmcx\") pod \"payload-pre-processing-e2e-ipp-b-4d1115-7fc4d545cf-8vqz6\" (UID: \"933767da-c7c7-4982-bddb-5629094464f1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4d1115-7fc4d545cf-8vqz6" Nov 03 13:56:40.822536 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:56:40.822440 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4d1115-7fc4d545cf-8vqz6" Nov 03 13:56:40.824868 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:56:40.824836 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef470bd5-cb90-49f7-a229-b7a4f64a0c3c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-4d1115-578bff4f88-fb7gg\" (UID: \"ef470bd5-cb90-49f7-a229-b7a4f64a0c3c\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-4d1115-578bff4f88-fb7gg" Nov 03 13:56:40.824992 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:56:40.824957 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-psm76\" (UniqueName: \"kubernetes.io/projected/ef470bd5-cb90-49f7-a229-b7a4f64a0c3c-kube-api-access-psm76\") pod \"payload-processing-e2e-ipp-b-4d1115-578bff4f88-fb7gg\" (UID: \"ef470bd5-cb90-49f7-a229-b7a4f64a0c3c\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-4d1115-578bff4f88-fb7gg" Nov 03 13:56:40.825830 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:56:40.825534 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef470bd5-cb90-49f7-a229-b7a4f64a0c3c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-4d1115-578bff4f88-fb7gg\" (UID: \"ef470bd5-cb90-49f7-a229-b7a4f64a0c3c\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-4d1115-578bff4f88-fb7gg" Nov 03 13:56:40.834614 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:56:40.834566 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-psm76\" (UniqueName: \"kubernetes.io/projected/ef470bd5-cb90-49f7-a229-b7a4f64a0c3c-kube-api-access-psm76\") pod \"payload-processing-e2e-ipp-b-4d1115-578bff4f88-fb7gg\" (UID: \"ef470bd5-cb90-49f7-a229-b7a4f64a0c3c\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-4d1115-578bff4f88-fb7gg" Nov 03 13:56:40.949402 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:56:40.949370 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-4d1115-578bff4f88-fb7gg" Nov 03 13:56:41.122976 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:56:41.122941 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-4d1115-7fc4d545cf-8vqz6"] Nov 03 13:56:41.124051 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:56:41.124014 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod933767da_c7c7_4982_bddb_5629094464f1.slice/crio-62565515f83fbd3d250b8af5fe743decba719a8d3b6479706455e51fb5e29970 WatchSource:0}: Error finding container 62565515f83fbd3d250b8af5fe743decba719a8d3b6479706455e51fb5e29970: Status 404 returned error can't find the container with id 62565515f83fbd3d250b8af5fe743decba719a8d3b6479706455e51fb5e29970 Nov 03 13:56:41.264832 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:56:41.264805 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-4d1115-578bff4f88-fb7gg"] Nov 03 13:56:41.265837 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:56:41.265809 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef470bd5_cb90_49f7_a229_b7a4f64a0c3c.slice/crio-ff77e564eea2274d210414b58c58a8ad2cf2d3ab30f79dcf5a561aec57e262ed WatchSource:0}: Error finding container ff77e564eea2274d210414b58c58a8ad2cf2d3ab30f79dcf5a561aec57e262ed: Status 404 returned error can't find the container with id ff77e564eea2274d210414b58c58a8ad2cf2d3ab30f79dcf5a561aec57e262ed Nov 03 13:56:41.318228 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:56:41.318192 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-4d1115-578bff4f88-fb7gg" event={"ID":"ef470bd5-cb90-49f7-a229-b7a4f64a0c3c","Type":"ContainerStarted","Data":"ff77e564eea2274d210414b58c58a8ad2cf2d3ab30f79dcf5a561aec57e262ed"} Nov 03 13:56:41.319775 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:56:41.319749 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4d1115-7fc4d545cf-8vqz6" event={"ID":"933767da-c7c7-4982-bddb-5629094464f1","Type":"ContainerStarted","Data":"12c7897abc86941dad0125e783b0236ae10a8f58a25c79d76ccb1a283e2c7c43"} Nov 03 13:56:41.319921 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:56:41.319809 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4d1115-7fc4d545cf-8vqz6" event={"ID":"933767da-c7c7-4982-bddb-5629094464f1","Type":"ContainerStarted","Data":"62565515f83fbd3d250b8af5fe743decba719a8d3b6479706455e51fb5e29970"} Nov 03 13:56:41.319921 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:56:41.319899 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4d1115-7fc4d545cf-8vqz6" Nov 03 13:56:41.339533 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:56:41.339474 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4d1115-7fc4d545cf-8vqz6" podStartSLOduration=1.339456334 podStartE2EDuration="1.339456334s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:56:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:56:41.338494841 +0000 UTC m=+2881.346686117" watchObservedRunningTime="2026-11-03 13:56:41.339456334 +0000 UTC m=+2881.347647613" Nov 03 13:56:41.692734 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:56:41.692686 5024 patch_prober.go:29] interesting pod/e2e-ipp-b-4d1115-openshift-default-59d4f94977-rmjd7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" start-of-body= Nov 03 13:56:41.693343 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:56:41.693303 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-4d1115-openshift-default-59d4f94977-rmjd7" podUID="76f44df8-f07c-4c0f-9405-18d2a3b576fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" Nov 03 13:56:42.330189 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:56:42.330148 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-4d1115-578bff4f88-fb7gg" event={"ID":"ef470bd5-cb90-49f7-a229-b7a4f64a0c3c","Type":"ContainerStarted","Data":"e362ee506bd2c087dc425c3ba27f504f43541035563b3ca4811542e7066c9f60"} Nov 03 13:56:42.330402 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:56:42.330246 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-4d1115-578bff4f88-fb7gg" Nov 03 13:56:42.349199 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:56:42.349144 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-4d1115-578bff4f88-fb7gg" podStartSLOduration=2.349131093 podStartE2EDuration="2.349131093s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:56:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:56:42.348133737 +0000 UTC m=+2882.356325013" watchObservedRunningTime="2026-11-03 13:56:42.349131093 +0000 UTC m=+2882.357322402" Nov 03 13:56:42.701835 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:56:42.701782 5024 patch_prober.go:29] interesting pod/e2e-ipp-b-4d1115-openshift-default-59d4f94977-rmjd7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" start-of-body= Nov 03 13:56:42.702379 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:56:42.701857 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-4d1115-openshift-default-59d4f94977-rmjd7" podUID="76f44df8-f07c-4c0f-9405-18d2a3b576fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" Nov 03 13:56:43.695760 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:56:43.695713 5024 patch_prober.go:29] interesting pod/e2e-ipp-b-4d1115-openshift-default-59d4f94977-rmjd7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" start-of-body= Nov 03 13:56:43.695974 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:56:43.695800 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-4d1115-openshift-default-59d4f94977-rmjd7" podUID="76f44df8-f07c-4c0f-9405-18d2a3b576fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" Nov 03 13:56:44.696632 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:56:44.696597 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-4d1115-openshift-default-59d4f94977-rmjd7" Nov 03 13:56:44.697074 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:56:44.696901 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-4d1115-openshift-default-59d4f94977-rmjd7" Nov 03 13:56:44.697573 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:56:44.697549 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-4d1115-openshift-default-59d4f94977-rmjd7" Nov 03 13:56:45.993163 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:56:45.993131 5024 scope.go:122] "RemoveContainer" containerID="a60f05635491f6a858b65a039c18c13423efa3069de8c5356b613340e6db9f5f" Nov 03 13:56:46.003124 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:56:46.003100 5024 scope.go:122] "RemoveContainer" containerID="2cc39713c2ecf611cb970f890e4b1cfff92a73d04feef518d0d4f48f07e52726" Nov 03 13:56:46.014410 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:56:46.014385 5024 scope.go:122] "RemoveContainer" containerID="4480a3386279d7f4ec5991a10d26cca8b2a510e758f3ab3aa8ac7f8dfb36c18a" Nov 03 13:56:46.022862 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:56:46.022839 5024 scope.go:122] "RemoveContainer" containerID="0e7c06a760570d9d832349a69a0ccdc8e0dbc070e230c562feabfbb42f390b84" Nov 03 13:56:46.031017 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:56:46.030996 5024 scope.go:122] "RemoveContainer" containerID="5733abf9a4f0cb407abdac0bc99053883f842a2ae96889fe77e4638d0f4dd1a1" Nov 03 13:56:46.039401 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:56:46.039378 5024 scope.go:122] "RemoveContainer" containerID="32440bdebfea8f756fbcfb365e13d7dd69e77d99dd6719d8bf3cc16929229446" Nov 03 13:56:52.333213 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:56:52.333180 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4d1115-7fc4d545cf-8vqz6" Nov 03 13:56:53.338214 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:56:53.338183 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-4d1115-578bff4f88-fb7gg" Nov 03 13:56:59.404656 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:56:59.404628 5024 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-khw76_36118654-3637-489e-b048-af4aa366f34d/payload-processing/0.log" Nov 03 13:56:59.600978 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:56:59.600948 5024 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-khw76_36118654-3637-489e-b048-af4aa366f34d/payload-processing/0.log" Nov 03 13:56:59.775238 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:56:59.775153 5024 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-4d1115-578bff4f88-fb7gg_ef470bd5-cb90-49f7-a229-b7a4f64a0c3c/payload-processing/0.log" Nov 03 13:57:02.026784 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:57:02.026710 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-b5plm"] Nov 03 13:57:02.029450 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:57:02.029417 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-b5plm"] Nov 03 13:57:02.108849 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:57:02.108811 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="53e3e37b-99af-4a19-baa8-0a1d954a220d" path="/var/lib/kubelet/pods/53e3e37b-99af-4a19-baa8-0a1d954a220d/volumes" Nov 03 13:57:09.508260 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:57:09.508165 5024 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d7mwz" containerName="extractor" Nov 03 13:57:25.025765 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:57:25.025723 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-w7kkm"] Nov 03 13:57:25.028054 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:57:25.028030 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-w7kkm"] Nov 03 13:57:26.107560 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:57:26.107525 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c854eda-d4f0-47a1-bde8-8ccd7f36907a" path="/var/lib/kubelet/pods/1c854eda-d4f0-47a1-bde8-8ccd7f36907a/volumes" Nov 03 13:57:35.024873 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:57:35.024801 5024 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-d0441c-7776b4999d-6kzsf_b2072f8f-fee3-45b5-acc9-f1be7a68943d/payload-processing/0.log" Nov 03 13:57:35.205406 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:57:35.205370 5024 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-khw76_36118654-3637-489e-b048-af4aa366f34d/payload-processing/0.log" Nov 03 13:57:35.382260 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:57:35.382233 5024 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-khw76_36118654-3637-489e-b048-af4aa366f34d/payload-processing/0.log" Nov 03 13:57:35.585813 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:57:35.585776 5024 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-4d1115-578bff4f88-fb7gg_ef470bd5-cb90-49f7-a229-b7a4f64a0c3c/payload-processing/0.log" Nov 03 13:57:39.510328 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:57:39.510258 5024 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d7mwz" containerName="extractor" Nov 03 13:57:42.020614 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:57:42.020578 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-gpnsl"] Nov 03 13:57:42.022764 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:57:42.022739 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-gpnsl"] Nov 03 13:57:42.107986 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:57:42.107943 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2535a241-4e8b-479f-ad62-2645a1b19874" path="/var/lib/kubelet/pods/2535a241-4e8b-479f-ad62-2645a1b19874/volumes" Nov 03 13:57:46.089719 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:57:46.089682 5024 scope.go:122] "RemoveContainer" containerID="32743266377cb4a0d9f5b923a71523eea2de494a3486e963c83ff38fef94bc03" Nov 03 13:57:46.098630 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:57:46.098608 5024 scope.go:122] "RemoveContainer" containerID="ab1de31f036da9d5134c9dd10d23365e3fed4330366df5804af2ee642948d920" Nov 03 13:57:53.030647 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:57:53.030596 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-cwxvr"] Nov 03 13:57:53.033829 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:57:53.033797 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-cwxvr"] Nov 03 13:57:54.108202 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:57:54.108161 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="91828ac5-9a2d-4704-95dc-e8b054e891ec" path="/var/lib/kubelet/pods/91828ac5-9a2d-4704-95dc-e8b054e891ec/volumes" Nov 03 13:58:09.507654 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:58:09.507612 5024 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d7mwz" containerName="extractor" Nov 03 13:58:34.157158 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:58:34.157115 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-d0441c-7744d76cb6-pfrhs"] Nov 03 13:58:34.158220 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:58:34.157370 5024 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d0441c-7744d76cb6-pfrhs" podUID="adda3b0a-50a1-4bca-918c-a2f8dbe0f4ee" containerName="maas-api" containerID="cri-o://a4e4700e7fb8f1be599b736b177a6bb49533c15678fcb40c031c3eccf8d95cd6" gracePeriod=30 Nov 03 13:58:34.234489 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:58:34.234451 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-d0441c-7776b4999d-6kzsf"] Nov 03 13:58:34.234693 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:58:34.234671 5024 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-d0441c-7776b4999d-6kzsf" podUID="b2072f8f-fee3-45b5-acc9-f1be7a68943d" containerName="payload-processing" containerID="cri-o://e9ac87ee8698e71dd056d97a0fb454f9abe5be2dc65235038b501f9e2fcd9251" gracePeriod=30 Nov 03 13:58:34.249821 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:58:34.249787 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-d0441c-66875c8bc9-tw57h"] Nov 03 13:58:34.250081 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:58:34.250035 5024 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d0441c-66875c8bc9-tw57h" podUID="9899225c-d0e5-4426-b09a-a1009b6b109e" containerName="payload-pre-processing" containerID="cri-o://d8f7d982a67fa5ddfbcf411888f6fe32fd62044bd7900dcb418a06327203634c" gracePeriod=30 Nov 03 13:58:34.568919 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:58:34.568884 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d0441c-66875c8bc9-tw57h" Nov 03 13:58:34.599253 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:58:34.599218 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-d0441c-7776b4999d-6kzsf" Nov 03 13:58:34.627182 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:58:34.627149 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2072f8f-fee3-45b5-acc9-f1be7a68943d-kube-api-access-vx4m4\" (UniqueName: \"kubernetes.io/projected/b2072f8f-fee3-45b5-acc9-f1be7a68943d-kube-api-access-vx4m4\") pod \"b2072f8f-fee3-45b5-acc9-f1be7a68943d\" (UID: \"b2072f8f-fee3-45b5-acc9-f1be7a68943d\") " Nov 03 13:58:34.627405 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:58:34.627216 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9899225c-d0e5-4426-b09a-a1009b6b109e-kube-api-access-k8mpc\" (UniqueName: \"kubernetes.io/projected/9899225c-d0e5-4426-b09a-a1009b6b109e-kube-api-access-k8mpc\") pod \"9899225c-d0e5-4426-b09a-a1009b6b109e\" (UID: \"9899225c-d0e5-4426-b09a-a1009b6b109e\") " Nov 03 13:58:34.627405 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:58:34.627270 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b2072f8f-fee3-45b5-acc9-f1be7a68943d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2072f8f-fee3-45b5-acc9-f1be7a68943d-plugins-config-volume\") pod \"b2072f8f-fee3-45b5-acc9-f1be7a68943d\" (UID: \"b2072f8f-fee3-45b5-acc9-f1be7a68943d\") " Nov 03 13:58:34.627405 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:58:34.627352 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9899225c-d0e5-4426-b09a-a1009b6b109e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9899225c-d0e5-4426-b09a-a1009b6b109e-plugins-config-volume\") pod \"9899225c-d0e5-4426-b09a-a1009b6b109e\" (UID: \"9899225c-d0e5-4426-b09a-a1009b6b109e\") " Nov 03 13:58:34.628042 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:58:34.628014 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2072f8f-fee3-45b5-acc9-f1be7a68943d-plugins-config-volume" pod "b2072f8f-fee3-45b5-acc9-f1be7a68943d" (UID: "b2072f8f-fee3-45b5-acc9-f1be7a68943d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:58:34.628166 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:58:34.628015 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9899225c-d0e5-4426-b09a-a1009b6b109e-plugins-config-volume" pod "9899225c-d0e5-4426-b09a-a1009b6b109e" (UID: "9899225c-d0e5-4426-b09a-a1009b6b109e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:58:34.630344 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:58:34.630322 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9899225c-d0e5-4426-b09a-a1009b6b109e-kube-api-access-k8mpc" pod "9899225c-d0e5-4426-b09a-a1009b6b109e" (UID: "9899225c-d0e5-4426-b09a-a1009b6b109e"). InnerVolumeSpecName "kube-api-access-k8mpc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:58:34.630448 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:58:34.630357 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2072f8f-fee3-45b5-acc9-f1be7a68943d-kube-api-access-vx4m4" pod "b2072f8f-fee3-45b5-acc9-f1be7a68943d" (UID: "b2072f8f-fee3-45b5-acc9-f1be7a68943d"). InnerVolumeSpecName "kube-api-access-vx4m4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:58:34.656980 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:58:34.656955 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d0441c-7744d76cb6-pfrhs" Nov 03 13:58:34.728531 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:58:34.728450 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/adda3b0a-50a1-4bca-918c-a2f8dbe0f4ee-kube-api-access-zcz9k\" (UniqueName: \"kubernetes.io/projected/adda3b0a-50a1-4bca-918c-a2f8dbe0f4ee-kube-api-access-zcz9k\") pod \"adda3b0a-50a1-4bca-918c-a2f8dbe0f4ee\" (UID: \"adda3b0a-50a1-4bca-918c-a2f8dbe0f4ee\") " Nov 03 13:58:34.728531 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:58:34.728524 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/adda3b0a-50a1-4bca-918c-a2f8dbe0f4ee-metrics-certs\" (UniqueName: \"kubernetes.io/secret/adda3b0a-50a1-4bca-918c-a2f8dbe0f4ee-metrics-certs\") pod \"adda3b0a-50a1-4bca-918c-a2f8dbe0f4ee\" (UID: \"adda3b0a-50a1-4bca-918c-a2f8dbe0f4ee\") " Nov 03 13:58:34.728742 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:58:34.728597 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/adda3b0a-50a1-4bca-918c-a2f8dbe0f4ee-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/adda3b0a-50a1-4bca-918c-a2f8dbe0f4ee-openshift-service-ca\") pod \"adda3b0a-50a1-4bca-918c-a2f8dbe0f4ee\" (UID: \"adda3b0a-50a1-4bca-918c-a2f8dbe0f4ee\") " Nov 03 13:58:34.728742 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:58:34.728707 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/adda3b0a-50a1-4bca-918c-a2f8dbe0f4ee-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/adda3b0a-50a1-4bca-918c-a2f8dbe0f4ee-maas-api-tls\") pod \"adda3b0a-50a1-4bca-918c-a2f8dbe0f4ee\" (UID: \"adda3b0a-50a1-4bca-918c-a2f8dbe0f4ee\") " Nov 03 13:58:34.728952 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:58:34.728926 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/adda3b0a-50a1-4bca-918c-a2f8dbe0f4ee-openshift-service-ca" pod "adda3b0a-50a1-4bca-918c-a2f8dbe0f4ee" (UID: "adda3b0a-50a1-4bca-918c-a2f8dbe0f4ee"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:58:34.729096 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:58:34.729081 5024 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2072f8f-fee3-45b5-acc9-f1be7a68943d-plugins-config-volume\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:58:34.729177 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:58:34.729103 5024 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9899225c-d0e5-4426-b09a-a1009b6b109e-plugins-config-volume\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:58:34.729177 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:58:34.729116 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vx4m4\" (UniqueName: \"kubernetes.io/projected/b2072f8f-fee3-45b5-acc9-f1be7a68943d-kube-api-access-vx4m4\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:58:34.729177 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:58:34.729127 5024 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/adda3b0a-50a1-4bca-918c-a2f8dbe0f4ee-openshift-service-ca\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:58:34.729177 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:58:34.729136 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k8mpc\" (UniqueName: \"kubernetes.io/projected/9899225c-d0e5-4426-b09a-a1009b6b109e-kube-api-access-k8mpc\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:58:34.730850 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:58:34.730823 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/adda3b0a-50a1-4bca-918c-a2f8dbe0f4ee-metrics-certs" pod "adda3b0a-50a1-4bca-918c-a2f8dbe0f4ee" (UID: "adda3b0a-50a1-4bca-918c-a2f8dbe0f4ee"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 13:58:34.730965 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:58:34.730847 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/adda3b0a-50a1-4bca-918c-a2f8dbe0f4ee-kube-api-access-zcz9k" pod "adda3b0a-50a1-4bca-918c-a2f8dbe0f4ee" (UID: "adda3b0a-50a1-4bca-918c-a2f8dbe0f4ee"). InnerVolumeSpecName "kube-api-access-zcz9k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:58:34.730965 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:58:34.730884 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/adda3b0a-50a1-4bca-918c-a2f8dbe0f4ee-maas-api-tls" pod "adda3b0a-50a1-4bca-918c-a2f8dbe0f4ee" (UID: "adda3b0a-50a1-4bca-918c-a2f8dbe0f4ee"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 13:58:34.830777 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:58:34.830729 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zcz9k\" (UniqueName: \"kubernetes.io/projected/adda3b0a-50a1-4bca-918c-a2f8dbe0f4ee-kube-api-access-zcz9k\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:58:34.830777 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:58:34.830769 5024 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/adda3b0a-50a1-4bca-918c-a2f8dbe0f4ee-metrics-certs\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:58:34.830777 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:58:34.830781 5024 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/adda3b0a-50a1-4bca-918c-a2f8dbe0f4ee-maas-api-tls\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:58:34.973605 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:58:34.973567 5024 generic.go:356] "Generic (PLEG): container finished" podID="adda3b0a-50a1-4bca-918c-a2f8dbe0f4ee" containerID="a4e4700e7fb8f1be599b736b177a6bb49533c15678fcb40c031c3eccf8d95cd6" exitCode=0 Nov 03 13:58:34.973808 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:58:34.973620 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d0441c-7744d76cb6-pfrhs" Nov 03 13:58:34.973808 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:58:34.973658 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d0441c-7744d76cb6-pfrhs" event={"ID":"adda3b0a-50a1-4bca-918c-a2f8dbe0f4ee","Type":"ContainerDied","Data":"a4e4700e7fb8f1be599b736b177a6bb49533c15678fcb40c031c3eccf8d95cd6"} Nov 03 13:58:34.973808 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:58:34.973721 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d0441c-7744d76cb6-pfrhs" event={"ID":"adda3b0a-50a1-4bca-918c-a2f8dbe0f4ee","Type":"ContainerDied","Data":"627d927a490a5687390838be09667f43688088061ce99ffe0d96bd31918ed4b2"} Nov 03 13:58:34.973808 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:58:34.973752 5024 scope.go:122] "RemoveContainer" containerID="a4e4700e7fb8f1be599b736b177a6bb49533c15678fcb40c031c3eccf8d95cd6" Nov 03 13:58:34.975697 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:58:34.975661 5024 generic.go:356] "Generic (PLEG): container finished" podID="b2072f8f-fee3-45b5-acc9-f1be7a68943d" containerID="e9ac87ee8698e71dd056d97a0fb454f9abe5be2dc65235038b501f9e2fcd9251" exitCode=0 Nov 03 13:58:34.975841 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:58:34.975731 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-d0441c-7776b4999d-6kzsf" event={"ID":"b2072f8f-fee3-45b5-acc9-f1be7a68943d","Type":"ContainerDied","Data":"e9ac87ee8698e71dd056d97a0fb454f9abe5be2dc65235038b501f9e2fcd9251"} Nov 03 13:58:34.975841 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:58:34.975804 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-d0441c-7776b4999d-6kzsf" event={"ID":"b2072f8f-fee3-45b5-acc9-f1be7a68943d","Type":"ContainerDied","Data":"318fde18a330defc27ae04397c632df2498a1c3a9bae45bcdc3bfc3703d83d51"} Nov 03 13:58:34.975958 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:58:34.975897 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-d0441c-7776b4999d-6kzsf" Nov 03 13:58:34.985471 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:58:34.983764 5024 generic.go:356] "Generic (PLEG): container finished" podID="9899225c-d0e5-4426-b09a-a1009b6b109e" containerID="d8f7d982a67fa5ddfbcf411888f6fe32fd62044bd7900dcb418a06327203634c" exitCode=0 Nov 03 13:58:34.985471 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:58:34.983847 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d0441c-66875c8bc9-tw57h" event={"ID":"9899225c-d0e5-4426-b09a-a1009b6b109e","Type":"ContainerDied","Data":"d8f7d982a67fa5ddfbcf411888f6fe32fd62044bd7900dcb418a06327203634c"} Nov 03 13:58:34.985471 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:58:34.983916 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d0441c-66875c8bc9-tw57h" event={"ID":"9899225c-d0e5-4426-b09a-a1009b6b109e","Type":"ContainerDied","Data":"2769b145ca0963ee4c568f25e3115da6982c95670475be1634b0314cfcb0a4e0"} Nov 03 13:58:34.985471 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:58:34.984087 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d0441c-66875c8bc9-tw57h" Nov 03 13:58:34.991191 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:58:34.991075 5024 scope.go:122] "RemoveContainer" containerID="a4e4700e7fb8f1be599b736b177a6bb49533c15678fcb40c031c3eccf8d95cd6" Nov 03 13:58:34.991559 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:58:34.991526 5024 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a4e4700e7fb8f1be599b736b177a6bb49533c15678fcb40c031c3eccf8d95cd6\": container with ID starting with a4e4700e7fb8f1be599b736b177a6bb49533c15678fcb40c031c3eccf8d95cd6 not found: ID does not exist" containerID="a4e4700e7fb8f1be599b736b177a6bb49533c15678fcb40c031c3eccf8d95cd6" Nov 03 13:58:34.991660 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:58:34.991573 5024 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4e4700e7fb8f1be599b736b177a6bb49533c15678fcb40c031c3eccf8d95cd6"} err="failed to get container status \"a4e4700e7fb8f1be599b736b177a6bb49533c15678fcb40c031c3eccf8d95cd6\": rpc error: code = NotFound desc = could not find container \"a4e4700e7fb8f1be599b736b177a6bb49533c15678fcb40c031c3eccf8d95cd6\": container with ID starting with a4e4700e7fb8f1be599b736b177a6bb49533c15678fcb40c031c3eccf8d95cd6 not found: ID does not exist" Nov 03 13:58:34.991660 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:58:34.991599 5024 scope.go:122] "RemoveContainer" containerID="e9ac87ee8698e71dd056d97a0fb454f9abe5be2dc65235038b501f9e2fcd9251" Nov 03 13:58:35.002477 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:58:35.002448 5024 scope.go:122] "RemoveContainer" containerID="e9ac87ee8698e71dd056d97a0fb454f9abe5be2dc65235038b501f9e2fcd9251" Nov 03 13:58:35.003016 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:58:35.002987 5024 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e9ac87ee8698e71dd056d97a0fb454f9abe5be2dc65235038b501f9e2fcd9251\": container with ID starting with e9ac87ee8698e71dd056d97a0fb454f9abe5be2dc65235038b501f9e2fcd9251 not found: ID does not exist" containerID="e9ac87ee8698e71dd056d97a0fb454f9abe5be2dc65235038b501f9e2fcd9251" Nov 03 13:58:35.003103 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:58:35.003029 5024 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9ac87ee8698e71dd056d97a0fb454f9abe5be2dc65235038b501f9e2fcd9251"} err="failed to get container status \"e9ac87ee8698e71dd056d97a0fb454f9abe5be2dc65235038b501f9e2fcd9251\": rpc error: code = NotFound desc = could not find container \"e9ac87ee8698e71dd056d97a0fb454f9abe5be2dc65235038b501f9e2fcd9251\": container with ID starting with e9ac87ee8698e71dd056d97a0fb454f9abe5be2dc65235038b501f9e2fcd9251 not found: ID does not exist" Nov 03 13:58:35.003103 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:58:35.003054 5024 scope.go:122] "RemoveContainer" containerID="d8f7d982a67fa5ddfbcf411888f6fe32fd62044bd7900dcb418a06327203634c" Nov 03 13:58:35.011180 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:58:35.011154 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-d0441c-7744d76cb6-pfrhs"] Nov 03 13:58:35.014709 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:58:35.014681 5024 scope.go:122] "RemoveContainer" containerID="d8f7d982a67fa5ddfbcf411888f6fe32fd62044bd7900dcb418a06327203634c" Nov 03 13:58:35.014835 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:58:35.014809 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-d0441c-7744d76cb6-pfrhs"] Nov 03 13:58:35.015077 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:58:35.015052 5024 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8f7d982a67fa5ddfbcf411888f6fe32fd62044bd7900dcb418a06327203634c\": container with ID starting with d8f7d982a67fa5ddfbcf411888f6fe32fd62044bd7900dcb418a06327203634c not found: ID does not exist" containerID="d8f7d982a67fa5ddfbcf411888f6fe32fd62044bd7900dcb418a06327203634c" Nov 03 13:58:35.015149 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:58:35.015088 5024 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8f7d982a67fa5ddfbcf411888f6fe32fd62044bd7900dcb418a06327203634c"} err="failed to get container status \"d8f7d982a67fa5ddfbcf411888f6fe32fd62044bd7900dcb418a06327203634c\": rpc error: code = NotFound desc = could not find container \"d8f7d982a67fa5ddfbcf411888f6fe32fd62044bd7900dcb418a06327203634c\": container with ID starting with d8f7d982a67fa5ddfbcf411888f6fe32fd62044bd7900dcb418a06327203634c not found: ID does not exist" Nov 03 13:58:35.029056 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:58:35.029029 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-d0441c-66875c8bc9-tw57h"] Nov 03 13:58:35.032900 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:58:35.032874 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-d0441c-66875c8bc9-tw57h"] Nov 03 13:58:35.046129 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:58:35.046099 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-d0441c-7776b4999d-6kzsf"] Nov 03 13:58:35.053163 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:58:35.053137 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-d0441c-7776b4999d-6kzsf"] Nov 03 13:58:36.108058 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:58:36.108021 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9899225c-d0e5-4426-b09a-a1009b6b109e" path="/var/lib/kubelet/pods/9899225c-d0e5-4426-b09a-a1009b6b109e/volumes" Nov 03 13:58:36.108499 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:58:36.108484 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="adda3b0a-50a1-4bca-918c-a2f8dbe0f4ee" path="/var/lib/kubelet/pods/adda3b0a-50a1-4bca-918c-a2f8dbe0f4ee/volumes" Nov 03 13:58:36.108827 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:58:36.108815 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b2072f8f-fee3-45b5-acc9-f1be7a68943d" path="/var/lib/kubelet/pods/b2072f8f-fee3-45b5-acc9-f1be7a68943d/volumes" Nov 03 13:58:38.116353 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:58:38.116221 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-d0441c-openshift-default-d67bc796c-57tvj"] Nov 03 13:58:38.116704 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:58:38.116494 5024 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-d0441c-openshift-default-d67bc796c-57tvj" podUID="2019d8e6-12bd-40f8-b387-06fb66b66a08" containerName="istio-proxy" containerID="cri-o://1cd9b458925c471109059b17a691630d8cce9d19b4c89d64b2600b7801a98e7a" gracePeriod=30 Nov 03 13:58:39.507853 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:58:39.507816 5024 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d7mwz" containerName="extractor" Nov 03 13:58:41.489786 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:58:41.489753 5024 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6htht_cd91de08-084e-4667-b13f-2e07b3cff624/console-operator/1.log" Nov 03 13:58:41.503311 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:58:41.503261 5024 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6htht_cd91de08-084e-4667-b13f-2e07b3cff624/console-operator/1.log" Nov 03 13:58:43.392697 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:58:43.392633 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-d0441c-openshift-default-d67bc796c-57tvj" Nov 03 13:58:43.417904 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:58:43.417872 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2019d8e6-12bd-40f8-b387-06fb66b66a08-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2019d8e6-12bd-40f8-b387-06fb66b66a08-workload-socket\") pod \"2019d8e6-12bd-40f8-b387-06fb66b66a08\" (UID: \"2019d8e6-12bd-40f8-b387-06fb66b66a08\") " Nov 03 13:58:43.418077 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:58:43.417962 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2019d8e6-12bd-40f8-b387-06fb66b66a08-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2019d8e6-12bd-40f8-b387-06fb66b66a08-istiod-ca-cert\") pod \"2019d8e6-12bd-40f8-b387-06fb66b66a08\" (UID: \"2019d8e6-12bd-40f8-b387-06fb66b66a08\") " Nov 03 13:58:43.418077 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:58:43.417994 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2019d8e6-12bd-40f8-b387-06fb66b66a08-istio-token\" (UniqueName: \"kubernetes.io/projected/2019d8e6-12bd-40f8-b387-06fb66b66a08-istio-token\") pod \"2019d8e6-12bd-40f8-b387-06fb66b66a08\" (UID: \"2019d8e6-12bd-40f8-b387-06fb66b66a08\") " Nov 03 13:58:43.418077 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:58:43.418024 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2019d8e6-12bd-40f8-b387-06fb66b66a08-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2019d8e6-12bd-40f8-b387-06fb66b66a08-istio-envoy\") pod \"2019d8e6-12bd-40f8-b387-06fb66b66a08\" (UID: \"2019d8e6-12bd-40f8-b387-06fb66b66a08\") " Nov 03 13:58:43.418077 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:58:43.418057 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2019d8e6-12bd-40f8-b387-06fb66b66a08-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2019d8e6-12bd-40f8-b387-06fb66b66a08-credential-socket\") pod \"2019d8e6-12bd-40f8-b387-06fb66b66a08\" (UID: \"2019d8e6-12bd-40f8-b387-06fb66b66a08\") " Nov 03 13:58:43.418438 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:58:43.418129 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2019d8e6-12bd-40f8-b387-06fb66b66a08-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2019d8e6-12bd-40f8-b387-06fb66b66a08-istio-podinfo\") pod \"2019d8e6-12bd-40f8-b387-06fb66b66a08\" (UID: \"2019d8e6-12bd-40f8-b387-06fb66b66a08\") " Nov 03 13:58:43.418438 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:58:43.418204 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2019d8e6-12bd-40f8-b387-06fb66b66a08-kube-api-access-wmsf6\" (UniqueName: \"kubernetes.io/projected/2019d8e6-12bd-40f8-b387-06fb66b66a08-kube-api-access-wmsf6\") pod \"2019d8e6-12bd-40f8-b387-06fb66b66a08\" (UID: \"2019d8e6-12bd-40f8-b387-06fb66b66a08\") " Nov 03 13:58:43.418438 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:58:43.418260 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2019d8e6-12bd-40f8-b387-06fb66b66a08-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2019d8e6-12bd-40f8-b387-06fb66b66a08-workload-certs\") pod \"2019d8e6-12bd-40f8-b387-06fb66b66a08\" (UID: \"2019d8e6-12bd-40f8-b387-06fb66b66a08\") " Nov 03 13:58:43.418438 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:58:43.418330 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2019d8e6-12bd-40f8-b387-06fb66b66a08-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2019d8e6-12bd-40f8-b387-06fb66b66a08-istio-data\") pod \"2019d8e6-12bd-40f8-b387-06fb66b66a08\" (UID: \"2019d8e6-12bd-40f8-b387-06fb66b66a08\") " Nov 03 13:58:43.418693 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:58:43.418440 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2019d8e6-12bd-40f8-b387-06fb66b66a08-istiod-ca-cert" pod "2019d8e6-12bd-40f8-b387-06fb66b66a08" (UID: "2019d8e6-12bd-40f8-b387-06fb66b66a08"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:58:43.418754 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:58:43.418696 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2019d8e6-12bd-40f8-b387-06fb66b66a08-istio-data" pod "2019d8e6-12bd-40f8-b387-06fb66b66a08" (UID: "2019d8e6-12bd-40f8-b387-06fb66b66a08"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:58:43.419141 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:58:43.419115 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2019d8e6-12bd-40f8-b387-06fb66b66a08-workload-socket" pod "2019d8e6-12bd-40f8-b387-06fb66b66a08" (UID: "2019d8e6-12bd-40f8-b387-06fb66b66a08"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:58:43.420882 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:58:43.420854 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2019d8e6-12bd-40f8-b387-06fb66b66a08-credential-socket" pod "2019d8e6-12bd-40f8-b387-06fb66b66a08" (UID: "2019d8e6-12bd-40f8-b387-06fb66b66a08"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:58:43.421004 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:58:43.420907 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2019d8e6-12bd-40f8-b387-06fb66b66a08-workload-certs" pod "2019d8e6-12bd-40f8-b387-06fb66b66a08" (UID: "2019d8e6-12bd-40f8-b387-06fb66b66a08"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:58:43.421783 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:58:43.421145 5024 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2019d8e6-12bd-40f8-b387-06fb66b66a08-istio-data\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:58:43.421783 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:58:43.421167 5024 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2019d8e6-12bd-40f8-b387-06fb66b66a08-workload-socket\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:58:43.421783 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:58:43.421182 5024 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2019d8e6-12bd-40f8-b387-06fb66b66a08-istiod-ca-cert\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:58:43.421783 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:58:43.421196 5024 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2019d8e6-12bd-40f8-b387-06fb66b66a08-credential-socket\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:58:43.421783 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:58:43.421209 5024 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2019d8e6-12bd-40f8-b387-06fb66b66a08-workload-certs\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:58:43.421783 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:58:43.421747 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2019d8e6-12bd-40f8-b387-06fb66b66a08-istio-token" pod "2019d8e6-12bd-40f8-b387-06fb66b66a08" (UID: "2019d8e6-12bd-40f8-b387-06fb66b66a08"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:58:43.422547 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:58:43.422513 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2019d8e6-12bd-40f8-b387-06fb66b66a08-istio-podinfo" pod "2019d8e6-12bd-40f8-b387-06fb66b66a08" (UID: "2019d8e6-12bd-40f8-b387-06fb66b66a08"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 13:58:43.423999 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:58:43.423655 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2019d8e6-12bd-40f8-b387-06fb66b66a08-istio-envoy" pod "2019d8e6-12bd-40f8-b387-06fb66b66a08" (UID: "2019d8e6-12bd-40f8-b387-06fb66b66a08"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:58:43.423999 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:58:43.423954 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2019d8e6-12bd-40f8-b387-06fb66b66a08-kube-api-access-wmsf6" pod "2019d8e6-12bd-40f8-b387-06fb66b66a08" (UID: "2019d8e6-12bd-40f8-b387-06fb66b66a08"). InnerVolumeSpecName "kube-api-access-wmsf6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:58:43.522580 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:58:43.522551 5024 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2019d8e6-12bd-40f8-b387-06fb66b66a08-istio-token\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:58:43.522580 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:58:43.522577 5024 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2019d8e6-12bd-40f8-b387-06fb66b66a08-istio-envoy\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:58:43.522731 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:58:43.522588 5024 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2019d8e6-12bd-40f8-b387-06fb66b66a08-istio-podinfo\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:58:43.522731 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:58:43.522597 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wmsf6\" (UniqueName: \"kubernetes.io/projected/2019d8e6-12bd-40f8-b387-06fb66b66a08-kube-api-access-wmsf6\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:58:44.038151 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:58:44.038111 5024 generic.go:356] "Generic (PLEG): container finished" podID="2019d8e6-12bd-40f8-b387-06fb66b66a08" containerID="1cd9b458925c471109059b17a691630d8cce9d19b4c89d64b2600b7801a98e7a" exitCode=0 Nov 03 13:58:44.038351 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:58:44.038157 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-d0441c-openshift-default-d67bc796c-57tvj" Nov 03 13:58:44.038351 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:58:44.038180 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-d0441c-openshift-default-d67bc796c-57tvj" event={"ID":"2019d8e6-12bd-40f8-b387-06fb66b66a08","Type":"ContainerDied","Data":"1cd9b458925c471109059b17a691630d8cce9d19b4c89d64b2600b7801a98e7a"} Nov 03 13:58:44.038351 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:58:44.038226 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-d0441c-openshift-default-d67bc796c-57tvj" event={"ID":"2019d8e6-12bd-40f8-b387-06fb66b66a08","Type":"ContainerDied","Data":"ae17617a5a0e44332c88304a48e42e6ab12cea2d2e5e5ef74431df0a7aabaeeb"} Nov 03 13:58:44.038351 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:58:44.038249 5024 scope.go:122] "RemoveContainer" containerID="1cd9b458925c471109059b17a691630d8cce9d19b4c89d64b2600b7801a98e7a" Nov 03 13:58:44.047913 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:58:44.047895 5024 scope.go:122] "RemoveContainer" containerID="cdcd6a2408f99154fc7f340499e8c5fc27c66b2d0084f9a39776ccef66be55f6" Nov 03 13:58:44.056208 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:58:44.056187 5024 scope.go:122] "RemoveContainer" containerID="1cd9b458925c471109059b17a691630d8cce9d19b4c89d64b2600b7801a98e7a" Nov 03 13:58:44.056560 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:58:44.056541 5024 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1cd9b458925c471109059b17a691630d8cce9d19b4c89d64b2600b7801a98e7a\": container with ID starting with 1cd9b458925c471109059b17a691630d8cce9d19b4c89d64b2600b7801a98e7a not found: ID does not exist" containerID="1cd9b458925c471109059b17a691630d8cce9d19b4c89d64b2600b7801a98e7a" Nov 03 13:58:44.056616 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:58:44.056572 5024 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1cd9b458925c471109059b17a691630d8cce9d19b4c89d64b2600b7801a98e7a"} err="failed to get container status \"1cd9b458925c471109059b17a691630d8cce9d19b4c89d64b2600b7801a98e7a\": rpc error: code = NotFound desc = could not find container \"1cd9b458925c471109059b17a691630d8cce9d19b4c89d64b2600b7801a98e7a\": container with ID starting with 1cd9b458925c471109059b17a691630d8cce9d19b4c89d64b2600b7801a98e7a not found: ID does not exist" Nov 03 13:58:44.056616 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:58:44.056589 5024 scope.go:122] "RemoveContainer" containerID="cdcd6a2408f99154fc7f340499e8c5fc27c66b2d0084f9a39776ccef66be55f6" Nov 03 13:58:44.056824 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:58:44.056805 5024 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cdcd6a2408f99154fc7f340499e8c5fc27c66b2d0084f9a39776ccef66be55f6\": container with ID starting with cdcd6a2408f99154fc7f340499e8c5fc27c66b2d0084f9a39776ccef66be55f6 not found: ID does not exist" containerID="cdcd6a2408f99154fc7f340499e8c5fc27c66b2d0084f9a39776ccef66be55f6" Nov 03 13:58:44.056869 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:58:44.056830 5024 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cdcd6a2408f99154fc7f340499e8c5fc27c66b2d0084f9a39776ccef66be55f6"} err="failed to get container status \"cdcd6a2408f99154fc7f340499e8c5fc27c66b2d0084f9a39776ccef66be55f6\": rpc error: code = NotFound desc = could not find container \"cdcd6a2408f99154fc7f340499e8c5fc27c66b2d0084f9a39776ccef66be55f6\": container with ID starting with cdcd6a2408f99154fc7f340499e8c5fc27c66b2d0084f9a39776ccef66be55f6 not found: ID does not exist" Nov 03 13:58:44.065613 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:58:44.065589 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-d0441c-openshift-default-d67bc796c-57tvj"] Nov 03 13:58:44.068610 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:58:44.068584 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-d0441c-openshift-default-d67bc796c-57tvj"] Nov 03 13:58:44.107841 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:58:44.107818 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2019d8e6-12bd-40f8-b387-06fb66b66a08" path="/var/lib/kubelet/pods/2019d8e6-12bd-40f8-b387-06fb66b66a08/volumes" Nov 03 13:58:46.123658 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:58:46.123627 5024 scope.go:122] "RemoveContainer" containerID="64505a03a78de5fb06c4527e226b4bcdda292424e4a82b5b9625c9f85212587c" Nov 03 13:58:46.132329 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:58:46.132308 5024 scope.go:122] "RemoveContainer" containerID="6c8ecbb00ccb09b6bed08c6595f5893fa4ca1353392e265f4034117058103c42" Nov 03 13:58:46.140199 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:58:46.140177 5024 scope.go:122] "RemoveContainer" containerID="47b5206e2d1011ae80bcef3e59082a8bad65f13f10145ad5ddcb13726e29f378" Nov 03 13:58:50.808512 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:58:50.808470 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-4d1115-6f8dd857bf-bgrsm"] Nov 03 13:58:50.809867 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:58:50.809497 5024 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4d1115-6f8dd857bf-bgrsm" podUID="c8793d34-41c6-4aaf-a675-19c9f080d516" containerName="maas-api" containerID="cri-o://ee134fdc65203ab4eb9624e663e6cab53d57d2bbba7654ee63d6887367e47a5b" gracePeriod=30 Nov 03 13:58:50.890124 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:58:50.890087 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-4d1115-578bff4f88-fb7gg"] Nov 03 13:58:50.890379 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:58:50.890315 5024 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-4d1115-578bff4f88-fb7gg" podUID="ef470bd5-cb90-49f7-a229-b7a4f64a0c3c" containerName="payload-processing" containerID="cri-o://e362ee506bd2c087dc425c3ba27f504f43541035563b3ca4811542e7066c9f60" gracePeriod=30 Nov 03 13:58:50.900700 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:58:50.900648 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-4d1115-7fc4d545cf-8vqz6"] Nov 03 13:58:50.900909 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:58:50.900870 5024 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4d1115-7fc4d545cf-8vqz6" podUID="933767da-c7c7-4982-bddb-5629094464f1" containerName="payload-pre-processing" containerID="cri-o://12c7897abc86941dad0125e783b0236ae10a8f58a25c79d76ccb1a283e2c7c43" gracePeriod=30 Nov 03 13:58:51.084630 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:58:51.080012 5024 generic.go:356] "Generic (PLEG): container finished" podID="c8793d34-41c6-4aaf-a675-19c9f080d516" containerID="ee134fdc65203ab4eb9624e663e6cab53d57d2bbba7654ee63d6887367e47a5b" exitCode=0 Nov 03 13:58:51.084630 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:58:51.080133 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4d1115-6f8dd857bf-bgrsm" event={"ID":"c8793d34-41c6-4aaf-a675-19c9f080d516","Type":"ContainerDied","Data":"ee134fdc65203ab4eb9624e663e6cab53d57d2bbba7654ee63d6887367e47a5b"} Nov 03 13:58:51.084630 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:58:51.082470 5024 generic.go:356] "Generic (PLEG): container finished" podID="ef470bd5-cb90-49f7-a229-b7a4f64a0c3c" containerID="e362ee506bd2c087dc425c3ba27f504f43541035563b3ca4811542e7066c9f60" exitCode=0 Nov 03 13:58:51.084630 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:58:51.082663 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-4d1115-578bff4f88-fb7gg" event={"ID":"ef470bd5-cb90-49f7-a229-b7a4f64a0c3c","Type":"ContainerDied","Data":"e362ee506bd2c087dc425c3ba27f504f43541035563b3ca4811542e7066c9f60"} Nov 03 13:58:51.135630 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:58:51.134695 5024 generic.go:356] "Generic (PLEG): container finished" podID="933767da-c7c7-4982-bddb-5629094464f1" containerID="12c7897abc86941dad0125e783b0236ae10a8f58a25c79d76ccb1a283e2c7c43" exitCode=0 Nov 03 13:58:51.135630 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:58:51.134754 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4d1115-7fc4d545cf-8vqz6" event={"ID":"933767da-c7c7-4982-bddb-5629094464f1","Type":"ContainerDied","Data":"12c7897abc86941dad0125e783b0236ae10a8f58a25c79d76ccb1a283e2c7c43"} Nov 03 13:58:51.219369 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:58:51.216195 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-4d1115-578bff4f88-fb7gg" Nov 03 13:58:51.268574 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:58:51.268550 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4d1115-6f8dd857bf-bgrsm" Nov 03 13:58:51.275786 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:58:51.275753 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4d1115-7fc4d545cf-8vqz6" Nov 03 13:58:51.338022 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:58:51.337949 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef470bd5-cb90-49f7-a229-b7a4f64a0c3c-kube-api-access-psm76\" (UniqueName: \"kubernetes.io/projected/ef470bd5-cb90-49f7-a229-b7a4f64a0c3c-kube-api-access-psm76\") pod \"ef470bd5-cb90-49f7-a229-b7a4f64a0c3c\" (UID: \"ef470bd5-cb90-49f7-a229-b7a4f64a0c3c\") " Nov 03 13:58:51.338213 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:58:51.338071 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef470bd5-cb90-49f7-a229-b7a4f64a0c3c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef470bd5-cb90-49f7-a229-b7a4f64a0c3c-plugins-config-volume\") pod \"ef470bd5-cb90-49f7-a229-b7a4f64a0c3c\" (UID: \"ef470bd5-cb90-49f7-a229-b7a4f64a0c3c\") " Nov 03 13:58:51.338447 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:58:51.338415 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef470bd5-cb90-49f7-a229-b7a4f64a0c3c-plugins-config-volume" pod "ef470bd5-cb90-49f7-a229-b7a4f64a0c3c" (UID: "ef470bd5-cb90-49f7-a229-b7a4f64a0c3c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:58:51.338584 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:58:51.338539 5024 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef470bd5-cb90-49f7-a229-b7a4f64a0c3c-plugins-config-volume\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:58:51.340188 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:58:51.340165 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef470bd5-cb90-49f7-a229-b7a4f64a0c3c-kube-api-access-psm76" pod "ef470bd5-cb90-49f7-a229-b7a4f64a0c3c" (UID: "ef470bd5-cb90-49f7-a229-b7a4f64a0c3c"). InnerVolumeSpecName "kube-api-access-psm76". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:58:51.439575 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:58:51.439539 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8793d34-41c6-4aaf-a675-19c9f080d516-kube-api-access-gc8b7\" (UniqueName: \"kubernetes.io/projected/c8793d34-41c6-4aaf-a675-19c9f080d516-kube-api-access-gc8b7\") pod \"c8793d34-41c6-4aaf-a675-19c9f080d516\" (UID: \"c8793d34-41c6-4aaf-a675-19c9f080d516\") " Nov 03 13:58:51.439754 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:58:51.439687 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c8793d34-41c6-4aaf-a675-19c9f080d516-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8793d34-41c6-4aaf-a675-19c9f080d516-openshift-service-ca\") pod \"c8793d34-41c6-4aaf-a675-19c9f080d516\" (UID: \"c8793d34-41c6-4aaf-a675-19c9f080d516\") " Nov 03 13:58:51.439754 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:58:51.439720 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c8793d34-41c6-4aaf-a675-19c9f080d516-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c8793d34-41c6-4aaf-a675-19c9f080d516-maas-api-tls\") pod \"c8793d34-41c6-4aaf-a675-19c9f080d516\" (UID: \"c8793d34-41c6-4aaf-a675-19c9f080d516\") " Nov 03 13:58:51.439893 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:58:51.439760 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c8793d34-41c6-4aaf-a675-19c9f080d516-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c8793d34-41c6-4aaf-a675-19c9f080d516-metrics-certs\") pod \"c8793d34-41c6-4aaf-a675-19c9f080d516\" (UID: \"c8793d34-41c6-4aaf-a675-19c9f080d516\") " Nov 03 13:58:51.439893 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:58:51.439798 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/933767da-c7c7-4982-bddb-5629094464f1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/933767da-c7c7-4982-bddb-5629094464f1-plugins-config-volume\") pod \"933767da-c7c7-4982-bddb-5629094464f1\" (UID: \"933767da-c7c7-4982-bddb-5629094464f1\") " Nov 03 13:58:51.440006 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:58:51.439893 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/933767da-c7c7-4982-bddb-5629094464f1-kube-api-access-8nmcx\" (UniqueName: \"kubernetes.io/projected/933767da-c7c7-4982-bddb-5629094464f1-kube-api-access-8nmcx\") pod \"933767da-c7c7-4982-bddb-5629094464f1\" (UID: \"933767da-c7c7-4982-bddb-5629094464f1\") " Nov 03 13:58:51.440182 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:58:51.440157 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8793d34-41c6-4aaf-a675-19c9f080d516-openshift-service-ca" pod "c8793d34-41c6-4aaf-a675-19c9f080d516" (UID: "c8793d34-41c6-4aaf-a675-19c9f080d516"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:58:51.440319 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:58:51.440206 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-psm76\" (UniqueName: \"kubernetes.io/projected/ef470bd5-cb90-49f7-a229-b7a4f64a0c3c-kube-api-access-psm76\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:58:51.440431 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:58:51.440317 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/933767da-c7c7-4982-bddb-5629094464f1-plugins-config-volume" pod "933767da-c7c7-4982-bddb-5629094464f1" (UID: "933767da-c7c7-4982-bddb-5629094464f1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:58:51.441994 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:58:51.441965 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8793d34-41c6-4aaf-a675-19c9f080d516-kube-api-access-gc8b7" pod "c8793d34-41c6-4aaf-a675-19c9f080d516" (UID: "c8793d34-41c6-4aaf-a675-19c9f080d516"). InnerVolumeSpecName "kube-api-access-gc8b7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:58:51.442169 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:58:51.442153 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c8793d34-41c6-4aaf-a675-19c9f080d516-maas-api-tls" pod "c8793d34-41c6-4aaf-a675-19c9f080d516" (UID: "c8793d34-41c6-4aaf-a675-19c9f080d516"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 13:58:51.442264 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:58:51.442246 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c8793d34-41c6-4aaf-a675-19c9f080d516-metrics-certs" pod "c8793d34-41c6-4aaf-a675-19c9f080d516" (UID: "c8793d34-41c6-4aaf-a675-19c9f080d516"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 13:58:51.442502 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:58:51.442486 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/933767da-c7c7-4982-bddb-5629094464f1-kube-api-access-8nmcx" pod "933767da-c7c7-4982-bddb-5629094464f1" (UID: "933767da-c7c7-4982-bddb-5629094464f1"). InnerVolumeSpecName "kube-api-access-8nmcx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:58:51.541847 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:58:51.541814 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8nmcx\" (UniqueName: \"kubernetes.io/projected/933767da-c7c7-4982-bddb-5629094464f1-kube-api-access-8nmcx\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:58:51.541847 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:58:51.541845 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gc8b7\" (UniqueName: \"kubernetes.io/projected/c8793d34-41c6-4aaf-a675-19c9f080d516-kube-api-access-gc8b7\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:58:51.542032 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:58:51.541860 5024 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8793d34-41c6-4aaf-a675-19c9f080d516-openshift-service-ca\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:58:51.542032 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:58:51.541875 5024 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c8793d34-41c6-4aaf-a675-19c9f080d516-maas-api-tls\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:58:51.542032 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:58:51.541889 5024 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c8793d34-41c6-4aaf-a675-19c9f080d516-metrics-certs\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:58:51.542032 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:58:51.541902 5024 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/933767da-c7c7-4982-bddb-5629094464f1-plugins-config-volume\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:58:52.140618 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:58:52.140594 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4d1115-7fc4d545cf-8vqz6" Nov 03 13:58:52.141336 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:58:52.140621 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4d1115-7fc4d545cf-8vqz6" event={"ID":"933767da-c7c7-4982-bddb-5629094464f1","Type":"ContainerDied","Data":"62565515f83fbd3d250b8af5fe743decba719a8d3b6479706455e51fb5e29970"} Nov 03 13:58:52.141336 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:58:52.140690 5024 scope.go:122] "RemoveContainer" containerID="12c7897abc86941dad0125e783b0236ae10a8f58a25c79d76ccb1a283e2c7c43" Nov 03 13:58:52.142095 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:58:52.142079 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4d1115-6f8dd857bf-bgrsm" Nov 03 13:58:52.142200 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:58:52.142104 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4d1115-6f8dd857bf-bgrsm" event={"ID":"c8793d34-41c6-4aaf-a675-19c9f080d516","Type":"ContainerDied","Data":"b80611fdc8ae0b710afd619350cfd937adfeee4f920b3addf09a1157084c07cb"} Nov 03 13:58:52.143651 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:58:52.143624 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-4d1115-578bff4f88-fb7gg" event={"ID":"ef470bd5-cb90-49f7-a229-b7a4f64a0c3c","Type":"ContainerDied","Data":"ff77e564eea2274d210414b58c58a8ad2cf2d3ab30f79dcf5a561aec57e262ed"} Nov 03 13:58:52.143729 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:58:52.143666 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-4d1115-578bff4f88-fb7gg" Nov 03 13:58:52.150665 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:58:52.150647 5024 scope.go:122] "RemoveContainer" containerID="ee134fdc65203ab4eb9624e663e6cab53d57d2bbba7654ee63d6887367e47a5b" Nov 03 13:58:52.158788 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:58:52.158773 5024 scope.go:122] "RemoveContainer" containerID="e362ee506bd2c087dc425c3ba27f504f43541035563b3ca4811542e7066c9f60" Nov 03 13:58:52.173475 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:58:52.173452 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-4d1115-7fc4d545cf-8vqz6"] Nov 03 13:58:52.178096 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:58:52.178076 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-4d1115-7fc4d545cf-8vqz6"] Nov 03 13:58:52.192454 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:58:52.192422 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-4d1115-578bff4f88-fb7gg"] Nov 03 13:58:52.197006 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:58:52.196987 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-4d1115-578bff4f88-fb7gg"] Nov 03 13:58:52.209019 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:58:52.208998 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-4d1115-6f8dd857bf-bgrsm"] Nov 03 13:58:52.215396 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:58:52.215374 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-4d1115-6f8dd857bf-bgrsm"] Nov 03 13:58:54.109449 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:58:54.109414 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="933767da-c7c7-4982-bddb-5629094464f1" path="/var/lib/kubelet/pods/933767da-c7c7-4982-bddb-5629094464f1/volumes" Nov 03 13:58:54.109956 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:58:54.109839 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c8793d34-41c6-4aaf-a675-19c9f080d516" path="/var/lib/kubelet/pods/c8793d34-41c6-4aaf-a675-19c9f080d516/volumes" Nov 03 13:58:54.110221 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:58:54.110204 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef470bd5-cb90-49f7-a229-b7a4f64a0c3c" path="/var/lib/kubelet/pods/ef470bd5-cb90-49f7-a229-b7a4f64a0c3c/volumes" Nov 03 13:58:56.738469 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:58:56.738435 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-4d1115-openshift-default-59d4f94977-rmjd7"] Nov 03 13:58:56.738961 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:58:56.738654 5024 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-4d1115-openshift-default-59d4f94977-rmjd7" podUID="76f44df8-f07c-4c0f-9405-18d2a3b576fb" containerName="istio-proxy" containerID="cri-o://9626726f9fd3389f8b8f00b102053b7d14b0981d55c95f508acea9eca57d7319" gracePeriod=30 Nov 03 13:58:58.215466 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:58:58.215415 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-5b44f68b67-gwbxk"] Nov 03 13:58:58.215864 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:58:58.215824 5024 cpu_manager.go:414] "containerMap: removing container" podUID="c8793d34-41c6-4aaf-a675-19c9f080d516" containerName="maas-api" Nov 03 13:58:58.215864 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:58:58.215835 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8793d34-41c6-4aaf-a675-19c9f080d516" containerName="maas-api" Nov 03 13:58:58.215864 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:58:58.215848 5024 cpu_manager.go:414] "containerMap: removing container" podUID="933767da-c7c7-4982-bddb-5629094464f1" containerName="payload-pre-processing" Nov 03 13:58:58.215864 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:58:58.215853 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="933767da-c7c7-4982-bddb-5629094464f1" containerName="payload-pre-processing" Nov 03 13:58:58.215864 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:58:58.215863 5024 cpu_manager.go:414] "containerMap: removing container" podUID="2019d8e6-12bd-40f8-b387-06fb66b66a08" containerName="istio-proxy" Nov 03 13:58:58.216036 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:58:58.215869 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2019d8e6-12bd-40f8-b387-06fb66b66a08" containerName="istio-proxy" Nov 03 13:58:58.216036 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:58:58.215877 5024 cpu_manager.go:414] "containerMap: removing container" podUID="9899225c-d0e5-4426-b09a-a1009b6b109e" containerName="payload-pre-processing" Nov 03 13:58:58.216036 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:58:58.215882 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9899225c-d0e5-4426-b09a-a1009b6b109e" containerName="payload-pre-processing" Nov 03 13:58:58.216036 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:58:58.215899 5024 cpu_manager.go:414] "containerMap: removing container" podUID="adda3b0a-50a1-4bca-918c-a2f8dbe0f4ee" containerName="maas-api" Nov 03 13:58:58.216036 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:58:58.215904 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="adda3b0a-50a1-4bca-918c-a2f8dbe0f4ee" containerName="maas-api" Nov 03 13:58:58.216036 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:58:58.215911 5024 cpu_manager.go:414] "containerMap: removing container" podUID="2019d8e6-12bd-40f8-b387-06fb66b66a08" containerName="istio-proxy" Nov 03 13:58:58.216036 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:58:58.215916 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2019d8e6-12bd-40f8-b387-06fb66b66a08" containerName="istio-proxy" Nov 03 13:58:58.216036 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:58:58.215924 5024 cpu_manager.go:414] "containerMap: removing container" podUID="b2072f8f-fee3-45b5-acc9-f1be7a68943d" containerName="payload-processing" Nov 03 13:58:58.216036 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:58:58.215928 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2072f8f-fee3-45b5-acc9-f1be7a68943d" containerName="payload-processing" Nov 03 13:58:58.216036 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:58:58.215937 5024 cpu_manager.go:414] "containerMap: removing container" podUID="ef470bd5-cb90-49f7-a229-b7a4f64a0c3c" containerName="payload-processing" Nov 03 13:58:58.216036 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:58:58.215942 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef470bd5-cb90-49f7-a229-b7a4f64a0c3c" containerName="payload-processing" Nov 03 13:58:58.216036 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:58:58.216006 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="adda3b0a-50a1-4bca-918c-a2f8dbe0f4ee" containerName="maas-api" Nov 03 13:58:58.216036 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:58:58.216015 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="2019d8e6-12bd-40f8-b387-06fb66b66a08" containerName="istio-proxy" Nov 03 13:58:58.216036 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:58:58.216021 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2072f8f-fee3-45b5-acc9-f1be7a68943d" containerName="payload-processing" Nov 03 13:58:58.216036 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:58:58.216028 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef470bd5-cb90-49f7-a229-b7a4f64a0c3c" containerName="payload-processing" Nov 03 13:58:58.216036 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:58:58.216036 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8793d34-41c6-4aaf-a675-19c9f080d516" containerName="maas-api" Nov 03 13:58:58.216559 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:58:58.216045 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="933767da-c7c7-4982-bddb-5629094464f1" containerName="payload-pre-processing" Nov 03 13:58:58.216559 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:58:58.216051 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="9899225c-d0e5-4426-b09a-a1009b6b109e" containerName="payload-pre-processing" Nov 03 13:58:58.219167 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:58:58.219140 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-5b44f68b67-gwbxk" Nov 03 13:58:58.230035 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:58:58.230010 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-5b44f68b67-gwbxk"] Nov 03 13:58:58.311815 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:58:58.311782 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8jfwz\" (UniqueName: \"kubernetes.io/projected/b1cb778e-d8e0-4d6c-8656-d7512ba3f1c8-kube-api-access-8jfwz\") pod \"payload-pre-processing-5b44f68b67-gwbxk\" (UID: \"b1cb778e-d8e0-4d6c-8656-d7512ba3f1c8\") " pod="openshift-ingress/payload-pre-processing-5b44f68b67-gwbxk" Nov 03 13:58:58.311979 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:58:58.311862 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1cb778e-d8e0-4d6c-8656-d7512ba3f1c8-plugins-config-volume\") pod \"payload-pre-processing-5b44f68b67-gwbxk\" (UID: \"b1cb778e-d8e0-4d6c-8656-d7512ba3f1c8\") " pod="openshift-ingress/payload-pre-processing-5b44f68b67-gwbxk" Nov 03 13:58:58.413418 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:58:58.413374 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jfwz\" (UniqueName: \"kubernetes.io/projected/b1cb778e-d8e0-4d6c-8656-d7512ba3f1c8-kube-api-access-8jfwz\") pod \"payload-pre-processing-5b44f68b67-gwbxk\" (UID: \"b1cb778e-d8e0-4d6c-8656-d7512ba3f1c8\") " pod="openshift-ingress/payload-pre-processing-5b44f68b67-gwbxk" Nov 03 13:58:58.413627 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:58:58.413471 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1cb778e-d8e0-4d6c-8656-d7512ba3f1c8-plugins-config-volume\") pod \"payload-pre-processing-5b44f68b67-gwbxk\" (UID: \"b1cb778e-d8e0-4d6c-8656-d7512ba3f1c8\") " pod="openshift-ingress/payload-pre-processing-5b44f68b67-gwbxk" Nov 03 13:58:58.414186 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:58:58.414166 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1cb778e-d8e0-4d6c-8656-d7512ba3f1c8-plugins-config-volume\") pod \"payload-pre-processing-5b44f68b67-gwbxk\" (UID: \"b1cb778e-d8e0-4d6c-8656-d7512ba3f1c8\") " pod="openshift-ingress/payload-pre-processing-5b44f68b67-gwbxk" Nov 03 13:58:58.423184 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:58:58.423156 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jfwz\" (UniqueName: \"kubernetes.io/projected/b1cb778e-d8e0-4d6c-8656-d7512ba3f1c8-kube-api-access-8jfwz\") pod \"payload-pre-processing-5b44f68b67-gwbxk\" (UID: \"b1cb778e-d8e0-4d6c-8656-d7512ba3f1c8\") " pod="openshift-ingress/payload-pre-processing-5b44f68b67-gwbxk" Nov 03 13:58:58.532673 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:58:58.532598 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-5b44f68b67-gwbxk" Nov 03 13:58:58.807643 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:58:58.807614 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-5b44f68b67-gwbxk"] Nov 03 13:58:58.809004 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:58:58.808964 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1cb778e_d8e0_4d6c_8656_d7512ba3f1c8.slice/crio-f854304e392fd3c45f4f0791cd44efab632a0622c329fbff7f45660655f19de8 WatchSource:0}: Error finding container f854304e392fd3c45f4f0791cd44efab632a0622c329fbff7f45660655f19de8: Status 404 returned error can't find the container with id f854304e392fd3c45f4f0791cd44efab632a0622c329fbff7f45660655f19de8 Nov 03 13:58:59.182757 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:58:59.182723 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-5b44f68b67-gwbxk" event={"ID":"b1cb778e-d8e0-4d6c-8656-d7512ba3f1c8","Type":"ContainerStarted","Data":"6445f9e2d1a9b4bf61bd2ed11a36d470601ccbce9d06606d8b6cac21e000b4aa"} Nov 03 13:58:59.182929 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:58:59.182773 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-5b44f68b67-gwbxk" event={"ID":"b1cb778e-d8e0-4d6c-8656-d7512ba3f1c8","Type":"ContainerStarted","Data":"f854304e392fd3c45f4f0791cd44efab632a0622c329fbff7f45660655f19de8"} Nov 03 13:58:59.182929 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:58:59.182854 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-5b44f68b67-gwbxk" Nov 03 13:58:59.205829 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:58:59.205786 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-5b44f68b67-gwbxk" podStartSLOduration=1.205772957 podStartE2EDuration="1.205772957s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:58:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:58:59.20351672 +0000 UTC m=+3019.211708002" watchObservedRunningTime="2026-11-03 13:58:59.205772957 +0000 UTC m=+3019.213964227" Nov 03 13:58:59.699358 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:58:59.699320 5024 patch_prober.go:29] interesting pod/e2e-ipp-b-4d1115-openshift-default-59d4f94977-rmjd7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 13:58:59.699739 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:58:59.699379 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-4d1115-openshift-default-59d4f94977-rmjd7" podUID="76f44df8-f07c-4c0f-9405-18d2a3b576fb" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 13:59:01.989494 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:59:01.989469 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-4d1115-openshift-default-59d4f94977-rmjd7" Nov 03 13:59:02.050960 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:59:02.050896 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76f44df8-f07c-4c0f-9405-18d2a3b576fb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76f44df8-f07c-4c0f-9405-18d2a3b576fb-credential-socket\") pod \"76f44df8-f07c-4c0f-9405-18d2a3b576fb\" (UID: \"76f44df8-f07c-4c0f-9405-18d2a3b576fb\") " Nov 03 13:59:02.050960 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:59:02.050944 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76f44df8-f07c-4c0f-9405-18d2a3b576fb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76f44df8-f07c-4c0f-9405-18d2a3b576fb-workload-socket\") pod \"76f44df8-f07c-4c0f-9405-18d2a3b576fb\" (UID: \"76f44df8-f07c-4c0f-9405-18d2a3b576fb\") " Nov 03 13:59:02.051193 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:59:02.050980 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76f44df8-f07c-4c0f-9405-18d2a3b576fb-istio-token\" (UniqueName: \"kubernetes.io/projected/76f44df8-f07c-4c0f-9405-18d2a3b576fb-istio-token\") pod \"76f44df8-f07c-4c0f-9405-18d2a3b576fb\" (UID: \"76f44df8-f07c-4c0f-9405-18d2a3b576fb\") " Nov 03 13:59:02.051193 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:59:02.051036 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76f44df8-f07c-4c0f-9405-18d2a3b576fb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76f44df8-f07c-4c0f-9405-18d2a3b576fb-istio-envoy\") pod \"76f44df8-f07c-4c0f-9405-18d2a3b576fb\" (UID: \"76f44df8-f07c-4c0f-9405-18d2a3b576fb\") " Nov 03 13:59:02.051193 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:59:02.051056 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/76f44df8-f07c-4c0f-9405-18d2a3b576fb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76f44df8-f07c-4c0f-9405-18d2a3b576fb-istio-podinfo\") pod \"76f44df8-f07c-4c0f-9405-18d2a3b576fb\" (UID: \"76f44df8-f07c-4c0f-9405-18d2a3b576fb\") " Nov 03 13:59:02.051193 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:59:02.051074 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76f44df8-f07c-4c0f-9405-18d2a3b576fb-kube-api-access-4mpzf\" (UniqueName: \"kubernetes.io/projected/76f44df8-f07c-4c0f-9405-18d2a3b576fb-kube-api-access-4mpzf\") pod \"76f44df8-f07c-4c0f-9405-18d2a3b576fb\" (UID: \"76f44df8-f07c-4c0f-9405-18d2a3b576fb\") " Nov 03 13:59:02.051193 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:59:02.051121 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/76f44df8-f07c-4c0f-9405-18d2a3b576fb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76f44df8-f07c-4c0f-9405-18d2a3b576fb-istiod-ca-cert\") pod \"76f44df8-f07c-4c0f-9405-18d2a3b576fb\" (UID: \"76f44df8-f07c-4c0f-9405-18d2a3b576fb\") " Nov 03 13:59:02.051193 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:59:02.051150 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76f44df8-f07c-4c0f-9405-18d2a3b576fb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76f44df8-f07c-4c0f-9405-18d2a3b576fb-workload-certs\") pod \"76f44df8-f07c-4c0f-9405-18d2a3b576fb\" (UID: \"76f44df8-f07c-4c0f-9405-18d2a3b576fb\") " Nov 03 13:59:02.051581 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:59:02.051215 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76f44df8-f07c-4c0f-9405-18d2a3b576fb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76f44df8-f07c-4c0f-9405-18d2a3b576fb-istio-data\") pod \"76f44df8-f07c-4c0f-9405-18d2a3b576fb\" (UID: \"76f44df8-f07c-4c0f-9405-18d2a3b576fb\") " Nov 03 13:59:02.051581 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:59:02.051213 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76f44df8-f07c-4c0f-9405-18d2a3b576fb-credential-socket" pod "76f44df8-f07c-4c0f-9405-18d2a3b576fb" (UID: "76f44df8-f07c-4c0f-9405-18d2a3b576fb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:59:02.051581 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:59:02.051361 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76f44df8-f07c-4c0f-9405-18d2a3b576fb-workload-socket" pod "76f44df8-f07c-4c0f-9405-18d2a3b576fb" (UID: "76f44df8-f07c-4c0f-9405-18d2a3b576fb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:59:02.051581 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:59:02.051556 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76f44df8-f07c-4c0f-9405-18d2a3b576fb-istio-data" pod "76f44df8-f07c-4c0f-9405-18d2a3b576fb" (UID: "76f44df8-f07c-4c0f-9405-18d2a3b576fb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:59:02.051813 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:59:02.051644 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76f44df8-f07c-4c0f-9405-18d2a3b576fb-workload-certs" pod "76f44df8-f07c-4c0f-9405-18d2a3b576fb" (UID: "76f44df8-f07c-4c0f-9405-18d2a3b576fb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:59:02.051813 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:59:02.051671 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76f44df8-f07c-4c0f-9405-18d2a3b576fb-istiod-ca-cert" pod "76f44df8-f07c-4c0f-9405-18d2a3b576fb" (UID: "76f44df8-f07c-4c0f-9405-18d2a3b576fb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:59:02.051813 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:59:02.051749 5024 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76f44df8-f07c-4c0f-9405-18d2a3b576fb-istio-data\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:59:02.051813 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:59:02.051784 5024 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76f44df8-f07c-4c0f-9405-18d2a3b576fb-credential-socket\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:59:02.051813 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:59:02.051799 5024 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76f44df8-f07c-4c0f-9405-18d2a3b576fb-workload-socket\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:59:02.053715 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:59:02.053685 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76f44df8-f07c-4c0f-9405-18d2a3b576fb-istio-envoy" pod "76f44df8-f07c-4c0f-9405-18d2a3b576fb" (UID: "76f44df8-f07c-4c0f-9405-18d2a3b576fb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:59:02.053834 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:59:02.053810 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76f44df8-f07c-4c0f-9405-18d2a3b576fb-istio-token" pod "76f44df8-f07c-4c0f-9405-18d2a3b576fb" (UID: "76f44df8-f07c-4c0f-9405-18d2a3b576fb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:59:02.054003 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:59:02.053977 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/76f44df8-f07c-4c0f-9405-18d2a3b576fb-istio-podinfo" pod "76f44df8-f07c-4c0f-9405-18d2a3b576fb" (UID: "76f44df8-f07c-4c0f-9405-18d2a3b576fb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 13:59:02.054003 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:59:02.053991 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76f44df8-f07c-4c0f-9405-18d2a3b576fb-kube-api-access-4mpzf" pod "76f44df8-f07c-4c0f-9405-18d2a3b576fb" (UID: "76f44df8-f07c-4c0f-9405-18d2a3b576fb"). InnerVolumeSpecName "kube-api-access-4mpzf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:59:02.153102 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:59:02.153072 5024 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76f44df8-f07c-4c0f-9405-18d2a3b576fb-istio-envoy\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:59:02.153102 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:59:02.153099 5024 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76f44df8-f07c-4c0f-9405-18d2a3b576fb-istio-podinfo\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:59:02.153301 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:59:02.153109 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4mpzf\" (UniqueName: \"kubernetes.io/projected/76f44df8-f07c-4c0f-9405-18d2a3b576fb-kube-api-access-4mpzf\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:59:02.153301 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:59:02.153119 5024 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76f44df8-f07c-4c0f-9405-18d2a3b576fb-istiod-ca-cert\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:59:02.153301 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:59:02.153129 5024 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76f44df8-f07c-4c0f-9405-18d2a3b576fb-workload-certs\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:59:02.153301 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:59:02.153137 5024 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76f44df8-f07c-4c0f-9405-18d2a3b576fb-istio-token\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:59:02.199755 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:59:02.199723 5024 generic.go:356] "Generic (PLEG): container finished" podID="76f44df8-f07c-4c0f-9405-18d2a3b576fb" containerID="9626726f9fd3389f8b8f00b102053b7d14b0981d55c95f508acea9eca57d7319" exitCode=0 Nov 03 13:59:02.199899 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:59:02.199769 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-4d1115-openshift-default-59d4f94977-rmjd7" Nov 03 13:59:02.199899 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:59:02.199810 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-4d1115-openshift-default-59d4f94977-rmjd7" event={"ID":"76f44df8-f07c-4c0f-9405-18d2a3b576fb","Type":"ContainerDied","Data":"9626726f9fd3389f8b8f00b102053b7d14b0981d55c95f508acea9eca57d7319"} Nov 03 13:59:02.199899 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:59:02.199869 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-4d1115-openshift-default-59d4f94977-rmjd7" event={"ID":"76f44df8-f07c-4c0f-9405-18d2a3b576fb","Type":"ContainerDied","Data":"a74231a166dca018e9b568ebcaea4d7595ef5239757bb7333ed81319c6d657cb"} Nov 03 13:59:02.200022 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:59:02.199920 5024 scope.go:122] "RemoveContainer" containerID="9626726f9fd3389f8b8f00b102053b7d14b0981d55c95f508acea9eca57d7319" Nov 03 13:59:02.209224 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:59:02.209205 5024 scope.go:122] "RemoveContainer" containerID="4e2f78a63cfeb387fde0d709b923a04821ac5151219d772b075f0ca1483c281d" Nov 03 13:59:02.217181 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:59:02.217165 5024 scope.go:122] "RemoveContainer" containerID="9626726f9fd3389f8b8f00b102053b7d14b0981d55c95f508acea9eca57d7319" Nov 03 13:59:02.217498 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:59:02.217475 5024 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9626726f9fd3389f8b8f00b102053b7d14b0981d55c95f508acea9eca57d7319\": container with ID starting with 9626726f9fd3389f8b8f00b102053b7d14b0981d55c95f508acea9eca57d7319 not found: ID does not exist" containerID="9626726f9fd3389f8b8f00b102053b7d14b0981d55c95f508acea9eca57d7319" Nov 03 13:59:02.217554 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:59:02.217507 5024 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9626726f9fd3389f8b8f00b102053b7d14b0981d55c95f508acea9eca57d7319"} err="failed to get container status \"9626726f9fd3389f8b8f00b102053b7d14b0981d55c95f508acea9eca57d7319\": rpc error: code = NotFound desc = could not find container \"9626726f9fd3389f8b8f00b102053b7d14b0981d55c95f508acea9eca57d7319\": container with ID starting with 9626726f9fd3389f8b8f00b102053b7d14b0981d55c95f508acea9eca57d7319 not found: ID does not exist" Nov 03 13:59:02.217554 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:59:02.217524 5024 scope.go:122] "RemoveContainer" containerID="4e2f78a63cfeb387fde0d709b923a04821ac5151219d772b075f0ca1483c281d" Nov 03 13:59:02.217781 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:59:02.217765 5024 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e2f78a63cfeb387fde0d709b923a04821ac5151219d772b075f0ca1483c281d\": container with ID starting with 4e2f78a63cfeb387fde0d709b923a04821ac5151219d772b075f0ca1483c281d not found: ID does not exist" containerID="4e2f78a63cfeb387fde0d709b923a04821ac5151219d772b075f0ca1483c281d" Nov 03 13:59:02.217827 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:59:02.217790 5024 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e2f78a63cfeb387fde0d709b923a04821ac5151219d772b075f0ca1483c281d"} err="failed to get container status \"4e2f78a63cfeb387fde0d709b923a04821ac5151219d772b075f0ca1483c281d\": rpc error: code = NotFound desc = could not find container \"4e2f78a63cfeb387fde0d709b923a04821ac5151219d772b075f0ca1483c281d\": container with ID starting with 4e2f78a63cfeb387fde0d709b923a04821ac5151219d772b075f0ca1483c281d not found: ID does not exist" Nov 03 13:59:02.221139 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:59:02.221117 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-4d1115-openshift-default-59d4f94977-rmjd7"] Nov 03 13:59:02.225024 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:59:02.225003 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-4d1115-openshift-default-59d4f94977-rmjd7"] Nov 03 13:59:04.108094 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:59:04.108059 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="76f44df8-f07c-4c0f-9405-18d2a3b576fb" path="/var/lib/kubelet/pods/76f44df8-f07c-4c0f-9405-18d2a3b576fb/volumes" Nov 03 13:59:09.506130 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:59:09.506088 5024 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d7mwz" containerName="extractor" Nov 03 13:59:10.190626 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:59:10.190589 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-5b44f68b67-gwbxk" Nov 03 13:59:10.234374 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:59:10.234342 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-p9ckq"] Nov 03 13:59:12.004231 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:59:12.004189 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-74d765fb54-fcczn"] Nov 03 13:59:12.004667 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:59:12.004655 5024 cpu_manager.go:414] "containerMap: removing container" podUID="76f44df8-f07c-4c0f-9405-18d2a3b576fb" containerName="istio-proxy" Nov 03 13:59:12.004712 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:59:12.004669 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76f44df8-f07c-4c0f-9405-18d2a3b576fb" containerName="istio-proxy" Nov 03 13:59:12.004779 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:59:12.004767 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="76f44df8-f07c-4c0f-9405-18d2a3b576fb" containerName="istio-proxy" Nov 03 13:59:12.004816 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:59:12.004781 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="2019d8e6-12bd-40f8-b387-06fb66b66a08" containerName="istio-proxy" Nov 03 13:59:12.009658 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:59:12.009635 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-74d765fb54-fcczn" Nov 03 13:59:12.015981 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:59:12.015949 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-74d765fb54-fcczn"] Nov 03 13:59:12.051800 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:59:12.051768 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c72cff36-b802-4c5a-b6ca-258eb6353616-plugins-config-volume\") pod \"payload-processing-74d765fb54-fcczn\" (UID: \"c72cff36-b802-4c5a-b6ca-258eb6353616\") " pod="openshift-ingress/payload-processing-74d765fb54-fcczn" Nov 03 13:59:12.051988 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:59:12.051820 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q77w9\" (UniqueName: \"kubernetes.io/projected/c72cff36-b802-4c5a-b6ca-258eb6353616-kube-api-access-q77w9\") pod \"payload-processing-74d765fb54-fcczn\" (UID: \"c72cff36-b802-4c5a-b6ca-258eb6353616\") " pod="openshift-ingress/payload-processing-74d765fb54-fcczn" Nov 03 13:59:12.153160 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:59:12.153130 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c72cff36-b802-4c5a-b6ca-258eb6353616-plugins-config-volume\") pod \"payload-processing-74d765fb54-fcczn\" (UID: \"c72cff36-b802-4c5a-b6ca-258eb6353616\") " pod="openshift-ingress/payload-processing-74d765fb54-fcczn" Nov 03 13:59:12.153343 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:59:12.153171 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q77w9\" (UniqueName: \"kubernetes.io/projected/c72cff36-b802-4c5a-b6ca-258eb6353616-kube-api-access-q77w9\") pod \"payload-processing-74d765fb54-fcczn\" (UID: \"c72cff36-b802-4c5a-b6ca-258eb6353616\") " pod="openshift-ingress/payload-processing-74d765fb54-fcczn" Nov 03 13:59:12.153862 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:59:12.153838 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c72cff36-b802-4c5a-b6ca-258eb6353616-plugins-config-volume\") pod \"payload-processing-74d765fb54-fcczn\" (UID: \"c72cff36-b802-4c5a-b6ca-258eb6353616\") " pod="openshift-ingress/payload-processing-74d765fb54-fcczn" Nov 03 13:59:12.162841 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:59:12.162812 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q77w9\" (UniqueName: \"kubernetes.io/projected/c72cff36-b802-4c5a-b6ca-258eb6353616-kube-api-access-q77w9\") pod \"payload-processing-74d765fb54-fcczn\" (UID: \"c72cff36-b802-4c5a-b6ca-258eb6353616\") " pod="openshift-ingress/payload-processing-74d765fb54-fcczn" Nov 03 13:59:12.327365 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:59:12.327259 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-74d765fb54-fcczn" Nov 03 13:59:12.611565 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:59:12.611540 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-74d765fb54-fcczn"] Nov 03 13:59:12.613132 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:59:12.613095 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc72cff36_b802_4c5a_b6ca_258eb6353616.slice/crio-b06893cde9f4c7c22662f6ee85a67fa3f252df7bd7a35546a97c7f31cbf0ca7d WatchSource:0}: Error finding container b06893cde9f4c7c22662f6ee85a67fa3f252df7bd7a35546a97c7f31cbf0ca7d: Status 404 returned error can't find the container with id b06893cde9f4c7c22662f6ee85a67fa3f252df7bd7a35546a97c7f31cbf0ca7d Nov 03 13:59:13.256778 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:59:13.256733 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-74d765fb54-fcczn" event={"ID":"c72cff36-b802-4c5a-b6ca-258eb6353616","Type":"ContainerStarted","Data":"88911aa9969d89ea2bf71412e26c4a6492b9a91014f8da99cad06f33d950f91c"} Nov 03 13:59:13.257269 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:59:13.256836 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-74d765fb54-fcczn" Nov 03 13:59:13.257269 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:59:13.256866 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-74d765fb54-fcczn" event={"ID":"c72cff36-b802-4c5a-b6ca-258eb6353616","Type":"ContainerStarted","Data":"b06893cde9f4c7c22662f6ee85a67fa3f252df7bd7a35546a97c7f31cbf0ca7d"} Nov 03 13:59:13.277946 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:59:13.277893 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-74d765fb54-fcczn" podStartSLOduration=2.277880416 podStartE2EDuration="2.277880416s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:59:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:59:13.276840688 +0000 UTC m=+3033.285031965" watchObservedRunningTime="2026-11-03 13:59:13.277880416 +0000 UTC m=+3033.286071692" Nov 03 13:59:15.235508 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:59:15.235468 5024 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-7b98bb678c-p9ckq" podUID="793b4955-68a1-4bf4-bcc7-7f0556c6783c" containerName="payload-pre-processing" containerID="cri-o://4a7e4a6fe1ecb74061d898007b6b280233b8fc1b2886d36e99e6a90ed4ee6a8a" gracePeriod=25 Nov 03 13:59:15.498655 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:59:15.498592 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-p9ckq" Nov 03 13:59:15.588892 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:59:15.588860 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/793b4955-68a1-4bf4-bcc7-7f0556c6783c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/793b4955-68a1-4bf4-bcc7-7f0556c6783c-plugins-config-volume\") pod \"793b4955-68a1-4bf4-bcc7-7f0556c6783c\" (UID: \"793b4955-68a1-4bf4-bcc7-7f0556c6783c\") " Nov 03 13:59:15.589056 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:59:15.588927 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/793b4955-68a1-4bf4-bcc7-7f0556c6783c-kube-api-access-gn5vr\" (UniqueName: \"kubernetes.io/projected/793b4955-68a1-4bf4-bcc7-7f0556c6783c-kube-api-access-gn5vr\") pod \"793b4955-68a1-4bf4-bcc7-7f0556c6783c\" (UID: \"793b4955-68a1-4bf4-bcc7-7f0556c6783c\") " Nov 03 13:59:15.589346 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:59:15.589324 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/793b4955-68a1-4bf4-bcc7-7f0556c6783c-plugins-config-volume" pod "793b4955-68a1-4bf4-bcc7-7f0556c6783c" (UID: "793b4955-68a1-4bf4-bcc7-7f0556c6783c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:59:15.591018 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:59:15.590998 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/793b4955-68a1-4bf4-bcc7-7f0556c6783c-kube-api-access-gn5vr" pod "793b4955-68a1-4bf4-bcc7-7f0556c6783c" (UID: "793b4955-68a1-4bf4-bcc7-7f0556c6783c"). InnerVolumeSpecName "kube-api-access-gn5vr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:59:15.690672 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:59:15.690642 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gn5vr\" (UniqueName: \"kubernetes.io/projected/793b4955-68a1-4bf4-bcc7-7f0556c6783c-kube-api-access-gn5vr\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:59:15.690672 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:59:15.690669 5024 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/793b4955-68a1-4bf4-bcc7-7f0556c6783c-plugins-config-volume\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:59:16.272566 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:59:16.272527 5024 generic.go:356] "Generic (PLEG): container finished" podID="793b4955-68a1-4bf4-bcc7-7f0556c6783c" containerID="4a7e4a6fe1ecb74061d898007b6b280233b8fc1b2886d36e99e6a90ed4ee6a8a" exitCode=0 Nov 03 13:59:16.273061 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:59:16.272577 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-p9ckq" Nov 03 13:59:16.273061 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:59:16.272610 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-p9ckq" event={"ID":"793b4955-68a1-4bf4-bcc7-7f0556c6783c","Type":"ContainerDied","Data":"4a7e4a6fe1ecb74061d898007b6b280233b8fc1b2886d36e99e6a90ed4ee6a8a"} Nov 03 13:59:16.273061 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:59:16.272663 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-p9ckq" event={"ID":"793b4955-68a1-4bf4-bcc7-7f0556c6783c","Type":"ContainerDied","Data":"3f592c5b43e686dab328ab678ea8d3edba2ce2d37f323fffb25f9e7f0a80215a"} Nov 03 13:59:16.273061 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:59:16.272687 5024 scope.go:122] "RemoveContainer" containerID="4a7e4a6fe1ecb74061d898007b6b280233b8fc1b2886d36e99e6a90ed4ee6a8a" Nov 03 13:59:16.281250 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:59:16.281233 5024 scope.go:122] "RemoveContainer" containerID="4a7e4a6fe1ecb74061d898007b6b280233b8fc1b2886d36e99e6a90ed4ee6a8a" Nov 03 13:59:16.281519 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:59:16.281500 5024 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a7e4a6fe1ecb74061d898007b6b280233b8fc1b2886d36e99e6a90ed4ee6a8a\": container with ID starting with 4a7e4a6fe1ecb74061d898007b6b280233b8fc1b2886d36e99e6a90ed4ee6a8a not found: ID does not exist" containerID="4a7e4a6fe1ecb74061d898007b6b280233b8fc1b2886d36e99e6a90ed4ee6a8a" Nov 03 13:59:16.281603 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:59:16.281527 5024 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a7e4a6fe1ecb74061d898007b6b280233b8fc1b2886d36e99e6a90ed4ee6a8a"} err="failed to get container status \"4a7e4a6fe1ecb74061d898007b6b280233b8fc1b2886d36e99e6a90ed4ee6a8a\": rpc error: code = NotFound desc = could not find container \"4a7e4a6fe1ecb74061d898007b6b280233b8fc1b2886d36e99e6a90ed4ee6a8a\": container with ID starting with 4a7e4a6fe1ecb74061d898007b6b280233b8fc1b2886d36e99e6a90ed4ee6a8a not found: ID does not exist" Nov 03 13:59:16.293251 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:59:16.293227 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-p9ckq"] Nov 03 13:59:16.297308 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:59:16.297271 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-p9ckq"] Nov 03 13:59:18.107296 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:59:18.107260 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="793b4955-68a1-4bf4-bcc7-7f0556c6783c" path="/var/lib/kubelet/pods/793b4955-68a1-4bf4-bcc7-7f0556c6783c/volumes" Nov 03 13:59:24.265136 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:59:24.265108 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-74d765fb54-fcczn" Nov 03 13:59:24.311001 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:59:24.310967 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-khw76"] Nov 03 13:59:26.244300 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:59:26.244244 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-776d6d99b-q7jqc"] Nov 03 13:59:26.244703 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:59:26.244655 5024 cpu_manager.go:414] "containerMap: removing container" podUID="793b4955-68a1-4bf4-bcc7-7f0556c6783c" containerName="payload-pre-processing" Nov 03 13:59:26.244703 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:59:26.244666 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="793b4955-68a1-4bf4-bcc7-7f0556c6783c" containerName="payload-pre-processing" Nov 03 13:59:26.244703 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:59:26.244693 5024 cpu_manager.go:414] "containerMap: removing container" podUID="76f44df8-f07c-4c0f-9405-18d2a3b576fb" containerName="istio-proxy" Nov 03 13:59:26.244703 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:59:26.244699 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76f44df8-f07c-4c0f-9405-18d2a3b576fb" containerName="istio-proxy" Nov 03 13:59:26.244858 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:59:26.244771 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="76f44df8-f07c-4c0f-9405-18d2a3b576fb" containerName="istio-proxy" Nov 03 13:59:26.244858 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:59:26.244785 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="793b4955-68a1-4bf4-bcc7-7f0556c6783c" containerName="payload-pre-processing" Nov 03 13:59:26.248497 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:59:26.248478 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-776d6d99b-q7jqc" Nov 03 13:59:26.256104 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:59:26.255835 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-776d6d99b-q7jqc"] Nov 03 13:59:26.395591 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:59:26.395557 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4mcmf\" (UniqueName: \"kubernetes.io/projected/b0918520-a9f4-4ebf-8c93-301ac3d62e8f-kube-api-access-4mcmf\") pod \"payload-processing-776d6d99b-q7jqc\" (UID: \"b0918520-a9f4-4ebf-8c93-301ac3d62e8f\") " pod="openshift-ingress/payload-processing-776d6d99b-q7jqc" Nov 03 13:59:26.395745 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:59:26.395635 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b0918520-a9f4-4ebf-8c93-301ac3d62e8f-plugins-config-volume\") pod \"payload-processing-776d6d99b-q7jqc\" (UID: \"b0918520-a9f4-4ebf-8c93-301ac3d62e8f\") " pod="openshift-ingress/payload-processing-776d6d99b-q7jqc" Nov 03 13:59:26.441603 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:59:26.441568 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-67f556869f-kv4cs"] Nov 03 13:59:26.445072 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:59:26.445054 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-67f556869f-kv4cs" Nov 03 13:59:26.453967 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:59:26.453941 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-67f556869f-kv4cs"] Nov 03 13:59:26.496981 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:59:26.496913 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4mcmf\" (UniqueName: \"kubernetes.io/projected/b0918520-a9f4-4ebf-8c93-301ac3d62e8f-kube-api-access-4mcmf\") pod \"payload-processing-776d6d99b-q7jqc\" (UID: \"b0918520-a9f4-4ebf-8c93-301ac3d62e8f\") " pod="openshift-ingress/payload-processing-776d6d99b-q7jqc" Nov 03 13:59:26.497116 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:59:26.496990 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b0918520-a9f4-4ebf-8c93-301ac3d62e8f-plugins-config-volume\") pod \"payload-processing-776d6d99b-q7jqc\" (UID: \"b0918520-a9f4-4ebf-8c93-301ac3d62e8f\") " pod="openshift-ingress/payload-processing-776d6d99b-q7jqc" Nov 03 13:59:26.497678 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:59:26.497659 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b0918520-a9f4-4ebf-8c93-301ac3d62e8f-plugins-config-volume\") pod \"payload-processing-776d6d99b-q7jqc\" (UID: \"b0918520-a9f4-4ebf-8c93-301ac3d62e8f\") " pod="openshift-ingress/payload-processing-776d6d99b-q7jqc" Nov 03 13:59:26.506263 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:59:26.506241 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4mcmf\" (UniqueName: \"kubernetes.io/projected/b0918520-a9f4-4ebf-8c93-301ac3d62e8f-kube-api-access-4mcmf\") pod \"payload-processing-776d6d99b-q7jqc\" (UID: \"b0918520-a9f4-4ebf-8c93-301ac3d62e8f\") " pod="openshift-ingress/payload-processing-776d6d99b-q7jqc" Nov 03 13:59:26.563303 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:59:26.563247 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-776d6d99b-q7jqc" Nov 03 13:59:26.597850 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:59:26.597797 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bad6aa6e-a56a-4955-9237-6f254c967c37-plugins-config-volume\") pod \"payload-pre-processing-67f556869f-kv4cs\" (UID: \"bad6aa6e-a56a-4955-9237-6f254c967c37\") " pod="openshift-ingress/payload-pre-processing-67f556869f-kv4cs" Nov 03 13:59:26.598064 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:59:26.597948 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-476b4\" (UniqueName: \"kubernetes.io/projected/bad6aa6e-a56a-4955-9237-6f254c967c37-kube-api-access-476b4\") pod \"payload-pre-processing-67f556869f-kv4cs\" (UID: \"bad6aa6e-a56a-4955-9237-6f254c967c37\") " pod="openshift-ingress/payload-pre-processing-67f556869f-kv4cs" Nov 03 13:59:26.698743 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:59:26.698701 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bad6aa6e-a56a-4955-9237-6f254c967c37-plugins-config-volume\") pod \"payload-pre-processing-67f556869f-kv4cs\" (UID: \"bad6aa6e-a56a-4955-9237-6f254c967c37\") " pod="openshift-ingress/payload-pre-processing-67f556869f-kv4cs" Nov 03 13:59:26.698903 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:59:26.698803 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-476b4\" (UniqueName: \"kubernetes.io/projected/bad6aa6e-a56a-4955-9237-6f254c967c37-kube-api-access-476b4\") pod \"payload-pre-processing-67f556869f-kv4cs\" (UID: \"bad6aa6e-a56a-4955-9237-6f254c967c37\") " pod="openshift-ingress/payload-pre-processing-67f556869f-kv4cs" Nov 03 13:59:26.699464 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:59:26.699440 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bad6aa6e-a56a-4955-9237-6f254c967c37-plugins-config-volume\") pod \"payload-pre-processing-67f556869f-kv4cs\" (UID: \"bad6aa6e-a56a-4955-9237-6f254c967c37\") " pod="openshift-ingress/payload-pre-processing-67f556869f-kv4cs" Nov 03 13:59:26.708745 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:59:26.708716 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-476b4\" (UniqueName: \"kubernetes.io/projected/bad6aa6e-a56a-4955-9237-6f254c967c37-kube-api-access-476b4\") pod \"payload-pre-processing-67f556869f-kv4cs\" (UID: \"bad6aa6e-a56a-4955-9237-6f254c967c37\") " pod="openshift-ingress/payload-pre-processing-67f556869f-kv4cs" Nov 03 13:59:26.759594 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:59:26.759504 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-67f556869f-kv4cs" Nov 03 13:59:26.844643 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:59:26.844608 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-776d6d99b-q7jqc"] Nov 03 13:59:26.848891 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:59:26.848863 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0918520_a9f4_4ebf_8c93_301ac3d62e8f.slice/crio-e0aad78aed5c350ffb8477ef69378121bbb028af02dc9d04872e2940118465ed WatchSource:0}: Error finding container e0aad78aed5c350ffb8477ef69378121bbb028af02dc9d04872e2940118465ed: Status 404 returned error can't find the container with id e0aad78aed5c350ffb8477ef69378121bbb028af02dc9d04872e2940118465ed Nov 03 13:59:27.032025 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:59:27.032000 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-67f556869f-kv4cs"] Nov 03 13:59:27.034319 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:59:27.034275 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbad6aa6e_a56a_4955_9237_6f254c967c37.slice/crio-b99c1b510a847c5e2c584b5d15066e20778d58b179d65c214d796838e572cd20 WatchSource:0}: Error finding container b99c1b510a847c5e2c584b5d15066e20778d58b179d65c214d796838e572cd20: Status 404 returned error can't find the container with id b99c1b510a847c5e2c584b5d15066e20778d58b179d65c214d796838e572cd20 Nov 03 13:59:27.329079 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:59:27.328988 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-776d6d99b-q7jqc" event={"ID":"b0918520-a9f4-4ebf-8c93-301ac3d62e8f","Type":"ContainerStarted","Data":"306247f81d0a2c3bd922ade11ecb44df7ddc11be35407404c18635a7331a2ca6"} Nov 03 13:59:27.329079 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:59:27.329047 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-776d6d99b-q7jqc" event={"ID":"b0918520-a9f4-4ebf-8c93-301ac3d62e8f","Type":"ContainerStarted","Data":"e0aad78aed5c350ffb8477ef69378121bbb028af02dc9d04872e2940118465ed"} Nov 03 13:59:27.329613 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:59:27.329095 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-776d6d99b-q7jqc" Nov 03 13:59:27.330789 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:59:27.330763 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-67f556869f-kv4cs" event={"ID":"bad6aa6e-a56a-4955-9237-6f254c967c37","Type":"ContainerStarted","Data":"2a1df76d58a777a668f450359383a96db74e34830d61513f1dc679dded335019"} Nov 03 13:59:27.330910 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:59:27.330808 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-67f556869f-kv4cs" Nov 03 13:59:27.330910 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:59:27.330825 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-67f556869f-kv4cs" event={"ID":"bad6aa6e-a56a-4955-9237-6f254c967c37","Type":"ContainerStarted","Data":"b99c1b510a847c5e2c584b5d15066e20778d58b179d65c214d796838e572cd20"} Nov 03 13:59:27.347857 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:59:27.347816 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-776d6d99b-q7jqc" podStartSLOduration=1.347803334 podStartE2EDuration="1.347803334s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:59:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:59:27.347566925 +0000 UTC m=+3047.355758203" watchObservedRunningTime="2026-11-03 13:59:27.347803334 +0000 UTC m=+3047.355994612" Nov 03 13:59:27.366422 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:59:27.366382 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-67f556869f-kv4cs" podStartSLOduration=1.366369836 podStartE2EDuration="1.366369836s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:59:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:59:27.365268441 +0000 UTC m=+3047.373459718" watchObservedRunningTime="2026-11-03 13:59:27.366369836 +0000 UTC m=+3047.374561137" Nov 03 13:59:29.312044 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:59:29.311961 5024 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-55fbbbf9dc-khw76" podUID="36118654-3637-489e-b048-af4aa366f34d" containerName="payload-processing" containerID="cri-o://79e8003191720eef732cc991efc1dc83c996233a833c5121a47c0a85cbc0f09a" gracePeriod=25 Nov 03 13:59:29.571059 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:59:29.570995 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-khw76" Nov 03 13:59:29.727642 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:59:29.727607 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/36118654-3637-489e-b048-af4aa366f34d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36118654-3637-489e-b048-af4aa366f34d-plugins-config-volume\") pod \"36118654-3637-489e-b048-af4aa366f34d\" (UID: \"36118654-3637-489e-b048-af4aa366f34d\") " Nov 03 13:59:29.727821 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:59:29.727705 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36118654-3637-489e-b048-af4aa366f34d-kube-api-access-rhmc8\" (UniqueName: \"kubernetes.io/projected/36118654-3637-489e-b048-af4aa366f34d-kube-api-access-rhmc8\") pod \"36118654-3637-489e-b048-af4aa366f34d\" (UID: \"36118654-3637-489e-b048-af4aa366f34d\") " Nov 03 13:59:29.728088 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:59:29.728062 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36118654-3637-489e-b048-af4aa366f34d-plugins-config-volume" pod "36118654-3637-489e-b048-af4aa366f34d" (UID: "36118654-3637-489e-b048-af4aa366f34d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:59:29.729897 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:59:29.729871 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36118654-3637-489e-b048-af4aa366f34d-kube-api-access-rhmc8" pod "36118654-3637-489e-b048-af4aa366f34d" (UID: "36118654-3637-489e-b048-af4aa366f34d"). InnerVolumeSpecName "kube-api-access-rhmc8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:59:29.829158 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:59:29.829076 5024 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36118654-3637-489e-b048-af4aa366f34d-plugins-config-volume\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:59:29.829158 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:59:29.829109 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rhmc8\" (UniqueName: \"kubernetes.io/projected/36118654-3637-489e-b048-af4aa366f34d-kube-api-access-rhmc8\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:59:30.347401 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:59:30.347366 5024 generic.go:356] "Generic (PLEG): container finished" podID="36118654-3637-489e-b048-af4aa366f34d" containerID="79e8003191720eef732cc991efc1dc83c996233a833c5121a47c0a85cbc0f09a" exitCode=0 Nov 03 13:59:30.347871 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:59:30.347414 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-khw76" Nov 03 13:59:30.347871 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:59:30.347451 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-khw76" event={"ID":"36118654-3637-489e-b048-af4aa366f34d","Type":"ContainerDied","Data":"79e8003191720eef732cc991efc1dc83c996233a833c5121a47c0a85cbc0f09a"} Nov 03 13:59:30.347871 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:59:30.347503 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-khw76" event={"ID":"36118654-3637-489e-b048-af4aa366f34d","Type":"ContainerDied","Data":"9449ddb9ff43f0e40f6783dec1afa165f7620367c34ce51fc455eb398d279e4d"} Nov 03 13:59:30.347871 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:59:30.347529 5024 scope.go:122] "RemoveContainer" containerID="79e8003191720eef732cc991efc1dc83c996233a833c5121a47c0a85cbc0f09a" Nov 03 13:59:30.356523 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:59:30.356489 5024 scope.go:122] "RemoveContainer" containerID="79e8003191720eef732cc991efc1dc83c996233a833c5121a47c0a85cbc0f09a" Nov 03 13:59:30.356783 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:59:30.356765 5024 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"79e8003191720eef732cc991efc1dc83c996233a833c5121a47c0a85cbc0f09a\": container with ID starting with 79e8003191720eef732cc991efc1dc83c996233a833c5121a47c0a85cbc0f09a not found: ID does not exist" containerID="79e8003191720eef732cc991efc1dc83c996233a833c5121a47c0a85cbc0f09a" Nov 03 13:59:30.356853 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:59:30.356790 5024 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79e8003191720eef732cc991efc1dc83c996233a833c5121a47c0a85cbc0f09a"} err="failed to get container status \"79e8003191720eef732cc991efc1dc83c996233a833c5121a47c0a85cbc0f09a\": rpc error: code = NotFound desc = could not find container \"79e8003191720eef732cc991efc1dc83c996233a833c5121a47c0a85cbc0f09a\": container with ID starting with 79e8003191720eef732cc991efc1dc83c996233a833c5121a47c0a85cbc0f09a not found: ID does not exist" Nov 03 13:59:30.368252 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:59:30.368189 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-khw76"] Nov 03 13:59:30.370074 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:59:30.370049 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-khw76"] Nov 03 13:59:32.107501 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:59:32.107467 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="36118654-3637-489e-b048-af4aa366f34d" path="/var/lib/kubelet/pods/36118654-3637-489e-b048-af4aa366f34d/volumes" Nov 03 13:59:38.338673 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:59:38.338636 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-67f556869f-kv4cs" Nov 03 13:59:38.339272 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:59:38.338723 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-776d6d99b-q7jqc" Nov 03 13:59:38.386890 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:59:38.386852 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-74d765fb54-fcczn"] Nov 03 13:59:38.412762 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:59:38.412731 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-5b44f68b67-gwbxk"] Nov 03 13:59:39.507795 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:59:39.507750 5024 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d7mwz" containerName="extractor" Nov 03 13:59:41.341789 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:59:41.341747 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-5bcd7997-h56sp"] Nov 03 13:59:41.342466 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:59:41.342445 5024 cpu_manager.go:414] "containerMap: removing container" podUID="36118654-3637-489e-b048-af4aa366f34d" containerName="payload-processing" Nov 03 13:59:41.342536 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:59:41.342470 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36118654-3637-489e-b048-af4aa366f34d" containerName="payload-processing" Nov 03 13:59:41.342620 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:59:41.342607 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="36118654-3637-489e-b048-af4aa366f34d" containerName="payload-processing" Nov 03 13:59:41.347673 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:59:41.347650 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5bcd7997-h56sp" Nov 03 13:59:41.358768 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:59:41.358739 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5bcd7997-h56sp"] Nov 03 13:59:41.363111 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:59:41.363085 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"e2e-rotation-external-model-api-key\"" Nov 03 13:59:41.363246 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:59:41.363126 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\"" Nov 03 13:59:41.363246 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:59:41.363174 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\"" Nov 03 13:59:41.363246 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:59:41.363178 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\"" Nov 03 13:59:41.363246 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:59:41.363225 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-mbjsn\"" Nov 03 13:59:41.363455 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:59:41.363266 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\"" Nov 03 13:59:41.439545 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:59:41.439497 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/542c62e1-47f9-4425-8e17-c7e2e72bc4ff-plugins-config-volume\") pod \"payload-processing-external-model-5bcd7997-h56sp\" (UID: \"542c62e1-47f9-4425-8e17-c7e2e72bc4ff\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-h56sp" Nov 03 13:59:41.439753 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:59:41.439551 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hv2rm\" (UniqueName: \"kubernetes.io/projected/542c62e1-47f9-4425-8e17-c7e2e72bc4ff-kube-api-access-hv2rm\") pod \"payload-processing-external-model-5bcd7997-h56sp\" (UID: \"542c62e1-47f9-4425-8e17-c7e2e72bc4ff\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-h56sp" Nov 03 13:59:41.439753 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:59:41.439657 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/542c62e1-47f9-4425-8e17-c7e2e72bc4ff-provider-credentials\") pod \"payload-processing-external-model-5bcd7997-h56sp\" (UID: \"542c62e1-47f9-4425-8e17-c7e2e72bc4ff\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-h56sp" Nov 03 13:59:41.439876 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:59:41.439807 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/542c62e1-47f9-4425-8e17-c7e2e72bc4ff-routing-overlay-volume\") pod \"payload-processing-external-model-5bcd7997-h56sp\" (UID: \"542c62e1-47f9-4425-8e17-c7e2e72bc4ff\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-h56sp" Nov 03 13:59:41.540465 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:59:41.540432 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/542c62e1-47f9-4425-8e17-c7e2e72bc4ff-plugins-config-volume\") pod \"payload-processing-external-model-5bcd7997-h56sp\" (UID: \"542c62e1-47f9-4425-8e17-c7e2e72bc4ff\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-h56sp" Nov 03 13:59:41.540465 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:59:41.540466 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hv2rm\" (UniqueName: \"kubernetes.io/projected/542c62e1-47f9-4425-8e17-c7e2e72bc4ff-kube-api-access-hv2rm\") pod \"payload-processing-external-model-5bcd7997-h56sp\" (UID: \"542c62e1-47f9-4425-8e17-c7e2e72bc4ff\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-h56sp" Nov 03 13:59:41.540776 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:59:41.540495 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/542c62e1-47f9-4425-8e17-c7e2e72bc4ff-provider-credentials\") pod \"payload-processing-external-model-5bcd7997-h56sp\" (UID: \"542c62e1-47f9-4425-8e17-c7e2e72bc4ff\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-h56sp" Nov 03 13:59:41.540776 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:59:41.540562 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/542c62e1-47f9-4425-8e17-c7e2e72bc4ff-routing-overlay-volume\") pod \"payload-processing-external-model-5bcd7997-h56sp\" (UID: \"542c62e1-47f9-4425-8e17-c7e2e72bc4ff\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-h56sp" Nov 03 13:59:41.541239 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:59:41.541216 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/542c62e1-47f9-4425-8e17-c7e2e72bc4ff-plugins-config-volume\") pod \"payload-processing-external-model-5bcd7997-h56sp\" (UID: \"542c62e1-47f9-4425-8e17-c7e2e72bc4ff\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-h56sp" Nov 03 13:59:41.541306 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:59:41.541269 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/542c62e1-47f9-4425-8e17-c7e2e72bc4ff-routing-overlay-volume\") pod \"payload-processing-external-model-5bcd7997-h56sp\" (UID: \"542c62e1-47f9-4425-8e17-c7e2e72bc4ff\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-h56sp" Nov 03 13:59:41.543019 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:59:41.543001 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/542c62e1-47f9-4425-8e17-c7e2e72bc4ff-provider-credentials\") pod \"payload-processing-external-model-5bcd7997-h56sp\" (UID: \"542c62e1-47f9-4425-8e17-c7e2e72bc4ff\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-h56sp" Nov 03 13:59:41.549714 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:59:41.549686 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hv2rm\" (UniqueName: \"kubernetes.io/projected/542c62e1-47f9-4425-8e17-c7e2e72bc4ff-kube-api-access-hv2rm\") pod \"payload-processing-external-model-5bcd7997-h56sp\" (UID: \"542c62e1-47f9-4425-8e17-c7e2e72bc4ff\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-h56sp" Nov 03 13:59:41.661965 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:59:41.661924 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5bcd7997-h56sp" Nov 03 13:59:41.944375 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:59:41.944340 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5bcd7997-h56sp"] Nov 03 13:59:41.946932 ip-10-0-131-36 kubenswrapper[5024]: W1103 13:59:41.946887 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod542c62e1_47f9_4425_8e17_c7e2e72bc4ff.slice/crio-8899b0a648ff8d3d11f39150b5a699a6c05c7068b05f02c9ca7e14e8bdacfe09 WatchSource:0}: Error finding container 8899b0a648ff8d3d11f39150b5a699a6c05c7068b05f02c9ca7e14e8bdacfe09: Status 404 returned error can't find the container with id 8899b0a648ff8d3d11f39150b5a699a6c05c7068b05f02c9ca7e14e8bdacfe09 Nov 03 13:59:42.438651 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:59:42.438596 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5bcd7997-h56sp" event={"ID":"542c62e1-47f9-4425-8e17-c7e2e72bc4ff","Type":"ContainerStarted","Data":"768ab52b18b7163915519395a96a997f4674b97129580e8bf9afa2628bae7081"} Nov 03 13:59:42.439165 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:59:42.438764 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-5bcd7997-h56sp" Nov 03 13:59:42.439165 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:59:42.438812 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5bcd7997-h56sp" event={"ID":"542c62e1-47f9-4425-8e17-c7e2e72bc4ff","Type":"ContainerStarted","Data":"8899b0a648ff8d3d11f39150b5a699a6c05c7068b05f02c9ca7e14e8bdacfe09"} Nov 03 13:59:42.458774 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:59:42.458723 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-5bcd7997-h56sp" podStartSLOduration=1.458707953 podStartE2EDuration="1.458707953s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:59:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:59:42.457267666 +0000 UTC m=+3062.465458944" watchObservedRunningTime="2026-11-03 13:59:42.458707953 +0000 UTC m=+3062.466899233" Nov 03 13:59:43.387269 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:59:43.387222 5024 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-74d765fb54-fcczn" podUID="c72cff36-b802-4c5a-b6ca-258eb6353616" containerName="payload-processing" containerID="cri-o://88911aa9969d89ea2bf71412e26c4a6492b9a91014f8da99cad06f33d950f91c" gracePeriod=25 Nov 03 13:59:43.415423 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:59:43.415353 5024 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-5b44f68b67-gwbxk" podUID="b1cb778e-d8e0-4d6c-8656-d7512ba3f1c8" containerName="payload-pre-processing" containerID="cri-o://6445f9e2d1a9b4bf61bd2ed11a36d470601ccbce9d06606d8b6cac21e000b4aa" gracePeriod=25 Nov 03 13:59:43.791144 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:59:43.791111 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-74d765fb54-fcczn" Nov 03 13:59:43.851945 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:59:43.850196 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-5b44f68b67-gwbxk" Nov 03 13:59:43.941681 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:59:43.941588 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c72cff36-b802-4c5a-b6ca-258eb6353616-kube-api-access-q77w9\" (UniqueName: \"kubernetes.io/projected/c72cff36-b802-4c5a-b6ca-258eb6353616-kube-api-access-q77w9\") pod \"c72cff36-b802-4c5a-b6ca-258eb6353616\" (UID: \"c72cff36-b802-4c5a-b6ca-258eb6353616\") " Nov 03 13:59:43.941988 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:59:43.941963 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c72cff36-b802-4c5a-b6ca-258eb6353616-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c72cff36-b802-4c5a-b6ca-258eb6353616-plugins-config-volume\") pod \"c72cff36-b802-4c5a-b6ca-258eb6353616\" (UID: \"c72cff36-b802-4c5a-b6ca-258eb6353616\") " Nov 03 13:59:43.946709 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:59:43.946684 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b1cb778e-d8e0-4d6c-8656-d7512ba3f1c8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1cb778e-d8e0-4d6c-8656-d7512ba3f1c8-plugins-config-volume\") pod \"b1cb778e-d8e0-4d6c-8656-d7512ba3f1c8\" (UID: \"b1cb778e-d8e0-4d6c-8656-d7512ba3f1c8\") " Nov 03 13:59:43.951664 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:59:43.951638 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b1cb778e-d8e0-4d6c-8656-d7512ba3f1c8-kube-api-access-8jfwz\" (UniqueName: \"kubernetes.io/projected/b1cb778e-d8e0-4d6c-8656-d7512ba3f1c8-kube-api-access-8jfwz\") pod \"b1cb778e-d8e0-4d6c-8656-d7512ba3f1c8\" (UID: \"b1cb778e-d8e0-4d6c-8656-d7512ba3f1c8\") " Nov 03 13:59:43.956515 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:59:43.946584 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c72cff36-b802-4c5a-b6ca-258eb6353616-plugins-config-volume" pod "c72cff36-b802-4c5a-b6ca-258eb6353616" (UID: "c72cff36-b802-4c5a-b6ca-258eb6353616"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:59:43.956661 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:59:43.951496 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1cb778e-d8e0-4d6c-8656-d7512ba3f1c8-plugins-config-volume" pod "b1cb778e-d8e0-4d6c-8656-d7512ba3f1c8" (UID: "b1cb778e-d8e0-4d6c-8656-d7512ba3f1c8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:59:43.956983 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:59:43.956963 5024 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c72cff36-b802-4c5a-b6ca-258eb6353616-plugins-config-volume\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:59:43.957105 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:59:43.957090 5024 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1cb778e-d8e0-4d6c-8656-d7512ba3f1c8-plugins-config-volume\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:59:44.167250 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:59:44.167199 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c72cff36-b802-4c5a-b6ca-258eb6353616-kube-api-access-q77w9" pod "c72cff36-b802-4c5a-b6ca-258eb6353616" (UID: "c72cff36-b802-4c5a-b6ca-258eb6353616"). InnerVolumeSpecName "kube-api-access-q77w9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:59:44.168455 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:59:44.168420 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1cb778e-d8e0-4d6c-8656-d7512ba3f1c8-kube-api-access-8jfwz" pod "b1cb778e-d8e0-4d6c-8656-d7512ba3f1c8" (UID: "b1cb778e-d8e0-4d6c-8656-d7512ba3f1c8"). InnerVolumeSpecName "kube-api-access-8jfwz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:59:44.270023 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:59:44.269917 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8jfwz\" (UniqueName: \"kubernetes.io/projected/b1cb778e-d8e0-4d6c-8656-d7512ba3f1c8-kube-api-access-8jfwz\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:59:44.294396 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:59:44.294355 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q77w9\" (UniqueName: \"kubernetes.io/projected/c72cff36-b802-4c5a-b6ca-258eb6353616-kube-api-access-q77w9\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 13:59:44.567546 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:59:44.567489 5024 generic.go:356] "Generic (PLEG): container finished" podID="b1cb778e-d8e0-4d6c-8656-d7512ba3f1c8" containerID="6445f9e2d1a9b4bf61bd2ed11a36d470601ccbce9d06606d8b6cac21e000b4aa" exitCode=0 Nov 03 13:59:44.567881 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:59:44.567852 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-5b44f68b67-gwbxk" event={"ID":"b1cb778e-d8e0-4d6c-8656-d7512ba3f1c8","Type":"ContainerDied","Data":"6445f9e2d1a9b4bf61bd2ed11a36d470601ccbce9d06606d8b6cac21e000b4aa"} Nov 03 13:59:44.568043 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:59:44.568026 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-5b44f68b67-gwbxk" event={"ID":"b1cb778e-d8e0-4d6c-8656-d7512ba3f1c8","Type":"ContainerDied","Data":"f854304e392fd3c45f4f0791cd44efab632a0622c329fbff7f45660655f19de8"} Nov 03 13:59:44.568158 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:59:44.568145 5024 scope.go:122] "RemoveContainer" containerID="6445f9e2d1a9b4bf61bd2ed11a36d470601ccbce9d06606d8b6cac21e000b4aa" Nov 03 13:59:44.568467 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:59:44.568446 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-5b44f68b67-gwbxk" Nov 03 13:59:44.594710 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:59:44.592587 5024 generic.go:356] "Generic (PLEG): container finished" podID="c72cff36-b802-4c5a-b6ca-258eb6353616" containerID="88911aa9969d89ea2bf71412e26c4a6492b9a91014f8da99cad06f33d950f91c" exitCode=0 Nov 03 13:59:44.594710 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:59:44.592690 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-74d765fb54-fcczn" event={"ID":"c72cff36-b802-4c5a-b6ca-258eb6353616","Type":"ContainerDied","Data":"88911aa9969d89ea2bf71412e26c4a6492b9a91014f8da99cad06f33d950f91c"} Nov 03 13:59:44.594710 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:59:44.592740 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-74d765fb54-fcczn" event={"ID":"c72cff36-b802-4c5a-b6ca-258eb6353616","Type":"ContainerDied","Data":"b06893cde9f4c7c22662f6ee85a67fa3f252df7bd7a35546a97c7f31cbf0ca7d"} Nov 03 13:59:44.594710 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:59:44.592842 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-74d765fb54-fcczn" Nov 03 13:59:44.610372 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:59:44.610329 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-5b44f68b67-gwbxk"] Nov 03 13:59:44.614136 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:59:44.614110 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-5b44f68b67-gwbxk"] Nov 03 13:59:44.616212 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:59:44.616185 5024 scope.go:122] "RemoveContainer" containerID="6445f9e2d1a9b4bf61bd2ed11a36d470601ccbce9d06606d8b6cac21e000b4aa" Nov 03 13:59:44.617001 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:59:44.616856 5024 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6445f9e2d1a9b4bf61bd2ed11a36d470601ccbce9d06606d8b6cac21e000b4aa\": container with ID starting with 6445f9e2d1a9b4bf61bd2ed11a36d470601ccbce9d06606d8b6cac21e000b4aa not found: ID does not exist" containerID="6445f9e2d1a9b4bf61bd2ed11a36d470601ccbce9d06606d8b6cac21e000b4aa" Nov 03 13:59:44.617001 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:59:44.616898 5024 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6445f9e2d1a9b4bf61bd2ed11a36d470601ccbce9d06606d8b6cac21e000b4aa"} err="failed to get container status \"6445f9e2d1a9b4bf61bd2ed11a36d470601ccbce9d06606d8b6cac21e000b4aa\": rpc error: code = NotFound desc = could not find container \"6445f9e2d1a9b4bf61bd2ed11a36d470601ccbce9d06606d8b6cac21e000b4aa\": container with ID starting with 6445f9e2d1a9b4bf61bd2ed11a36d470601ccbce9d06606d8b6cac21e000b4aa not found: ID does not exist" Nov 03 13:59:44.617001 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:59:44.616921 5024 scope.go:122] "RemoveContainer" containerID="88911aa9969d89ea2bf71412e26c4a6492b9a91014f8da99cad06f33d950f91c" Nov 03 13:59:44.652917 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:59:44.652716 5024 scope.go:122] "RemoveContainer" containerID="88911aa9969d89ea2bf71412e26c4a6492b9a91014f8da99cad06f33d950f91c" Nov 03 13:59:44.652917 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:59:44.652858 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-74d765fb54-fcczn"] Nov 03 13:59:44.654192 ip-10-0-131-36 kubenswrapper[5024]: E1103 13:59:44.654104 5024 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"88911aa9969d89ea2bf71412e26c4a6492b9a91014f8da99cad06f33d950f91c\": container with ID starting with 88911aa9969d89ea2bf71412e26c4a6492b9a91014f8da99cad06f33d950f91c not found: ID does not exist" containerID="88911aa9969d89ea2bf71412e26c4a6492b9a91014f8da99cad06f33d950f91c" Nov 03 13:59:44.654192 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:59:44.654142 5024 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88911aa9969d89ea2bf71412e26c4a6492b9a91014f8da99cad06f33d950f91c"} err="failed to get container status \"88911aa9969d89ea2bf71412e26c4a6492b9a91014f8da99cad06f33d950f91c\": rpc error: code = NotFound desc = could not find container \"88911aa9969d89ea2bf71412e26c4a6492b9a91014f8da99cad06f33d950f91c\": container with ID starting with 88911aa9969d89ea2bf71412e26c4a6492b9a91014f8da99cad06f33d950f91c not found: ID does not exist" Nov 03 13:59:44.675319 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:59:44.671110 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-74d765fb54-fcczn"] Nov 03 13:59:46.114167 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:59:46.114123 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b1cb778e-d8e0-4d6c-8656-d7512ba3f1c8" path="/var/lib/kubelet/pods/b1cb778e-d8e0-4d6c-8656-d7512ba3f1c8/volumes" Nov 03 13:59:46.114774 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:59:46.114748 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c72cff36-b802-4c5a-b6ca-258eb6353616" path="/var/lib/kubelet/pods/c72cff36-b802-4c5a-b6ca-258eb6353616/volumes" Nov 03 13:59:46.182979 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:59:46.182939 5024 scope.go:122] "RemoveContainer" containerID="ddd893680d229e89241431680ca35705de2e44615da723d4ca40d5d9450b5556" Nov 03 13:59:53.449896 ip-10-0-131-36 kubenswrapper[5024]: I1103 13:59:53.449859 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-5bcd7997-h56sp" Nov 03 14:00:00.025644 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:00:00.025605 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-fg58r"] Nov 03 14:00:00.029155 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:00:00.029126 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-fg58r"] Nov 03 14:00:00.108406 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:00:00.108369 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="796ad0b0-d02e-46b7-8018-c252134da079" path="/var/lib/kubelet/pods/796ad0b0-d02e-46b7-8018-c252134da079/volumes" Nov 03 14:00:00.138692 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:00:00.138655 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895240-p2nj4"] Nov 03 14:00:00.139083 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:00:00.139069 5024 cpu_manager.go:414] "containerMap: removing container" podUID="c72cff36-b802-4c5a-b6ca-258eb6353616" containerName="payload-processing" Nov 03 14:00:00.139131 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:00:00.139083 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c72cff36-b802-4c5a-b6ca-258eb6353616" containerName="payload-processing" Nov 03 14:00:00.139131 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:00:00.139110 5024 cpu_manager.go:414] "containerMap: removing container" podUID="b1cb778e-d8e0-4d6c-8656-d7512ba3f1c8" containerName="payload-pre-processing" Nov 03 14:00:00.139131 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:00:00.139115 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1cb778e-d8e0-4d6c-8656-d7512ba3f1c8" containerName="payload-pre-processing" Nov 03 14:00:00.139253 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:00:00.139174 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="b1cb778e-d8e0-4d6c-8656-d7512ba3f1c8" containerName="payload-pre-processing" Nov 03 14:00:00.139253 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:00:00.139186 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="c72cff36-b802-4c5a-b6ca-258eb6353616" containerName="payload-processing" Nov 03 14:00:00.142400 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:00:00.142384 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895240-p2nj4" Nov 03 14:00:00.150237 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:00:00.150196 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 14:00:00.150387 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:00:00.150252 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 14:00:00.153992 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:00:00.152462 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w6vqv\"" Nov 03 14:00:00.154204 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:00:00.154175 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895240-p2nj4"] Nov 03 14:00:00.265637 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:00:00.265585 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/782a71da-d497-447d-abd1-b5e2a146976c-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895240-p2nj4\" (UID: \"782a71da-d497-447d-abd1-b5e2a146976c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895240-p2nj4" Nov 03 14:00:00.265833 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:00:00.265652 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sgbfg\" (UniqueName: \"kubernetes.io/projected/782a71da-d497-447d-abd1-b5e2a146976c-kube-api-access-sgbfg\") pod \"maas-api-key-cleanup-29895240-p2nj4\" (UID: \"782a71da-d497-447d-abd1-b5e2a146976c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895240-p2nj4" Nov 03 14:00:00.366969 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:00:00.366932 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/782a71da-d497-447d-abd1-b5e2a146976c-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895240-p2nj4\" (UID: \"782a71da-d497-447d-abd1-b5e2a146976c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895240-p2nj4" Nov 03 14:00:00.367152 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:00:00.366997 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sgbfg\" (UniqueName: \"kubernetes.io/projected/782a71da-d497-447d-abd1-b5e2a146976c-kube-api-access-sgbfg\") pod \"maas-api-key-cleanup-29895240-p2nj4\" (UID: \"782a71da-d497-447d-abd1-b5e2a146976c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895240-p2nj4" Nov 03 14:00:00.367659 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:00:00.367635 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/782a71da-d497-447d-abd1-b5e2a146976c-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895240-p2nj4\" (UID: \"782a71da-d497-447d-abd1-b5e2a146976c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895240-p2nj4" Nov 03 14:00:00.377960 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:00:00.377925 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sgbfg\" (UniqueName: \"kubernetes.io/projected/782a71da-d497-447d-abd1-b5e2a146976c-kube-api-access-sgbfg\") pod \"maas-api-key-cleanup-29895240-p2nj4\" (UID: \"782a71da-d497-447d-abd1-b5e2a146976c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895240-p2nj4" Nov 03 14:00:00.458557 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:00:00.458506 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895240-p2nj4" Nov 03 14:00:00.734053 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:00:00.734006 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895240-p2nj4"] Nov 03 14:00:00.736608 ip-10-0-131-36 kubenswrapper[5024]: W1103 14:00:00.736574 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod782a71da_d497_447d_abd1_b5e2a146976c.slice/crio-dd4fea00562346050ecc42880254e4509d97df37d94b33e368a1e30a17bbc7fc WatchSource:0}: Error finding container dd4fea00562346050ecc42880254e4509d97df37d94b33e368a1e30a17bbc7fc: Status 404 returned error can't find the container with id dd4fea00562346050ecc42880254e4509d97df37d94b33e368a1e30a17bbc7fc Nov 03 14:00:01.693344 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:00:01.693264 5024 generic.go:356] "Generic (PLEG): container finished" podID="782a71da-d497-447d-abd1-b5e2a146976c" containerID="a3d02e4050087c4e039c46c79c98ed2b5d200a1f9bdb7bfdc9253640c847a7f5" exitCode=60 Nov 03 14:00:01.693805 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:00:01.693355 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895240-p2nj4" event={"ID":"782a71da-d497-447d-abd1-b5e2a146976c","Type":"ContainerDied","Data":"a3d02e4050087c4e039c46c79c98ed2b5d200a1f9bdb7bfdc9253640c847a7f5"} Nov 03 14:00:01.693805 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:00:01.693409 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895240-p2nj4" event={"ID":"782a71da-d497-447d-abd1-b5e2a146976c","Type":"ContainerStarted","Data":"dd4fea00562346050ecc42880254e4509d97df37d94b33e368a1e30a17bbc7fc"} Nov 03 14:00:01.693805 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:00:01.693576 5024 scope.go:122] "RemoveContainer" containerID="a3d02e4050087c4e039c46c79c98ed2b5d200a1f9bdb7bfdc9253640c847a7f5" Nov 03 14:00:02.709407 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:00:02.709315 5024 generic.go:356] "Generic (PLEG): container finished" podID="782a71da-d497-447d-abd1-b5e2a146976c" containerID="f11d61395c5e15bcd4ccab19f73acb16207ca514b2a607079d06c03410a74f35" exitCode=60 Nov 03 14:00:02.709797 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:00:02.709401 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895240-p2nj4" event={"ID":"782a71da-d497-447d-abd1-b5e2a146976c","Type":"ContainerDied","Data":"f11d61395c5e15bcd4ccab19f73acb16207ca514b2a607079d06c03410a74f35"} Nov 03 14:00:02.709797 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:00:02.709478 5024 scope.go:122] "RemoveContainer" containerID="a3d02e4050087c4e039c46c79c98ed2b5d200a1f9bdb7bfdc9253640c847a7f5" Nov 03 14:00:02.709797 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:00:02.709633 5024 scope.go:122] "RemoveContainer" containerID="f11d61395c5e15bcd4ccab19f73acb16207ca514b2a607079d06c03410a74f35" Nov 03 14:00:02.709975 ip-10-0-131-36 kubenswrapper[5024]: E1103 14:00:02.709883 5024 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895240-p2nj4_odh-ai-gateway-infra(782a71da-d497-447d-abd1-b5e2a146976c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895240-p2nj4" podUID="782a71da-d497-447d-abd1-b5e2a146976c" Nov 03 14:00:03.716003 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:00:03.715975 5024 scope.go:122] "RemoveContainer" containerID="f11d61395c5e15bcd4ccab19f73acb16207ca514b2a607079d06c03410a74f35" Nov 03 14:00:03.716419 ip-10-0-131-36 kubenswrapper[5024]: E1103 14:00:03.716201 5024 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895240-p2nj4_odh-ai-gateway-infra(782a71da-d497-447d-abd1-b5e2a146976c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895240-p2nj4" podUID="782a71da-d497-447d-abd1-b5e2a146976c" Nov 03 14:00:09.506469 ip-10-0-131-36 kubenswrapper[5024]: E1103 14:00:09.506380 5024 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d7mwz" containerName="extractor" Nov 03 14:00:13.031145 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:00:13.031105 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-xqg8w"] Nov 03 14:00:13.036065 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:00:13.036037 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-xqg8w"] Nov 03 14:00:14.101953 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:00:14.101915 5024 scope.go:122] "RemoveContainer" containerID="f11d61395c5e15bcd4ccab19f73acb16207ca514b2a607079d06c03410a74f35" Nov 03 14:00:14.108452 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:00:14.108420 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a6ad3d1-e739-4ace-b7f6-13d3fefdb87d" path="/var/lib/kubelet/pods/2a6ad3d1-e739-4ace-b7f6-13d3fefdb87d/volumes" Nov 03 14:00:14.770172 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:00:14.770090 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895240-p2nj4" event={"ID":"782a71da-d497-447d-abd1-b5e2a146976c","Type":"ContainerStarted","Data":"72456e110e056ca5e2eeb2f3613204994bc717018a384f8187551d80dff10ceb"} Nov 03 14:00:14.795531 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:00:14.795479 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895240-p2nj4" podStartSLOduration=14.795466389 podStartE2EDuration="14.795466389s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:00:14.794294007 +0000 UTC m=+3094.802485286" watchObservedRunningTime="2026-11-03 14:00:14.795466389 +0000 UTC m=+3094.803657660" Nov 03 14:00:15.785554 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:00:15.785517 5024 generic.go:356] "Generic (PLEG): container finished" podID="782a71da-d497-447d-abd1-b5e2a146976c" containerID="72456e110e056ca5e2eeb2f3613204994bc717018a384f8187551d80dff10ceb" exitCode=60 Nov 03 14:00:15.786002 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:00:15.785592 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895240-p2nj4" event={"ID":"782a71da-d497-447d-abd1-b5e2a146976c","Type":"ContainerDied","Data":"72456e110e056ca5e2eeb2f3613204994bc717018a384f8187551d80dff10ceb"} Nov 03 14:00:15.786002 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:00:15.785679 5024 scope.go:122] "RemoveContainer" containerID="f11d61395c5e15bcd4ccab19f73acb16207ca514b2a607079d06c03410a74f35" Nov 03 14:00:15.786002 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:00:15.785975 5024 scope.go:122] "RemoveContainer" containerID="72456e110e056ca5e2eeb2f3613204994bc717018a384f8187551d80dff10ceb" Nov 03 14:00:15.786261 ip-10-0-131-36 kubenswrapper[5024]: E1103 14:00:15.786208 5024 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29895240-p2nj4_odh-ai-gateway-infra(782a71da-d497-447d-abd1-b5e2a146976c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895240-p2nj4" podUID="782a71da-d497-447d-abd1-b5e2a146976c" Nov 03 14:00:15.805207 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:00:15.805168 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895240-p2nj4"] Nov 03 14:00:16.930450 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:00:16.930425 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895240-p2nj4" Nov 03 14:00:17.035162 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:00:17.035127 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/782a71da-d497-447d-abd1-b5e2a146976c-kube-api-access-sgbfg\" (UniqueName: \"kubernetes.io/projected/782a71da-d497-447d-abd1-b5e2a146976c-kube-api-access-sgbfg\") pod \"782a71da-d497-447d-abd1-b5e2a146976c\" (UID: \"782a71da-d497-447d-abd1-b5e2a146976c\") " Nov 03 14:00:17.035360 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:00:17.035251 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/782a71da-d497-447d-abd1-b5e2a146976c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/782a71da-d497-447d-abd1-b5e2a146976c-maas-api-service-ca\") pod \"782a71da-d497-447d-abd1-b5e2a146976c\" (UID: \"782a71da-d497-447d-abd1-b5e2a146976c\") " Nov 03 14:00:17.035586 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:00:17.035561 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/782a71da-d497-447d-abd1-b5e2a146976c-maas-api-service-ca" pod "782a71da-d497-447d-abd1-b5e2a146976c" (UID: "782a71da-d497-447d-abd1-b5e2a146976c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:00:17.037330 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:00:17.037269 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/782a71da-d497-447d-abd1-b5e2a146976c-kube-api-access-sgbfg" pod "782a71da-d497-447d-abd1-b5e2a146976c" (UID: "782a71da-d497-447d-abd1-b5e2a146976c"). InnerVolumeSpecName "kube-api-access-sgbfg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:00:17.136148 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:00:17.136114 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sgbfg\" (UniqueName: \"kubernetes.io/projected/782a71da-d497-447d-abd1-b5e2a146976c-kube-api-access-sgbfg\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 14:00:17.136148 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:00:17.136142 5024 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/782a71da-d497-447d-abd1-b5e2a146976c-maas-api-service-ca\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 14:00:17.797647 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:00:17.797609 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895240-p2nj4" Nov 03 14:00:17.797854 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:00:17.797638 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895240-p2nj4" event={"ID":"782a71da-d497-447d-abd1-b5e2a146976c","Type":"ContainerDied","Data":"dd4fea00562346050ecc42880254e4509d97df37d94b33e368a1e30a17bbc7fc"} Nov 03 14:00:17.797854 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:00:17.797705 5024 scope.go:122] "RemoveContainer" containerID="72456e110e056ca5e2eeb2f3613204994bc717018a384f8187551d80dff10ceb" Nov 03 14:00:17.822356 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:00:17.822322 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895240-p2nj4"] Nov 03 14:00:17.828945 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:00:17.828918 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895240-p2nj4"] Nov 03 14:00:18.107753 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:00:18.107726 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="782a71da-d497-447d-abd1-b5e2a146976c" path="/var/lib/kubelet/pods/782a71da-d497-447d-abd1-b5e2a146976c/volumes" Nov 03 14:00:30.024152 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:00:30.024111 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3740ae17-khkcv"] Nov 03 14:00:30.028741 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:00:30.028714 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3740ae17-khkcv"] Nov 03 14:00:30.108478 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:00:30.108439 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2231d96c-22b1-410c-b663-d5f015bf7b43" path="/var/lib/kubelet/pods/2231d96c-22b1-410c-b663-d5f015bf7b43/volumes" Nov 03 14:00:39.505858 ip-10-0-131-36 kubenswrapper[5024]: E1103 14:00:39.505820 5024 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d7mwz" containerName="extractor" Nov 03 14:01:09.506932 ip-10-0-131-36 kubenswrapper[5024]: E1103 14:01:09.506893 5024 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d7mwz" containerName="extractor" Nov 03 14:01:15.724086 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:01:15.724050 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-qxwsd"] Nov 03 14:01:15.724527 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:01:15.724513 5024 cpu_manager.go:414] "containerMap: removing container" podUID="782a71da-d497-447d-abd1-b5e2a146976c" containerName="cleanup" Nov 03 14:01:15.724573 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:01:15.724529 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="782a71da-d497-447d-abd1-b5e2a146976c" containerName="cleanup" Nov 03 14:01:15.724573 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:01:15.724544 5024 cpu_manager.go:414] "containerMap: removing container" podUID="782a71da-d497-447d-abd1-b5e2a146976c" containerName="cleanup" Nov 03 14:01:15.724573 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:01:15.724549 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="782a71da-d497-447d-abd1-b5e2a146976c" containerName="cleanup" Nov 03 14:01:15.724573 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:01:15.724559 5024 cpu_manager.go:414] "containerMap: removing container" podUID="782a71da-d497-447d-abd1-b5e2a146976c" containerName="cleanup" Nov 03 14:01:15.724573 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:01:15.724564 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="782a71da-d497-447d-abd1-b5e2a146976c" containerName="cleanup" Nov 03 14:01:15.724737 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:01:15.724635 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="782a71da-d497-447d-abd1-b5e2a146976c" containerName="cleanup" Nov 03 14:01:15.724737 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:01:15.724644 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="782a71da-d497-447d-abd1-b5e2a146976c" containerName="cleanup" Nov 03 14:01:15.727906 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:01:15.727884 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-qxwsd" Nov 03 14:01:15.736224 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:01:15.736201 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w6vqv\"" Nov 03 14:01:15.736224 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:01:15.736201 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 14:01:15.736464 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:01:15.736409 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 14:01:15.736998 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:01:15.736936 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 14:01:15.746257 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:01:15.740916 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-qxwsd"] Nov 03 14:01:15.806990 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:01:15.806953 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d5f05b0-68e0-486a-babd-20c35127c2e5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2547622-qxwsd\" (UID: \"8d5f05b0-68e0-486a-babd-20c35127c2e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-qxwsd" Nov 03 14:01:15.807132 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:01:15.807025 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l2xlb\" (UniqueName: \"kubernetes.io/projected/8d5f05b0-68e0-486a-babd-20c35127c2e5-kube-api-access-l2xlb\") pod \"maas-api-revoke-sub-e2547622-qxwsd\" (UID: \"8d5f05b0-68e0-486a-babd-20c35127c2e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-qxwsd" Nov 03 14:01:15.807132 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:01:15.807120 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d5f05b0-68e0-486a-babd-20c35127c2e5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2547622-qxwsd\" (UID: \"8d5f05b0-68e0-486a-babd-20c35127c2e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-qxwsd" Nov 03 14:01:15.908136 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:01:15.908102 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d5f05b0-68e0-486a-babd-20c35127c2e5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2547622-qxwsd\" (UID: \"8d5f05b0-68e0-486a-babd-20c35127c2e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-qxwsd" Nov 03 14:01:15.908275 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:01:15.908196 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d5f05b0-68e0-486a-babd-20c35127c2e5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2547622-qxwsd\" (UID: \"8d5f05b0-68e0-486a-babd-20c35127c2e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-qxwsd" Nov 03 14:01:15.908275 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:01:15.908255 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2xlb\" (UniqueName: \"kubernetes.io/projected/8d5f05b0-68e0-486a-babd-20c35127c2e5-kube-api-access-l2xlb\") pod \"maas-api-revoke-sub-e2547622-qxwsd\" (UID: \"8d5f05b0-68e0-486a-babd-20c35127c2e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-qxwsd" Nov 03 14:01:15.908799 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:01:15.908774 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d5f05b0-68e0-486a-babd-20c35127c2e5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2547622-qxwsd\" (UID: \"8d5f05b0-68e0-486a-babd-20c35127c2e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-qxwsd" Nov 03 14:01:15.910615 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:01:15.910583 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d5f05b0-68e0-486a-babd-20c35127c2e5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2547622-qxwsd\" (UID: \"8d5f05b0-68e0-486a-babd-20c35127c2e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-qxwsd" Nov 03 14:01:15.917706 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:01:15.917682 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l2xlb\" (UniqueName: \"kubernetes.io/projected/8d5f05b0-68e0-486a-babd-20c35127c2e5-kube-api-access-l2xlb\") pod \"maas-api-revoke-sub-e2547622-qxwsd\" (UID: \"8d5f05b0-68e0-486a-babd-20c35127c2e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-qxwsd" Nov 03 14:01:16.052257 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:01:16.052188 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-qxwsd" Nov 03 14:01:16.327169 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:01:16.327136 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-qxwsd"] Nov 03 14:01:16.328452 ip-10-0-131-36 kubenswrapper[5024]: W1103 14:01:16.328417 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d5f05b0_68e0_486a_babd_20c35127c2e5.slice/crio-ce0f9dd26c67f10ec9818912ec8f27647a167a792fe807c676d34bd99713d840 WatchSource:0}: Error finding container ce0f9dd26c67f10ec9818912ec8f27647a167a792fe807c676d34bd99713d840: Status 404 returned error can't find the container with id ce0f9dd26c67f10ec9818912ec8f27647a167a792fe807c676d34bd99713d840 Nov 03 14:01:17.090508 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:01:17.090468 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-qxwsd" event={"ID":"8d5f05b0-68e0-486a-babd-20c35127c2e5","Type":"ContainerStarted","Data":"eb5d0e5d2fa1622740dce4eeb5553fb9eef197f0f0ab331a2ee64cb10956d234"} Nov 03 14:01:17.090959 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:01:17.090537 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-qxwsd" event={"ID":"8d5f05b0-68e0-486a-babd-20c35127c2e5","Type":"ContainerStarted","Data":"ce0f9dd26c67f10ec9818912ec8f27647a167a792fe807c676d34bd99713d840"} Nov 03 14:01:17.114807 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:01:17.114760 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-qxwsd" podStartSLOduration=2.114746565 podStartE2EDuration="2.114746565s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:01:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:01:17.112838633 +0000 UTC m=+3157.121029909" watchObservedRunningTime="2026-11-03 14:01:17.114746565 +0000 UTC m=+3157.122937842" Nov 03 14:01:18.095953 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:01:18.095913 5024 generic.go:356] "Generic (PLEG): container finished" podID="8d5f05b0-68e0-486a-babd-20c35127c2e5" containerID="eb5d0e5d2fa1622740dce4eeb5553fb9eef197f0f0ab331a2ee64cb10956d234" exitCode=0 Nov 03 14:01:18.096357 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:01:18.095997 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-qxwsd" event={"ID":"8d5f05b0-68e0-486a-babd-20c35127c2e5","Type":"ContainerDied","Data":"eb5d0e5d2fa1622740dce4eeb5553fb9eef197f0f0ab331a2ee64cb10956d234"} Nov 03 14:01:19.233160 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:01:19.233131 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-qxwsd" Nov 03 14:01:19.344274 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:01:19.344242 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d5f05b0-68e0-486a-babd-20c35127c2e5-kube-api-access-l2xlb\" (UniqueName: \"kubernetes.io/projected/8d5f05b0-68e0-486a-babd-20c35127c2e5-kube-api-access-l2xlb\") pod \"8d5f05b0-68e0-486a-babd-20c35127c2e5\" (UID: \"8d5f05b0-68e0-486a-babd-20c35127c2e5\") " Nov 03 14:01:19.344460 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:01:19.344336 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d5f05b0-68e0-486a-babd-20c35127c2e5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d5f05b0-68e0-486a-babd-20c35127c2e5-maas-api-openshift-ca\") pod \"8d5f05b0-68e0-486a-babd-20c35127c2e5\" (UID: \"8d5f05b0-68e0-486a-babd-20c35127c2e5\") " Nov 03 14:01:19.344507 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:01:19.344489 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8d5f05b0-68e0-486a-babd-20c35127c2e5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d5f05b0-68e0-486a-babd-20c35127c2e5-maas-api-xks-ca\") pod \"8d5f05b0-68e0-486a-babd-20c35127c2e5\" (UID: \"8d5f05b0-68e0-486a-babd-20c35127c2e5\") " Nov 03 14:01:19.344749 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:01:19.344724 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d5f05b0-68e0-486a-babd-20c35127c2e5-maas-api-openshift-ca" pod "8d5f05b0-68e0-486a-babd-20c35127c2e5" (UID: "8d5f05b0-68e0-486a-babd-20c35127c2e5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:01:19.344846 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:01:19.344833 5024 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d5f05b0-68e0-486a-babd-20c35127c2e5-maas-api-openshift-ca\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 14:01:19.346337 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:01:19.346310 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d5f05b0-68e0-486a-babd-20c35127c2e5-kube-api-access-l2xlb" pod "8d5f05b0-68e0-486a-babd-20c35127c2e5" (UID: "8d5f05b0-68e0-486a-babd-20c35127c2e5"). InnerVolumeSpecName "kube-api-access-l2xlb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:01:19.346430 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:01:19.346389 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d5f05b0-68e0-486a-babd-20c35127c2e5-maas-api-xks-ca" pod "8d5f05b0-68e0-486a-babd-20c35127c2e5" (UID: "8d5f05b0-68e0-486a-babd-20c35127c2e5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:01:19.445783 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:01:19.445750 5024 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d5f05b0-68e0-486a-babd-20c35127c2e5-maas-api-xks-ca\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 14:01:19.445783 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:01:19.445780 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l2xlb\" (UniqueName: \"kubernetes.io/projected/8d5f05b0-68e0-486a-babd-20c35127c2e5-kube-api-access-l2xlb\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 14:01:20.105975 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:01:20.105946 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-qxwsd" Nov 03 14:01:20.108238 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:01:20.108206 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-qxwsd" event={"ID":"8d5f05b0-68e0-486a-babd-20c35127c2e5","Type":"ContainerDied","Data":"ce0f9dd26c67f10ec9818912ec8f27647a167a792fe807c676d34bd99713d840"} Nov 03 14:01:20.108430 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:01:20.108248 5024 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ce0f9dd26c67f10ec9818912ec8f27647a167a792fe807c676d34bd99713d840" Nov 03 14:01:26.414112 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:01:26.414076 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5bcd7997-h56sp"] Nov 03 14:01:31.414797 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:01:31.414749 5024 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-5bcd7997-h56sp" podUID="542c62e1-47f9-4425-8e17-c7e2e72bc4ff" containerName="payload-processing" containerID="cri-o://768ab52b18b7163915519395a96a997f4674b97129580e8bf9afa2628bae7081" gracePeriod=25 Nov 03 14:01:31.779843 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:01:31.779490 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5bcd7997-h56sp" Nov 03 14:01:31.940845 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:01:31.940797 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/542c62e1-47f9-4425-8e17-c7e2e72bc4ff-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/542c62e1-47f9-4425-8e17-c7e2e72bc4ff-routing-overlay-volume\") pod \"542c62e1-47f9-4425-8e17-c7e2e72bc4ff\" (UID: \"542c62e1-47f9-4425-8e17-c7e2e72bc4ff\") " Nov 03 14:01:31.941057 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:01:31.940891 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/542c62e1-47f9-4425-8e17-c7e2e72bc4ff-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/542c62e1-47f9-4425-8e17-c7e2e72bc4ff-plugins-config-volume\") pod \"542c62e1-47f9-4425-8e17-c7e2e72bc4ff\" (UID: \"542c62e1-47f9-4425-8e17-c7e2e72bc4ff\") " Nov 03 14:01:31.941057 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:01:31.941029 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/542c62e1-47f9-4425-8e17-c7e2e72bc4ff-kube-api-access-hv2rm\" (UniqueName: \"kubernetes.io/projected/542c62e1-47f9-4425-8e17-c7e2e72bc4ff-kube-api-access-hv2rm\") pod \"542c62e1-47f9-4425-8e17-c7e2e72bc4ff\" (UID: \"542c62e1-47f9-4425-8e17-c7e2e72bc4ff\") " Nov 03 14:01:31.941192 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:01:31.941071 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/542c62e1-47f9-4425-8e17-c7e2e72bc4ff-provider-credentials\" (UniqueName: \"kubernetes.io/projected/542c62e1-47f9-4425-8e17-c7e2e72bc4ff-provider-credentials\") pod \"542c62e1-47f9-4425-8e17-c7e2e72bc4ff\" (UID: \"542c62e1-47f9-4425-8e17-c7e2e72bc4ff\") " Nov 03 14:01:31.942693 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:01:31.942659 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/542c62e1-47f9-4425-8e17-c7e2e72bc4ff-routing-overlay-volume" pod "542c62e1-47f9-4425-8e17-c7e2e72bc4ff" (UID: "542c62e1-47f9-4425-8e17-c7e2e72bc4ff"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:01:31.943040 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:01:31.943013 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/542c62e1-47f9-4425-8e17-c7e2e72bc4ff-plugins-config-volume" pod "542c62e1-47f9-4425-8e17-c7e2e72bc4ff" (UID: "542c62e1-47f9-4425-8e17-c7e2e72bc4ff"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:01:31.944622 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:01:31.944594 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/542c62e1-47f9-4425-8e17-c7e2e72bc4ff-kube-api-access-hv2rm" pod "542c62e1-47f9-4425-8e17-c7e2e72bc4ff" (UID: "542c62e1-47f9-4425-8e17-c7e2e72bc4ff"). InnerVolumeSpecName "kube-api-access-hv2rm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:01:31.944987 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:01:31.944960 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/542c62e1-47f9-4425-8e17-c7e2e72bc4ff-provider-credentials" pod "542c62e1-47f9-4425-8e17-c7e2e72bc4ff" (UID: "542c62e1-47f9-4425-8e17-c7e2e72bc4ff"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:01:32.042802 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:01:32.042700 5024 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/542c62e1-47f9-4425-8e17-c7e2e72bc4ff-routing-overlay-volume\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 14:01:32.042802 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:01:32.042745 5024 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/542c62e1-47f9-4425-8e17-c7e2e72bc4ff-plugins-config-volume\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 14:01:32.042802 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:01:32.042761 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hv2rm\" (UniqueName: \"kubernetes.io/projected/542c62e1-47f9-4425-8e17-c7e2e72bc4ff-kube-api-access-hv2rm\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 14:01:32.042802 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:01:32.042776 5024 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/542c62e1-47f9-4425-8e17-c7e2e72bc4ff-provider-credentials\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 14:01:32.173898 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:01:32.169720 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-8ggxb"] Nov 03 14:01:32.173898 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:01:32.170473 5024 cpu_manager.go:414] "containerMap: removing container" podUID="542c62e1-47f9-4425-8e17-c7e2e72bc4ff" containerName="payload-processing" Nov 03 14:01:32.173898 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:01:32.170496 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="542c62e1-47f9-4425-8e17-c7e2e72bc4ff" containerName="payload-processing" Nov 03 14:01:32.173898 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:01:32.170571 5024 cpu_manager.go:414] "containerMap: removing container" podUID="8d5f05b0-68e0-486a-babd-20c35127c2e5" containerName="revoke-keys" Nov 03 14:01:32.173898 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:01:32.170581 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d5f05b0-68e0-486a-babd-20c35127c2e5" containerName="revoke-keys" Nov 03 14:01:32.173898 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:01:32.170688 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d5f05b0-68e0-486a-babd-20c35127c2e5" containerName="revoke-keys" Nov 03 14:01:32.173898 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:01:32.170703 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="782a71da-d497-447d-abd1-b5e2a146976c" containerName="cleanup" Nov 03 14:01:32.173898 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:01:32.170715 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="542c62e1-47f9-4425-8e17-c7e2e72bc4ff" containerName="payload-processing" Nov 03 14:01:32.191323 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:01:32.180590 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-8ggxb" Nov 03 14:01:32.192969 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:01:32.192014 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 14:01:32.198314 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:01:32.196542 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-8ggxb"] Nov 03 14:01:32.198314 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:01:32.197195 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 14:01:32.198314 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:01:32.197370 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w6vqv\"" Nov 03 14:01:32.198314 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:01:32.197443 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 14:01:32.223775 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:01:32.222231 5024 generic.go:356] "Generic (PLEG): container finished" podID="542c62e1-47f9-4425-8e17-c7e2e72bc4ff" containerID="768ab52b18b7163915519395a96a997f4674b97129580e8bf9afa2628bae7081" exitCode=0 Nov 03 14:01:32.223775 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:01:32.222346 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5bcd7997-h56sp" event={"ID":"542c62e1-47f9-4425-8e17-c7e2e72bc4ff","Type":"ContainerDied","Data":"768ab52b18b7163915519395a96a997f4674b97129580e8bf9afa2628bae7081"} Nov 03 14:01:32.223775 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:01:32.222398 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5bcd7997-h56sp" event={"ID":"542c62e1-47f9-4425-8e17-c7e2e72bc4ff","Type":"ContainerDied","Data":"8899b0a648ff8d3d11f39150b5a699a6c05c7068b05f02c9ca7e14e8bdacfe09"} Nov 03 14:01:32.223775 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:01:32.222428 5024 scope.go:122] "RemoveContainer" containerID="768ab52b18b7163915519395a96a997f4674b97129580e8bf9afa2628bae7081" Nov 03 14:01:32.223775 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:01:32.222594 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5bcd7997-h56sp" Nov 03 14:01:32.247967 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:01:32.247939 5024 scope.go:122] "RemoveContainer" containerID="768ab52b18b7163915519395a96a997f4674b97129580e8bf9afa2628bae7081" Nov 03 14:01:32.248555 ip-10-0-131-36 kubenswrapper[5024]: E1103 14:01:32.248476 5024 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"768ab52b18b7163915519395a96a997f4674b97129580e8bf9afa2628bae7081\": container with ID starting with 768ab52b18b7163915519395a96a997f4674b97129580e8bf9afa2628bae7081 not found: ID does not exist" containerID="768ab52b18b7163915519395a96a997f4674b97129580e8bf9afa2628bae7081" Nov 03 14:01:32.248555 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:01:32.248510 5024 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"768ab52b18b7163915519395a96a997f4674b97129580e8bf9afa2628bae7081"} err="failed to get container status \"768ab52b18b7163915519395a96a997f4674b97129580e8bf9afa2628bae7081\": rpc error: code = NotFound desc = could not find container \"768ab52b18b7163915519395a96a997f4674b97129580e8bf9afa2628bae7081\": container with ID starting with 768ab52b18b7163915519395a96a997f4674b97129580e8bf9afa2628bae7081 not found: ID does not exist" Nov 03 14:01:32.256117 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:01:32.256090 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5bcd7997-h56sp"] Nov 03 14:01:32.256323 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:01:32.256305 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-5bcd7997-h56sp"] Nov 03 14:01:32.347519 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:01:32.347420 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16a37f2f-c747-4493-bc05-a78846706028-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-8ggxb\" (UID: \"16a37f2f-c747-4493-bc05-a78846706028\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-8ggxb" Nov 03 14:01:32.347519 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:01:32.347486 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h7t2k\" (UniqueName: \"kubernetes.io/projected/16a37f2f-c747-4493-bc05-a78846706028-kube-api-access-h7t2k\") pod \"maas-api-revoke-sub-c1e271a9-8ggxb\" (UID: \"16a37f2f-c747-4493-bc05-a78846706028\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-8ggxb" Nov 03 14:01:32.347780 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:01:32.347551 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16a37f2f-c747-4493-bc05-a78846706028-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-8ggxb\" (UID: \"16a37f2f-c747-4493-bc05-a78846706028\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-8ggxb" Nov 03 14:01:32.450009 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:01:32.449966 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16a37f2f-c747-4493-bc05-a78846706028-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-8ggxb\" (UID: \"16a37f2f-c747-4493-bc05-a78846706028\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-8ggxb" Nov 03 14:01:32.450631 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:01:32.450605 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h7t2k\" (UniqueName: \"kubernetes.io/projected/16a37f2f-c747-4493-bc05-a78846706028-kube-api-access-h7t2k\") pod \"maas-api-revoke-sub-c1e271a9-8ggxb\" (UID: \"16a37f2f-c747-4493-bc05-a78846706028\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-8ggxb" Nov 03 14:01:32.450788 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:01:32.450770 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16a37f2f-c747-4493-bc05-a78846706028-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-8ggxb\" (UID: \"16a37f2f-c747-4493-bc05-a78846706028\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-8ggxb" Nov 03 14:01:32.454275 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:01:32.453972 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16a37f2f-c747-4493-bc05-a78846706028-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-8ggxb\" (UID: \"16a37f2f-c747-4493-bc05-a78846706028\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-8ggxb" Nov 03 14:01:32.465249 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:01:32.465192 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h7t2k\" (UniqueName: \"kubernetes.io/projected/16a37f2f-c747-4493-bc05-a78846706028-kube-api-access-h7t2k\") pod \"maas-api-revoke-sub-c1e271a9-8ggxb\" (UID: \"16a37f2f-c747-4493-bc05-a78846706028\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-8ggxb" Nov 03 14:01:32.465892 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:01:32.465865 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16a37f2f-c747-4493-bc05-a78846706028-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-8ggxb\" (UID: \"16a37f2f-c747-4493-bc05-a78846706028\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-8ggxb" Nov 03 14:01:32.549780 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:01:32.549464 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-8ggxb" Nov 03 14:01:33.532701 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:01:33.505812 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-8ggxb"] Nov 03 14:01:33.955418 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:01:33.955374 5024 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-vrbqc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 14:01:33.955629 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:01:33.955445 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-vrbqc" podUID="d12a0b88-93e3-4542-90b0-56a37dd91387" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 14:01:34.153306 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:01:34.150430 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="542c62e1-47f9-4425-8e17-c7e2e72bc4ff" path="/var/lib/kubelet/pods/542c62e1-47f9-4425-8e17-c7e2e72bc4ff/volumes" Nov 03 14:01:34.411366 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:01:34.407219 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-8ggxb" event={"ID":"16a37f2f-c747-4493-bc05-a78846706028","Type":"ContainerStarted","Data":"277784f3c9cf82da06d90c1866b0325ca9c956f4335caffe01422b482af63994"} Nov 03 14:01:35.414269 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:01:35.414231 5024 generic.go:356] "Generic (PLEG): container finished" podID="16a37f2f-c747-4493-bc05-a78846706028" containerID="bea008ac902a22ee255a9ea60b1adc49a988f80f58bfd420537048edb25720dc" exitCode=0 Nov 03 14:01:35.414701 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:01:35.414317 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-8ggxb" event={"ID":"16a37f2f-c747-4493-bc05-a78846706028","Type":"ContainerDied","Data":"bea008ac902a22ee255a9ea60b1adc49a988f80f58bfd420537048edb25720dc"} Nov 03 14:01:36.562450 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:01:36.562423 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-8ggxb" Nov 03 14:01:36.566369 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:01:36.566348 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/16a37f2f-c747-4493-bc05-a78846706028-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16a37f2f-c747-4493-bc05-a78846706028-maas-api-xks-ca\") pod \"16a37f2f-c747-4493-bc05-a78846706028\" (UID: \"16a37f2f-c747-4493-bc05-a78846706028\") " Nov 03 14:01:36.566452 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:01:36.566392 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/16a37f2f-c747-4493-bc05-a78846706028-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16a37f2f-c747-4493-bc05-a78846706028-maas-api-openshift-ca\") pod \"16a37f2f-c747-4493-bc05-a78846706028\" (UID: \"16a37f2f-c747-4493-bc05-a78846706028\") " Nov 03 14:01:36.566492 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:01:36.566482 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16a37f2f-c747-4493-bc05-a78846706028-kube-api-access-h7t2k\" (UniqueName: \"kubernetes.io/projected/16a37f2f-c747-4493-bc05-a78846706028-kube-api-access-h7t2k\") pod \"16a37f2f-c747-4493-bc05-a78846706028\" (UID: \"16a37f2f-c747-4493-bc05-a78846706028\") " Nov 03 14:01:36.566758 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:01:36.566733 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16a37f2f-c747-4493-bc05-a78846706028-maas-api-openshift-ca" pod "16a37f2f-c747-4493-bc05-a78846706028" (UID: "16a37f2f-c747-4493-bc05-a78846706028"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:01:36.566896 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:01:36.566821 5024 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16a37f2f-c747-4493-bc05-a78846706028-maas-api-openshift-ca\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 14:01:36.568520 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:01:36.568492 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16a37f2f-c747-4493-bc05-a78846706028-kube-api-access-h7t2k" pod "16a37f2f-c747-4493-bc05-a78846706028" (UID: "16a37f2f-c747-4493-bc05-a78846706028"). InnerVolumeSpecName "kube-api-access-h7t2k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:01:36.568617 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:01:36.568513 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/16a37f2f-c747-4493-bc05-a78846706028-maas-api-xks-ca" pod "16a37f2f-c747-4493-bc05-a78846706028" (UID: "16a37f2f-c747-4493-bc05-a78846706028"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:01:36.667727 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:01:36.667690 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h7t2k\" (UniqueName: \"kubernetes.io/projected/16a37f2f-c747-4493-bc05-a78846706028-kube-api-access-h7t2k\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 14:01:36.667727 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:01:36.667722 5024 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16a37f2f-c747-4493-bc05-a78846706028-maas-api-xks-ca\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 14:01:37.427333 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:01:37.427274 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-8ggxb" Nov 03 14:01:37.427545 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:01:37.427307 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-8ggxb" event={"ID":"16a37f2f-c747-4493-bc05-a78846706028","Type":"ContainerDied","Data":"277784f3c9cf82da06d90c1866b0325ca9c956f4335caffe01422b482af63994"} Nov 03 14:01:37.427545 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:01:37.427431 5024 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="277784f3c9cf82da06d90c1866b0325ca9c956f4335caffe01422b482af63994" Nov 03 14:01:39.506995 ip-10-0-131-36 kubenswrapper[5024]: E1103 14:01:39.506946 5024 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d7mwz" containerName="extractor" Nov 03 14:01:43.393312 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:01:43.393263 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-8wc9s"] Nov 03 14:01:43.395025 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:01:43.394724 5024 cpu_manager.go:414] "containerMap: removing container" podUID="16a37f2f-c747-4493-bc05-a78846706028" containerName="revoke-keys" Nov 03 14:01:43.395215 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:01:43.395195 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16a37f2f-c747-4493-bc05-a78846706028" containerName="revoke-keys" Nov 03 14:01:43.395461 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:01:43.395446 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="16a37f2f-c747-4493-bc05-a78846706028" containerName="revoke-keys" Nov 03 14:01:43.402583 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:01:43.402563 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-8wc9s"] Nov 03 14:01:43.402762 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:01:43.402714 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-8wc9s" Nov 03 14:01:43.408930 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:01:43.408907 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 14:01:43.409428 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:01:43.409409 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w6vqv\"" Nov 03 14:01:43.425153 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:01:43.425088 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b216c9c7-9352-4585-94b5-c6852ac7429d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-8wc9s\" (UID: \"b216c9c7-9352-4585-94b5-c6852ac7429d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-8wc9s" Nov 03 14:01:43.425336 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:01:43.425151 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hktxj\" (UniqueName: \"kubernetes.io/projected/b216c9c7-9352-4585-94b5-c6852ac7429d-kube-api-access-hktxj\") pod \"maas-api-revoke-sub-ec843018-8wc9s\" (UID: \"b216c9c7-9352-4585-94b5-c6852ac7429d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-8wc9s" Nov 03 14:01:43.425336 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:01:43.425210 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b216c9c7-9352-4585-94b5-c6852ac7429d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-8wc9s\" (UID: \"b216c9c7-9352-4585-94b5-c6852ac7429d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-8wc9s" Nov 03 14:01:43.444463 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:01:43.444383 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 14:01:43.444640 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:01:43.444471 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 14:01:43.526344 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:01:43.526277 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b216c9c7-9352-4585-94b5-c6852ac7429d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-8wc9s\" (UID: \"b216c9c7-9352-4585-94b5-c6852ac7429d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-8wc9s" Nov 03 14:01:43.526513 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:01:43.526411 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b216c9c7-9352-4585-94b5-c6852ac7429d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-8wc9s\" (UID: \"b216c9c7-9352-4585-94b5-c6852ac7429d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-8wc9s" Nov 03 14:01:43.526513 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:01:43.526436 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hktxj\" (UniqueName: \"kubernetes.io/projected/b216c9c7-9352-4585-94b5-c6852ac7429d-kube-api-access-hktxj\") pod \"maas-api-revoke-sub-ec843018-8wc9s\" (UID: \"b216c9c7-9352-4585-94b5-c6852ac7429d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-8wc9s" Nov 03 14:01:43.527018 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:01:43.526994 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b216c9c7-9352-4585-94b5-c6852ac7429d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-8wc9s\" (UID: \"b216c9c7-9352-4585-94b5-c6852ac7429d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-8wc9s" Nov 03 14:01:43.528668 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:01:43.528643 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b216c9c7-9352-4585-94b5-c6852ac7429d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-8wc9s\" (UID: \"b216c9c7-9352-4585-94b5-c6852ac7429d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-8wc9s" Nov 03 14:01:43.536540 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:01:43.536498 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hktxj\" (UniqueName: \"kubernetes.io/projected/b216c9c7-9352-4585-94b5-c6852ac7429d-kube-api-access-hktxj\") pod \"maas-api-revoke-sub-ec843018-8wc9s\" (UID: \"b216c9c7-9352-4585-94b5-c6852ac7429d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-8wc9s" Nov 03 14:01:43.739795 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:01:43.739701 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-8wc9s" Nov 03 14:01:44.014680 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:01:44.014652 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-8wc9s"] Nov 03 14:01:44.016568 ip-10-0-131-36 kubenswrapper[5024]: W1103 14:01:44.016536 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb216c9c7_9352_4585_94b5_c6852ac7429d.slice/crio-b8daf39026fb68a960eedea1b096e5799b52d477dfa28f9baf57b0c0b89d5d2a WatchSource:0}: Error finding container b8daf39026fb68a960eedea1b096e5799b52d477dfa28f9baf57b0c0b89d5d2a: Status 404 returned error can't find the container with id b8daf39026fb68a960eedea1b096e5799b52d477dfa28f9baf57b0c0b89d5d2a Nov 03 14:01:44.462978 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:01:44.462930 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-8wc9s" event={"ID":"b216c9c7-9352-4585-94b5-c6852ac7429d","Type":"ContainerStarted","Data":"2da0b595bad46a280bd5c933ce4672f3ad648f65a249c180abc480e83f43289e"} Nov 03 14:01:44.463486 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:01:44.462998 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-8wc9s" event={"ID":"b216c9c7-9352-4585-94b5-c6852ac7429d","Type":"ContainerStarted","Data":"b8daf39026fb68a960eedea1b096e5799b52d477dfa28f9baf57b0c0b89d5d2a"} Nov 03 14:01:44.486757 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:01:44.486698 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-8wc9s" podStartSLOduration=1.48667861 podStartE2EDuration="1.48667861s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:01:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:01:44.482722338 +0000 UTC m=+3184.490913615" watchObservedRunningTime="2026-11-03 14:01:44.48667861 +0000 UTC m=+3184.494869894" Nov 03 14:01:45.473870 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:01:45.473452 5024 generic.go:356] "Generic (PLEG): container finished" podID="b216c9c7-9352-4585-94b5-c6852ac7429d" containerID="2da0b595bad46a280bd5c933ce4672f3ad648f65a249c180abc480e83f43289e" exitCode=0 Nov 03 14:01:45.474442 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:01:45.473526 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-8wc9s" event={"ID":"b216c9c7-9352-4585-94b5-c6852ac7429d","Type":"ContainerDied","Data":"2da0b595bad46a280bd5c933ce4672f3ad648f65a249c180abc480e83f43289e"} Nov 03 14:01:46.282904 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:01:46.282861 5024 scope.go:122] "RemoveContainer" containerID="b67487554ef8f62eff8374c3e2fba92a4f6f06bf233732917661e3989e571f06" Nov 03 14:01:46.319307 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:01:46.315523 5024 scope.go:122] "RemoveContainer" containerID="7f259cd499cb40c776274d9bdd4cad3413a65faa7492b4049015b27fb551e0ea" Nov 03 14:01:46.360145 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:01:46.360117 5024 scope.go:122] "RemoveContainer" containerID="feeb92acc04af50276d3081cea3f29cb25e9911fc682a9b78cec8021952e4947" Nov 03 14:01:47.376027 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:01:47.375741 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-8wc9s" Nov 03 14:01:47.378881 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:01:47.378847 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b216c9c7-9352-4585-94b5-c6852ac7429d-kube-api-access-hktxj\" (UniqueName: \"kubernetes.io/projected/b216c9c7-9352-4585-94b5-c6852ac7429d-kube-api-access-hktxj\") pod \"b216c9c7-9352-4585-94b5-c6852ac7429d\" (UID: \"b216c9c7-9352-4585-94b5-c6852ac7429d\") " Nov 03 14:01:47.379477 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:01:47.379452 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b216c9c7-9352-4585-94b5-c6852ac7429d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b216c9c7-9352-4585-94b5-c6852ac7429d-maas-api-xks-ca\") pod \"b216c9c7-9352-4585-94b5-c6852ac7429d\" (UID: \"b216c9c7-9352-4585-94b5-c6852ac7429d\") " Nov 03 14:01:47.379686 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:01:47.379666 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b216c9c7-9352-4585-94b5-c6852ac7429d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b216c9c7-9352-4585-94b5-c6852ac7429d-maas-api-openshift-ca\") pod \"b216c9c7-9352-4585-94b5-c6852ac7429d\" (UID: \"b216c9c7-9352-4585-94b5-c6852ac7429d\") " Nov 03 14:01:47.386686 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:01:47.385761 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b216c9c7-9352-4585-94b5-c6852ac7429d-maas-api-openshift-ca" pod "b216c9c7-9352-4585-94b5-c6852ac7429d" (UID: "b216c9c7-9352-4585-94b5-c6852ac7429d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:01:47.428782 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:01:47.428725 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b216c9c7-9352-4585-94b5-c6852ac7429d-kube-api-access-hktxj" pod "b216c9c7-9352-4585-94b5-c6852ac7429d" (UID: "b216c9c7-9352-4585-94b5-c6852ac7429d"). InnerVolumeSpecName "kube-api-access-hktxj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:01:47.430484 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:01:47.430454 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b216c9c7-9352-4585-94b5-c6852ac7429d-maas-api-xks-ca" pod "b216c9c7-9352-4585-94b5-c6852ac7429d" (UID: "b216c9c7-9352-4585-94b5-c6852ac7429d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:01:47.508366 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:01:47.504856 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hktxj\" (UniqueName: \"kubernetes.io/projected/b216c9c7-9352-4585-94b5-c6852ac7429d-kube-api-access-hktxj\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 14:01:47.508630 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:01:47.508604 5024 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b216c9c7-9352-4585-94b5-c6852ac7429d-maas-api-xks-ca\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 14:01:47.508758 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:01:47.508743 5024 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b216c9c7-9352-4585-94b5-c6852ac7429d-maas-api-openshift-ca\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 14:01:47.767365 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:01:47.743269 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-8wc9s" event={"ID":"b216c9c7-9352-4585-94b5-c6852ac7429d","Type":"ContainerDied","Data":"b8daf39026fb68a960eedea1b096e5799b52d477dfa28f9baf57b0c0b89d5d2a"} Nov 03 14:01:47.767365 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:01:47.743354 5024 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b8daf39026fb68a960eedea1b096e5799b52d477dfa28f9baf57b0c0b89d5d2a" Nov 03 14:01:47.767365 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:01:47.743432 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-8wc9s" Nov 03 14:02:09.506691 ip-10-0-131-36 kubenswrapper[5024]: E1103 14:02:09.506649 5024 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d7mwz" containerName="extractor" Nov 03 14:02:10.027232 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:02:10.027197 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-dkcsb"] Nov 03 14:02:10.027677 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:02:10.027662 5024 cpu_manager.go:414] "containerMap: removing container" podUID="b216c9c7-9352-4585-94b5-c6852ac7429d" containerName="revoke-keys" Nov 03 14:02:10.027736 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:02:10.027678 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b216c9c7-9352-4585-94b5-c6852ac7429d" containerName="revoke-keys" Nov 03 14:02:10.027775 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:02:10.027751 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="b216c9c7-9352-4585-94b5-c6852ac7429d" containerName="revoke-keys" Nov 03 14:02:10.030878 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:02:10.030858 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-dkcsb" Nov 03 14:02:10.037329 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:02:10.037263 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 14:02:10.039260 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:02:10.039235 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 14:02:10.039590 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:02:10.039572 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 14:02:10.039961 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:02:10.039695 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w6vqv\"" Nov 03 14:02:10.041660 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:02:10.041637 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-dkcsb"] Nov 03 14:02:10.104640 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:02:10.104600 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58z57\" (UniqueName: \"kubernetes.io/projected/258ec688-8b0f-4fcf-992c-f539b9532ee5-kube-api-access-58z57\") pod \"maas-api-revoke-sub-046e1343-dkcsb\" (UID: \"258ec688-8b0f-4fcf-992c-f539b9532ee5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-dkcsb" Nov 03 14:02:10.104836 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:02:10.104807 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/258ec688-8b0f-4fcf-992c-f539b9532ee5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-dkcsb\" (UID: \"258ec688-8b0f-4fcf-992c-f539b9532ee5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-dkcsb" Nov 03 14:02:10.104929 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:02:10.104914 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/258ec688-8b0f-4fcf-992c-f539b9532ee5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-dkcsb\" (UID: \"258ec688-8b0f-4fcf-992c-f539b9532ee5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-dkcsb" Nov 03 14:02:10.206549 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:02:10.206495 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/258ec688-8b0f-4fcf-992c-f539b9532ee5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-dkcsb\" (UID: \"258ec688-8b0f-4fcf-992c-f539b9532ee5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-dkcsb" Nov 03 14:02:10.206745 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:02:10.206568 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/258ec688-8b0f-4fcf-992c-f539b9532ee5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-dkcsb\" (UID: \"258ec688-8b0f-4fcf-992c-f539b9532ee5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-dkcsb" Nov 03 14:02:10.206745 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:02:10.206615 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58z57\" (UniqueName: \"kubernetes.io/projected/258ec688-8b0f-4fcf-992c-f539b9532ee5-kube-api-access-58z57\") pod \"maas-api-revoke-sub-046e1343-dkcsb\" (UID: \"258ec688-8b0f-4fcf-992c-f539b9532ee5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-dkcsb" Nov 03 14:02:10.207196 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:02:10.207172 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/258ec688-8b0f-4fcf-992c-f539b9532ee5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-dkcsb\" (UID: \"258ec688-8b0f-4fcf-992c-f539b9532ee5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-dkcsb" Nov 03 14:02:10.208890 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:02:10.208869 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/258ec688-8b0f-4fcf-992c-f539b9532ee5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-dkcsb\" (UID: \"258ec688-8b0f-4fcf-992c-f539b9532ee5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-dkcsb" Nov 03 14:02:10.216454 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:02:10.216430 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-58z57\" (UniqueName: \"kubernetes.io/projected/258ec688-8b0f-4fcf-992c-f539b9532ee5-kube-api-access-58z57\") pod \"maas-api-revoke-sub-046e1343-dkcsb\" (UID: \"258ec688-8b0f-4fcf-992c-f539b9532ee5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-dkcsb" Nov 03 14:02:10.346597 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:02:10.346502 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-dkcsb" Nov 03 14:02:10.625769 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:02:10.625743 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-dkcsb"] Nov 03 14:02:10.627774 ip-10-0-131-36 kubenswrapper[5024]: W1103 14:02:10.627741 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod258ec688_8b0f_4fcf_992c_f539b9532ee5.slice/crio-ee210b148c21b0a976b66c8b1bd56f2b987cb596f685dfffbadfd34392682298 WatchSource:0}: Error finding container ee210b148c21b0a976b66c8b1bd56f2b987cb596f685dfffbadfd34392682298: Status 404 returned error can't find the container with id ee210b148c21b0a976b66c8b1bd56f2b987cb596f685dfffbadfd34392682298 Nov 03 14:02:10.885746 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:02:10.885713 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-dkcsb" event={"ID":"258ec688-8b0f-4fcf-992c-f539b9532ee5","Type":"ContainerStarted","Data":"ee210b148c21b0a976b66c8b1bd56f2b987cb596f685dfffbadfd34392682298"} Nov 03 14:02:11.897335 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:02:11.897274 5024 generic.go:356] "Generic (PLEG): container finished" podID="258ec688-8b0f-4fcf-992c-f539b9532ee5" containerID="7f17286da1c3782e96f75a1a79312d92b9302a703355e2c632e7c54fdc91bf4f" exitCode=0 Nov 03 14:02:11.897335 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:02:11.897319 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-dkcsb" event={"ID":"258ec688-8b0f-4fcf-992c-f539b9532ee5","Type":"ContainerDied","Data":"7f17286da1c3782e96f75a1a79312d92b9302a703355e2c632e7c54fdc91bf4f"} Nov 03 14:02:13.042870 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:02:13.042840 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-dkcsb" Nov 03 14:02:13.136009 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:02:13.135970 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/258ec688-8b0f-4fcf-992c-f539b9532ee5-kube-api-access-58z57\" (UniqueName: \"kubernetes.io/projected/258ec688-8b0f-4fcf-992c-f539b9532ee5-kube-api-access-58z57\") pod \"258ec688-8b0f-4fcf-992c-f539b9532ee5\" (UID: \"258ec688-8b0f-4fcf-992c-f539b9532ee5\") " Nov 03 14:02:13.136193 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:02:13.136102 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/258ec688-8b0f-4fcf-992c-f539b9532ee5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/258ec688-8b0f-4fcf-992c-f539b9532ee5-maas-api-xks-ca\") pod \"258ec688-8b0f-4fcf-992c-f539b9532ee5\" (UID: \"258ec688-8b0f-4fcf-992c-f539b9532ee5\") " Nov 03 14:02:13.136193 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:02:13.136157 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/258ec688-8b0f-4fcf-992c-f539b9532ee5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/258ec688-8b0f-4fcf-992c-f539b9532ee5-maas-api-openshift-ca\") pod \"258ec688-8b0f-4fcf-992c-f539b9532ee5\" (UID: \"258ec688-8b0f-4fcf-992c-f539b9532ee5\") " Nov 03 14:02:13.136563 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:02:13.136538 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/258ec688-8b0f-4fcf-992c-f539b9532ee5-maas-api-openshift-ca" pod "258ec688-8b0f-4fcf-992c-f539b9532ee5" (UID: "258ec688-8b0f-4fcf-992c-f539b9532ee5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:02:13.138153 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:02:13.138130 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/258ec688-8b0f-4fcf-992c-f539b9532ee5-maas-api-xks-ca" pod "258ec688-8b0f-4fcf-992c-f539b9532ee5" (UID: "258ec688-8b0f-4fcf-992c-f539b9532ee5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:02:13.138221 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:02:13.138154 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/258ec688-8b0f-4fcf-992c-f539b9532ee5-kube-api-access-58z57" pod "258ec688-8b0f-4fcf-992c-f539b9532ee5" (UID: "258ec688-8b0f-4fcf-992c-f539b9532ee5"). InnerVolumeSpecName "kube-api-access-58z57". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:02:13.237594 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:02:13.237504 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-58z57\" (UniqueName: \"kubernetes.io/projected/258ec688-8b0f-4fcf-992c-f539b9532ee5-kube-api-access-58z57\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 14:02:13.237594 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:02:13.237539 5024 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/258ec688-8b0f-4fcf-992c-f539b9532ee5-maas-api-xks-ca\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 14:02:13.237594 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:02:13.237549 5024 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/258ec688-8b0f-4fcf-992c-f539b9532ee5-maas-api-openshift-ca\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 14:02:13.909579 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:02:13.909549 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-dkcsb" Nov 03 14:02:13.909757 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:02:13.909548 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-dkcsb" event={"ID":"258ec688-8b0f-4fcf-992c-f539b9532ee5","Type":"ContainerDied","Data":"ee210b148c21b0a976b66c8b1bd56f2b987cb596f685dfffbadfd34392682298"} Nov 03 14:02:13.909757 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:02:13.909696 5024 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ee210b148c21b0a976b66c8b1bd56f2b987cb596f685dfffbadfd34392682298" Nov 03 14:02:20.241805 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:02:20.241768 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-5g7xc"] Nov 03 14:02:20.242469 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:02:20.242447 5024 cpu_manager.go:414] "containerMap: removing container" podUID="258ec688-8b0f-4fcf-992c-f539b9532ee5" containerName="revoke-keys" Nov 03 14:02:20.242571 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:02:20.242469 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="258ec688-8b0f-4fcf-992c-f539b9532ee5" containerName="revoke-keys" Nov 03 14:02:20.242630 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:02:20.242591 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="258ec688-8b0f-4fcf-992c-f539b9532ee5" containerName="revoke-keys" Nov 03 14:02:20.245749 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:02:20.245726 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-5g7xc" Nov 03 14:02:20.257768 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:02:20.257738 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 14:02:20.257906 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:02:20.257834 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 14:02:20.257966 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:02:20.257925 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 14:02:20.258860 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:02:20.258821 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w6vqv\"" Nov 03 14:02:20.260137 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:02:20.260115 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-5g7xc"] Nov 03 14:02:20.412693 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:02:20.412652 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xst7n\" (UniqueName: \"kubernetes.io/projected/9dcdd272-6b65-4e42-8eb0-a072d3d60992-kube-api-access-xst7n\") pod \"maas-api-revoke-sub-d3fba056-5g7xc\" (UID: \"9dcdd272-6b65-4e42-8eb0-a072d3d60992\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-5g7xc" Nov 03 14:02:20.412896 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:02:20.412703 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9dcdd272-6b65-4e42-8eb0-a072d3d60992-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-5g7xc\" (UID: \"9dcdd272-6b65-4e42-8eb0-a072d3d60992\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-5g7xc" Nov 03 14:02:20.412959 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:02:20.412932 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9dcdd272-6b65-4e42-8eb0-a072d3d60992-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-5g7xc\" (UID: \"9dcdd272-6b65-4e42-8eb0-a072d3d60992\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-5g7xc" Nov 03 14:02:20.513608 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:02:20.513529 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xst7n\" (UniqueName: \"kubernetes.io/projected/9dcdd272-6b65-4e42-8eb0-a072d3d60992-kube-api-access-xst7n\") pod \"maas-api-revoke-sub-d3fba056-5g7xc\" (UID: \"9dcdd272-6b65-4e42-8eb0-a072d3d60992\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-5g7xc" Nov 03 14:02:20.513608 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:02:20.513565 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9dcdd272-6b65-4e42-8eb0-a072d3d60992-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-5g7xc\" (UID: \"9dcdd272-6b65-4e42-8eb0-a072d3d60992\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-5g7xc" Nov 03 14:02:20.513794 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:02:20.513624 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9dcdd272-6b65-4e42-8eb0-a072d3d60992-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-5g7xc\" (UID: \"9dcdd272-6b65-4e42-8eb0-a072d3d60992\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-5g7xc" Nov 03 14:02:20.514246 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:02:20.514227 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9dcdd272-6b65-4e42-8eb0-a072d3d60992-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-5g7xc\" (UID: \"9dcdd272-6b65-4e42-8eb0-a072d3d60992\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-5g7xc" Nov 03 14:02:20.516146 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:02:20.516122 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9dcdd272-6b65-4e42-8eb0-a072d3d60992-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-5g7xc\" (UID: \"9dcdd272-6b65-4e42-8eb0-a072d3d60992\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-5g7xc" Nov 03 14:02:20.524215 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:02:20.524192 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xst7n\" (UniqueName: \"kubernetes.io/projected/9dcdd272-6b65-4e42-8eb0-a072d3d60992-kube-api-access-xst7n\") pod \"maas-api-revoke-sub-d3fba056-5g7xc\" (UID: \"9dcdd272-6b65-4e42-8eb0-a072d3d60992\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-5g7xc" Nov 03 14:02:20.564417 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:02:20.564385 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-5g7xc" Nov 03 14:02:20.846330 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:02:20.846275 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-5g7xc"] Nov 03 14:02:20.848185 ip-10-0-131-36 kubenswrapper[5024]: W1103 14:02:20.848152 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9dcdd272_6b65_4e42_8eb0_a072d3d60992.slice/crio-6ff4301ad2ce20d007912b402f238a67f2f04258a98e8c04c9681a9155c7b901 WatchSource:0}: Error finding container 6ff4301ad2ce20d007912b402f238a67f2f04258a98e8c04c9681a9155c7b901: Status 404 returned error can't find the container with id 6ff4301ad2ce20d007912b402f238a67f2f04258a98e8c04c9681a9155c7b901 Nov 03 14:02:20.942268 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:02:20.942234 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-5g7xc" event={"ID":"9dcdd272-6b65-4e42-8eb0-a072d3d60992","Type":"ContainerStarted","Data":"6ff4301ad2ce20d007912b402f238a67f2f04258a98e8c04c9681a9155c7b901"} Nov 03 14:02:21.952011 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:02:21.951964 5024 generic.go:356] "Generic (PLEG): container finished" podID="9dcdd272-6b65-4e42-8eb0-a072d3d60992" containerID="ed19b009b9dba53c066928dcec16837b50544670cc8a26c7c0e64b97c600e902" exitCode=0 Nov 03 14:02:21.952528 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:02:21.952053 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-5g7xc" event={"ID":"9dcdd272-6b65-4e42-8eb0-a072d3d60992","Type":"ContainerDied","Data":"ed19b009b9dba53c066928dcec16837b50544670cc8a26c7c0e64b97c600e902"} Nov 03 14:02:23.796306 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:02:23.795842 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-5g7xc" Nov 03 14:02:23.941313 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:02:23.934699 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9dcdd272-6b65-4e42-8eb0-a072d3d60992-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9dcdd272-6b65-4e42-8eb0-a072d3d60992-maas-api-xks-ca\") pod \"9dcdd272-6b65-4e42-8eb0-a072d3d60992\" (UID: \"9dcdd272-6b65-4e42-8eb0-a072d3d60992\") " Nov 03 14:02:23.952448 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:02:23.952413 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9dcdd272-6b65-4e42-8eb0-a072d3d60992-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9dcdd272-6b65-4e42-8eb0-a072d3d60992-maas-api-openshift-ca\") pod \"9dcdd272-6b65-4e42-8eb0-a072d3d60992\" (UID: \"9dcdd272-6b65-4e42-8eb0-a072d3d60992\") " Nov 03 14:02:23.964682 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:02:23.964609 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9dcdd272-6b65-4e42-8eb0-a072d3d60992-kube-api-access-xst7n\" (UniqueName: \"kubernetes.io/projected/9dcdd272-6b65-4e42-8eb0-a072d3d60992-kube-api-access-xst7n\") pod \"9dcdd272-6b65-4e42-8eb0-a072d3d60992\" (UID: \"9dcdd272-6b65-4e42-8eb0-a072d3d60992\") " Nov 03 14:02:24.013746 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:02:23.964518 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9dcdd272-6b65-4e42-8eb0-a072d3d60992-maas-api-openshift-ca" pod "9dcdd272-6b65-4e42-8eb0-a072d3d60992" (UID: "9dcdd272-6b65-4e42-8eb0-a072d3d60992"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:02:24.015056 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:02:24.015023 5024 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9dcdd272-6b65-4e42-8eb0-a072d3d60992-maas-api-openshift-ca\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 14:02:24.063415 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:02:24.039717 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9dcdd272-6b65-4e42-8eb0-a072d3d60992-kube-api-access-xst7n" pod "9dcdd272-6b65-4e42-8eb0-a072d3d60992" (UID: "9dcdd272-6b65-4e42-8eb0-a072d3d60992"). InnerVolumeSpecName "kube-api-access-xst7n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:02:24.128036 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:02:24.126820 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9dcdd272-6b65-4e42-8eb0-a072d3d60992-maas-api-xks-ca" pod "9dcdd272-6b65-4e42-8eb0-a072d3d60992" (UID: "9dcdd272-6b65-4e42-8eb0-a072d3d60992"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:02:24.128036 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:02:24.127965 5024 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9dcdd272-6b65-4e42-8eb0-a072d3d60992-maas-api-xks-ca\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 14:02:24.128036 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:02:24.127993 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xst7n\" (UniqueName: \"kubernetes.io/projected/9dcdd272-6b65-4e42-8eb0-a072d3d60992-kube-api-access-xst7n\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 14:02:24.162466 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:02:24.162418 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-5g7xc" event={"ID":"9dcdd272-6b65-4e42-8eb0-a072d3d60992","Type":"ContainerDied","Data":"6ff4301ad2ce20d007912b402f238a67f2f04258a98e8c04c9681a9155c7b901"} Nov 03 14:02:24.162725 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:02:24.162702 5024 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6ff4301ad2ce20d007912b402f238a67f2f04258a98e8c04c9681a9155c7b901" Nov 03 14:02:24.182308 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:02:24.162923 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-5g7xc" Nov 03 14:02:39.508338 ip-10-0-131-36 kubenswrapper[5024]: E1103 14:02:39.508266 5024 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d7mwz" containerName="extractor" Nov 03 14:02:41.043091 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:02:41.043047 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-cmhcd"] Nov 03 14:02:41.043672 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:02:41.043482 5024 cpu_manager.go:414] "containerMap: removing container" podUID="9dcdd272-6b65-4e42-8eb0-a072d3d60992" containerName="revoke-keys" Nov 03 14:02:41.043672 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:02:41.043494 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9dcdd272-6b65-4e42-8eb0-a072d3d60992" containerName="revoke-keys" Nov 03 14:02:41.043672 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:02:41.043568 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="9dcdd272-6b65-4e42-8eb0-a072d3d60992" containerName="revoke-keys" Nov 03 14:02:41.046662 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:02:41.046633 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-cmhcd" Nov 03 14:02:41.054646 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:02:41.054001 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 14:02:41.054646 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:02:41.054052 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 14:02:41.054646 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:02:41.054166 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w6vqv\"" Nov 03 14:02:41.058582 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:02:41.058549 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-cmhcd"] Nov 03 14:02:41.058705 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:02:41.058683 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 14:02:41.145136 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:02:41.145093 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/789b5ef9-0394-4398-a707-13dabe0fa3b6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-cmhcd\" (UID: \"789b5ef9-0394-4398-a707-13dabe0fa3b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-cmhcd" Nov 03 14:02:41.145136 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:02:41.145134 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/789b5ef9-0394-4398-a707-13dabe0fa3b6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-cmhcd\" (UID: \"789b5ef9-0394-4398-a707-13dabe0fa3b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-cmhcd" Nov 03 14:02:41.145439 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:02:41.145412 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kw4vc\" (UniqueName: \"kubernetes.io/projected/789b5ef9-0394-4398-a707-13dabe0fa3b6-kube-api-access-kw4vc\") pod \"maas-api-revoke-sub-ae594979-cmhcd\" (UID: \"789b5ef9-0394-4398-a707-13dabe0fa3b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-cmhcd" Nov 03 14:02:41.246060 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:02:41.246022 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/789b5ef9-0394-4398-a707-13dabe0fa3b6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-cmhcd\" (UID: \"789b5ef9-0394-4398-a707-13dabe0fa3b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-cmhcd" Nov 03 14:02:41.246060 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:02:41.246063 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/789b5ef9-0394-4398-a707-13dabe0fa3b6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-cmhcd\" (UID: \"789b5ef9-0394-4398-a707-13dabe0fa3b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-cmhcd" Nov 03 14:02:41.246304 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:02:41.246149 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kw4vc\" (UniqueName: \"kubernetes.io/projected/789b5ef9-0394-4398-a707-13dabe0fa3b6-kube-api-access-kw4vc\") pod \"maas-api-revoke-sub-ae594979-cmhcd\" (UID: \"789b5ef9-0394-4398-a707-13dabe0fa3b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-cmhcd" Nov 03 14:02:41.246781 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:02:41.246743 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/789b5ef9-0394-4398-a707-13dabe0fa3b6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-cmhcd\" (UID: \"789b5ef9-0394-4398-a707-13dabe0fa3b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-cmhcd" Nov 03 14:02:41.248497 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:02:41.248471 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/789b5ef9-0394-4398-a707-13dabe0fa3b6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-cmhcd\" (UID: \"789b5ef9-0394-4398-a707-13dabe0fa3b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-cmhcd" Nov 03 14:02:41.255735 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:02:41.255711 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kw4vc\" (UniqueName: \"kubernetes.io/projected/789b5ef9-0394-4398-a707-13dabe0fa3b6-kube-api-access-kw4vc\") pod \"maas-api-revoke-sub-ae594979-cmhcd\" (UID: \"789b5ef9-0394-4398-a707-13dabe0fa3b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-cmhcd" Nov 03 14:02:41.361721 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:02:41.361694 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-cmhcd" Nov 03 14:02:41.638618 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:02:41.638586 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-cmhcd"] Nov 03 14:02:41.640685 ip-10-0-131-36 kubenswrapper[5024]: W1103 14:02:41.640643 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod789b5ef9_0394_4398_a707_13dabe0fa3b6.slice/crio-2eff4bbc7959123aa2ea2eb297402310808dceb60ca2a085c9203deafbdbf786 WatchSource:0}: Error finding container 2eff4bbc7959123aa2ea2eb297402310808dceb60ca2a085c9203deafbdbf786: Status 404 returned error can't find the container with id 2eff4bbc7959123aa2ea2eb297402310808dceb60ca2a085c9203deafbdbf786 Nov 03 14:02:42.275369 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:02:42.275261 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-cmhcd" event={"ID":"789b5ef9-0394-4398-a707-13dabe0fa3b6","Type":"ContainerStarted","Data":"dd8911be1bdaf8fda5c87d9f6133d109b50174b989d17e2f88f669bd08ca7009"} Nov 03 14:02:42.275369 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:02:42.275347 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-cmhcd" event={"ID":"789b5ef9-0394-4398-a707-13dabe0fa3b6","Type":"ContainerStarted","Data":"2eff4bbc7959123aa2ea2eb297402310808dceb60ca2a085c9203deafbdbf786"} Nov 03 14:02:42.296149 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:02:42.296096 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-cmhcd" podStartSLOduration=1.296082265 podStartE2EDuration="1.296082265s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:02:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:02:42.294497837 +0000 UTC m=+3242.302689113" watchObservedRunningTime="2026-11-03 14:02:42.296082265 +0000 UTC m=+3242.304273543" Nov 03 14:02:43.281280 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:02:43.281243 5024 generic.go:356] "Generic (PLEG): container finished" podID="789b5ef9-0394-4398-a707-13dabe0fa3b6" containerID="dd8911be1bdaf8fda5c87d9f6133d109b50174b989d17e2f88f669bd08ca7009" exitCode=0 Nov 03 14:02:43.281763 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:02:43.281326 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-cmhcd" event={"ID":"789b5ef9-0394-4398-a707-13dabe0fa3b6","Type":"ContainerDied","Data":"dd8911be1bdaf8fda5c87d9f6133d109b50174b989d17e2f88f669bd08ca7009"} Nov 03 14:02:44.431452 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:02:44.431428 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-cmhcd" Nov 03 14:02:44.478279 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:02:44.478245 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/789b5ef9-0394-4398-a707-13dabe0fa3b6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/789b5ef9-0394-4398-a707-13dabe0fa3b6-maas-api-openshift-ca\") pod \"789b5ef9-0394-4398-a707-13dabe0fa3b6\" (UID: \"789b5ef9-0394-4398-a707-13dabe0fa3b6\") " Nov 03 14:02:44.478460 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:02:44.478316 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/789b5ef9-0394-4398-a707-13dabe0fa3b6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/789b5ef9-0394-4398-a707-13dabe0fa3b6-maas-api-xks-ca\") pod \"789b5ef9-0394-4398-a707-13dabe0fa3b6\" (UID: \"789b5ef9-0394-4398-a707-13dabe0fa3b6\") " Nov 03 14:02:44.478460 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:02:44.478436 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/789b5ef9-0394-4398-a707-13dabe0fa3b6-kube-api-access-kw4vc\" (UniqueName: \"kubernetes.io/projected/789b5ef9-0394-4398-a707-13dabe0fa3b6-kube-api-access-kw4vc\") pod \"789b5ef9-0394-4398-a707-13dabe0fa3b6\" (UID: \"789b5ef9-0394-4398-a707-13dabe0fa3b6\") " Nov 03 14:02:44.478718 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:02:44.478691 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/789b5ef9-0394-4398-a707-13dabe0fa3b6-maas-api-openshift-ca" pod "789b5ef9-0394-4398-a707-13dabe0fa3b6" (UID: "789b5ef9-0394-4398-a707-13dabe0fa3b6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:02:44.478877 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:02:44.478857 5024 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/789b5ef9-0394-4398-a707-13dabe0fa3b6-maas-api-openshift-ca\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 14:02:44.480559 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:02:44.480531 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/789b5ef9-0394-4398-a707-13dabe0fa3b6-maas-api-xks-ca" pod "789b5ef9-0394-4398-a707-13dabe0fa3b6" (UID: "789b5ef9-0394-4398-a707-13dabe0fa3b6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:02:44.481080 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:02:44.481055 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/789b5ef9-0394-4398-a707-13dabe0fa3b6-kube-api-access-kw4vc" pod "789b5ef9-0394-4398-a707-13dabe0fa3b6" (UID: "789b5ef9-0394-4398-a707-13dabe0fa3b6"). InnerVolumeSpecName "kube-api-access-kw4vc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:02:44.580223 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:02:44.580152 5024 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/789b5ef9-0394-4398-a707-13dabe0fa3b6-maas-api-xks-ca\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 14:02:44.580223 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:02:44.580178 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kw4vc\" (UniqueName: \"kubernetes.io/projected/789b5ef9-0394-4398-a707-13dabe0fa3b6-kube-api-access-kw4vc\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 14:02:45.291448 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:02:45.291407 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-cmhcd" event={"ID":"789b5ef9-0394-4398-a707-13dabe0fa3b6","Type":"ContainerDied","Data":"2eff4bbc7959123aa2ea2eb297402310808dceb60ca2a085c9203deafbdbf786"} Nov 03 14:02:45.291821 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:02:45.291776 5024 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2eff4bbc7959123aa2ea2eb297402310808dceb60ca2a085c9203deafbdbf786" Nov 03 14:02:45.291821 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:02:45.291800 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-cmhcd" Nov 03 14:02:56.288310 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:02:56.287388 5024 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-92vh9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.173:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 14:02:56.288310 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:02:56.287479 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-92vh9" podUID="825941b6-3ed3-47f2-a859-c891c4f29434" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.173:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 14:02:56.312309 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:02:56.309388 5024 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-zndxf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.172:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 14:02:56.312309 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:02:56.309454 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-zndxf" podUID="73f8f24f-b955-433b-95ed-5f441f7fceb2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.172:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 14:02:56.355421 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:02:56.355375 5024 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-k4c4j container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.174:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 14:02:56.355653 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:02:56.355445 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-k4c4j" podUID="4b70e366-6fba-4bfa-ac3e-0f0c7b0b64d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.174:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 14:02:57.478248 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:02:57.478207 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-4xdw9"] Nov 03 14:02:57.478887 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:02:57.478860 5024 cpu_manager.go:414] "containerMap: removing container" podUID="789b5ef9-0394-4398-a707-13dabe0fa3b6" containerName="revoke-keys" Nov 03 14:02:57.478887 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:02:57.478886 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="789b5ef9-0394-4398-a707-13dabe0fa3b6" containerName="revoke-keys" Nov 03 14:02:57.479063 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:02:57.479030 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="789b5ef9-0394-4398-a707-13dabe0fa3b6" containerName="revoke-keys" Nov 03 14:02:57.482934 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:02:57.482912 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-4xdw9" Nov 03 14:02:57.489176 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:02:57.489104 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-4xdw9"] Nov 03 14:02:57.490577 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:02:57.490529 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 14:02:57.490577 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:02:57.490537 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 14:02:57.490773 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:02:57.490600 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 14:02:57.490773 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:02:57.490603 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w6vqv\"" Nov 03 14:02:57.567334 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:02:57.567294 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0e844e0-9520-47c9-84a0-5bf65c116a54-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-4xdw9\" (UID: \"e0e844e0-9520-47c9-84a0-5bf65c116a54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-4xdw9" Nov 03 14:02:57.567508 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:02:57.567413 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0e844e0-9520-47c9-84a0-5bf65c116a54-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-4xdw9\" (UID: \"e0e844e0-9520-47c9-84a0-5bf65c116a54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-4xdw9" Nov 03 14:02:57.567508 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:02:57.567456 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mw7zr\" (UniqueName: \"kubernetes.io/projected/e0e844e0-9520-47c9-84a0-5bf65c116a54-kube-api-access-mw7zr\") pod \"maas-api-revoke-sub-a4aef7d7-4xdw9\" (UID: \"e0e844e0-9520-47c9-84a0-5bf65c116a54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-4xdw9" Nov 03 14:02:57.668867 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:02:57.668824 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0e844e0-9520-47c9-84a0-5bf65c116a54-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-4xdw9\" (UID: \"e0e844e0-9520-47c9-84a0-5bf65c116a54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-4xdw9" Nov 03 14:02:57.669059 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:02:57.668886 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0e844e0-9520-47c9-84a0-5bf65c116a54-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-4xdw9\" (UID: \"e0e844e0-9520-47c9-84a0-5bf65c116a54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-4xdw9" Nov 03 14:02:57.669059 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:02:57.669041 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mw7zr\" (UniqueName: \"kubernetes.io/projected/e0e844e0-9520-47c9-84a0-5bf65c116a54-kube-api-access-mw7zr\") pod \"maas-api-revoke-sub-a4aef7d7-4xdw9\" (UID: \"e0e844e0-9520-47c9-84a0-5bf65c116a54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-4xdw9" Nov 03 14:02:57.669625 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:02:57.669601 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0e844e0-9520-47c9-84a0-5bf65c116a54-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-4xdw9\" (UID: \"e0e844e0-9520-47c9-84a0-5bf65c116a54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-4xdw9" Nov 03 14:02:57.671244 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:02:57.671225 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0e844e0-9520-47c9-84a0-5bf65c116a54-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-4xdw9\" (UID: \"e0e844e0-9520-47c9-84a0-5bf65c116a54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-4xdw9" Nov 03 14:02:57.678783 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:02:57.678759 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mw7zr\" (UniqueName: \"kubernetes.io/projected/e0e844e0-9520-47c9-84a0-5bf65c116a54-kube-api-access-mw7zr\") pod \"maas-api-revoke-sub-a4aef7d7-4xdw9\" (UID: \"e0e844e0-9520-47c9-84a0-5bf65c116a54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-4xdw9" Nov 03 14:02:57.802176 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:02:57.802086 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-4xdw9" Nov 03 14:02:58.079978 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:02:58.079953 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-4xdw9"] Nov 03 14:02:58.081926 ip-10-0-131-36 kubenswrapper[5024]: W1103 14:02:58.081892 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0e844e0_9520_47c9_84a0_5bf65c116a54.slice/crio-18f69b08e11f83ae948a39ee09a5cea0574716076db433b0681b6df5346384bd WatchSource:0}: Error finding container 18f69b08e11f83ae948a39ee09a5cea0574716076db433b0681b6df5346384bd: Status 404 returned error can't find the container with id 18f69b08e11f83ae948a39ee09a5cea0574716076db433b0681b6df5346384bd Nov 03 14:02:58.370635 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:02:58.370588 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-4xdw9" event={"ID":"e0e844e0-9520-47c9-84a0-5bf65c116a54","Type":"ContainerStarted","Data":"18f69b08e11f83ae948a39ee09a5cea0574716076db433b0681b6df5346384bd"} Nov 03 14:02:59.379617 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:02:59.379574 5024 generic.go:356] "Generic (PLEG): container finished" podID="e0e844e0-9520-47c9-84a0-5bf65c116a54" containerID="bf55f05d0c3d793d5fd614d78363905e3f74990f87abef2d3fe489d5e722d9dc" exitCode=0 Nov 03 14:02:59.380102 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:02:59.379649 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-4xdw9" event={"ID":"e0e844e0-9520-47c9-84a0-5bf65c116a54","Type":"ContainerDied","Data":"bf55f05d0c3d793d5fd614d78363905e3f74990f87abef2d3fe489d5e722d9dc"} Nov 03 14:03:00.527083 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:03:00.527056 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-4xdw9" Nov 03 14:03:00.598115 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:03:00.598075 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e0e844e0-9520-47c9-84a0-5bf65c116a54-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0e844e0-9520-47c9-84a0-5bf65c116a54-maas-api-openshift-ca\") pod \"e0e844e0-9520-47c9-84a0-5bf65c116a54\" (UID: \"e0e844e0-9520-47c9-84a0-5bf65c116a54\") " Nov 03 14:03:00.598349 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:03:00.598196 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e0e844e0-9520-47c9-84a0-5bf65c116a54-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0e844e0-9520-47c9-84a0-5bf65c116a54-maas-api-xks-ca\") pod \"e0e844e0-9520-47c9-84a0-5bf65c116a54\" (UID: \"e0e844e0-9520-47c9-84a0-5bf65c116a54\") " Nov 03 14:03:00.598349 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:03:00.598247 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0e844e0-9520-47c9-84a0-5bf65c116a54-kube-api-access-mw7zr\" (UniqueName: \"kubernetes.io/projected/e0e844e0-9520-47c9-84a0-5bf65c116a54-kube-api-access-mw7zr\") pod \"e0e844e0-9520-47c9-84a0-5bf65c116a54\" (UID: \"e0e844e0-9520-47c9-84a0-5bf65c116a54\") " Nov 03 14:03:00.598569 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:03:00.598541 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0e844e0-9520-47c9-84a0-5bf65c116a54-maas-api-openshift-ca" pod "e0e844e0-9520-47c9-84a0-5bf65c116a54" (UID: "e0e844e0-9520-47c9-84a0-5bf65c116a54"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:03:00.598678 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:03:00.598659 5024 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0e844e0-9520-47c9-84a0-5bf65c116a54-maas-api-openshift-ca\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 14:03:00.600480 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:03:00.600448 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0e844e0-9520-47c9-84a0-5bf65c116a54-maas-api-xks-ca" pod "e0e844e0-9520-47c9-84a0-5bf65c116a54" (UID: "e0e844e0-9520-47c9-84a0-5bf65c116a54"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:03:00.600614 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:03:00.600587 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0e844e0-9520-47c9-84a0-5bf65c116a54-kube-api-access-mw7zr" pod "e0e844e0-9520-47c9-84a0-5bf65c116a54" (UID: "e0e844e0-9520-47c9-84a0-5bf65c116a54"). InnerVolumeSpecName "kube-api-access-mw7zr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:03:00.699642 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:03:00.699554 5024 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0e844e0-9520-47c9-84a0-5bf65c116a54-maas-api-xks-ca\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 14:03:00.699642 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:03:00.699596 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mw7zr\" (UniqueName: \"kubernetes.io/projected/e0e844e0-9520-47c9-84a0-5bf65c116a54-kube-api-access-mw7zr\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 14:03:01.391228 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:03:01.391174 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-4xdw9" event={"ID":"e0e844e0-9520-47c9-84a0-5bf65c116a54","Type":"ContainerDied","Data":"18f69b08e11f83ae948a39ee09a5cea0574716076db433b0681b6df5346384bd"} Nov 03 14:03:01.391443 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:03:01.391249 5024 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="18f69b08e11f83ae948a39ee09a5cea0574716076db433b0681b6df5346384bd" Nov 03 14:03:01.391443 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:03:01.391198 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-4xdw9" Nov 03 14:03:09.507679 ip-10-0-131-36 kubenswrapper[5024]: E1103 14:03:09.507594 5024 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d7mwz" containerName="extractor" Nov 03 14:03:13.159307 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:03:13.156983 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-w9vnh"] Nov 03 14:03:13.159307 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:03:13.157634 5024 cpu_manager.go:414] "containerMap: removing container" podUID="e0e844e0-9520-47c9-84a0-5bf65c116a54" containerName="revoke-keys" Nov 03 14:03:13.159307 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:03:13.157658 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0e844e0-9520-47c9-84a0-5bf65c116a54" containerName="revoke-keys" Nov 03 14:03:13.159307 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:03:13.157812 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0e844e0-9520-47c9-84a0-5bf65c116a54" containerName="revoke-keys" Nov 03 14:03:13.168305 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:03:13.161582 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-w9vnh" Nov 03 14:03:13.173184 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:03:13.173080 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 14:03:13.202819 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:03:13.202764 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w6vqv\"" Nov 03 14:03:13.214308 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:03:13.211204 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-w9vnh"] Nov 03 14:03:13.272519 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:03:13.272351 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 14:03:13.272519 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:03:13.272485 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 14:03:13.280784 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:03:13.280746 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a1deed4-a72f-4ac9-9911-9213c507b151-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-w9vnh\" (UID: \"9a1deed4-a72f-4ac9-9911-9213c507b151\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-w9vnh" Nov 03 14:03:13.281083 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:03:13.281062 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nrzk2\" (UniqueName: \"kubernetes.io/projected/9a1deed4-a72f-4ac9-9911-9213c507b151-kube-api-access-nrzk2\") pod \"maas-api-revoke-sub-79805ca9-w9vnh\" (UID: \"9a1deed4-a72f-4ac9-9911-9213c507b151\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-w9vnh" Nov 03 14:03:13.281299 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:03:13.281260 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a1deed4-a72f-4ac9-9911-9213c507b151-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-w9vnh\" (UID: \"9a1deed4-a72f-4ac9-9911-9213c507b151\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-w9vnh" Nov 03 14:03:13.396316 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:03:13.387709 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a1deed4-a72f-4ac9-9911-9213c507b151-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-w9vnh\" (UID: \"9a1deed4-a72f-4ac9-9911-9213c507b151\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-w9vnh" Nov 03 14:03:13.396316 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:03:13.387834 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrzk2\" (UniqueName: \"kubernetes.io/projected/9a1deed4-a72f-4ac9-9911-9213c507b151-kube-api-access-nrzk2\") pod \"maas-api-revoke-sub-79805ca9-w9vnh\" (UID: \"9a1deed4-a72f-4ac9-9911-9213c507b151\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-w9vnh" Nov 03 14:03:13.396316 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:03:13.387916 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a1deed4-a72f-4ac9-9911-9213c507b151-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-w9vnh\" (UID: \"9a1deed4-a72f-4ac9-9911-9213c507b151\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-w9vnh" Nov 03 14:03:13.424962 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:03:13.424874 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a1deed4-a72f-4ac9-9911-9213c507b151-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-w9vnh\" (UID: \"9a1deed4-a72f-4ac9-9911-9213c507b151\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-w9vnh" Nov 03 14:03:13.445754 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:03:13.445698 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a1deed4-a72f-4ac9-9911-9213c507b151-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-w9vnh\" (UID: \"9a1deed4-a72f-4ac9-9911-9213c507b151\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-w9vnh" Nov 03 14:03:13.493872 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:03:13.493062 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nrzk2\" (UniqueName: \"kubernetes.io/projected/9a1deed4-a72f-4ac9-9911-9213c507b151-kube-api-access-nrzk2\") pod \"maas-api-revoke-sub-79805ca9-w9vnh\" (UID: \"9a1deed4-a72f-4ac9-9911-9213c507b151\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-w9vnh" Nov 03 14:03:13.505635 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:03:13.505599 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-w9vnh" Nov 03 14:03:14.087924 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:03:14.087890 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-w9vnh"] Nov 03 14:03:14.089635 ip-10-0-131-36 kubenswrapper[5024]: W1103 14:03:14.089591 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a1deed4_a72f_4ac9_9911_9213c507b151.slice/crio-cbfee1de48bbb96f0afe4e820e9c7e4d88f188bd1ace02fa51e820d74c71b5d9 WatchSource:0}: Error finding container cbfee1de48bbb96f0afe4e820e9c7e4d88f188bd1ace02fa51e820d74c71b5d9: Status 404 returned error can't find the container with id cbfee1de48bbb96f0afe4e820e9c7e4d88f188bd1ace02fa51e820d74c71b5d9 Nov 03 14:03:14.504458 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:03:14.504370 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-w9vnh" event={"ID":"9a1deed4-a72f-4ac9-9911-9213c507b151","Type":"ContainerStarted","Data":"9bfc2f749f207ae75018b48ad3757e6091aae4bd3c40632dbf1ef93929be9748"} Nov 03 14:03:14.504458 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:03:14.504421 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-w9vnh" event={"ID":"9a1deed4-a72f-4ac9-9911-9213c507b151","Type":"ContainerStarted","Data":"cbfee1de48bbb96f0afe4e820e9c7e4d88f188bd1ace02fa51e820d74c71b5d9"} Nov 03 14:03:14.525187 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:03:14.525136 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-w9vnh" podStartSLOduration=1.525123067 podStartE2EDuration="1.525123067s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:03:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:03:14.522614322 +0000 UTC m=+3274.530805600" watchObservedRunningTime="2026-11-03 14:03:14.525123067 +0000 UTC m=+3274.533314371" Nov 03 14:03:15.511900 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:03:15.511857 5024 generic.go:356] "Generic (PLEG): container finished" podID="9a1deed4-a72f-4ac9-9911-9213c507b151" containerID="9bfc2f749f207ae75018b48ad3757e6091aae4bd3c40632dbf1ef93929be9748" exitCode=0 Nov 03 14:03:15.512331 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:03:15.511904 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-w9vnh" event={"ID":"9a1deed4-a72f-4ac9-9911-9213c507b151","Type":"ContainerDied","Data":"9bfc2f749f207ae75018b48ad3757e6091aae4bd3c40632dbf1ef93929be9748"} Nov 03 14:03:16.659515 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:03:16.659491 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-w9vnh" Nov 03 14:03:16.785008 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:03:16.784928 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a1deed4-a72f-4ac9-9911-9213c507b151-kube-api-access-nrzk2\" (UniqueName: \"kubernetes.io/projected/9a1deed4-a72f-4ac9-9911-9213c507b151-kube-api-access-nrzk2\") pod \"9a1deed4-a72f-4ac9-9911-9213c507b151\" (UID: \"9a1deed4-a72f-4ac9-9911-9213c507b151\") " Nov 03 14:03:16.785008 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:03:16.784977 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9a1deed4-a72f-4ac9-9911-9213c507b151-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a1deed4-a72f-4ac9-9911-9213c507b151-maas-api-xks-ca\") pod \"9a1deed4-a72f-4ac9-9911-9213c507b151\" (UID: \"9a1deed4-a72f-4ac9-9911-9213c507b151\") " Nov 03 14:03:16.785217 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:03:16.785102 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a1deed4-a72f-4ac9-9911-9213c507b151-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a1deed4-a72f-4ac9-9911-9213c507b151-maas-api-openshift-ca\") pod \"9a1deed4-a72f-4ac9-9911-9213c507b151\" (UID: \"9a1deed4-a72f-4ac9-9911-9213c507b151\") " Nov 03 14:03:16.785494 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:03:16.785467 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a1deed4-a72f-4ac9-9911-9213c507b151-maas-api-openshift-ca" pod "9a1deed4-a72f-4ac9-9911-9213c507b151" (UID: "9a1deed4-a72f-4ac9-9911-9213c507b151"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:03:16.787082 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:03:16.787057 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a1deed4-a72f-4ac9-9911-9213c507b151-kube-api-access-nrzk2" pod "9a1deed4-a72f-4ac9-9911-9213c507b151" (UID: "9a1deed4-a72f-4ac9-9911-9213c507b151"). InnerVolumeSpecName "kube-api-access-nrzk2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:03:16.787082 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:03:16.787064 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9a1deed4-a72f-4ac9-9911-9213c507b151-maas-api-xks-ca" pod "9a1deed4-a72f-4ac9-9911-9213c507b151" (UID: "9a1deed4-a72f-4ac9-9911-9213c507b151"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:03:16.886762 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:03:16.886735 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nrzk2\" (UniqueName: \"kubernetes.io/projected/9a1deed4-a72f-4ac9-9911-9213c507b151-kube-api-access-nrzk2\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 14:03:16.886762 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:03:16.886761 5024 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a1deed4-a72f-4ac9-9911-9213c507b151-maas-api-xks-ca\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 14:03:16.886887 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:03:16.886773 5024 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a1deed4-a72f-4ac9-9911-9213c507b151-maas-api-openshift-ca\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 14:03:17.523217 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:03:17.523180 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-w9vnh" event={"ID":"9a1deed4-a72f-4ac9-9911-9213c507b151","Type":"ContainerDied","Data":"cbfee1de48bbb96f0afe4e820e9c7e4d88f188bd1ace02fa51e820d74c71b5d9"} Nov 03 14:03:17.523217 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:03:17.523203 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-w9vnh" Nov 03 14:03:17.523473 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:03:17.523230 5024 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cbfee1de48bbb96f0afe4e820e9c7e4d88f188bd1ace02fa51e820d74c71b5d9" Nov 03 14:03:27.857312 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:03:27.852145 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-s79rx"] Nov 03 14:03:27.857312 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:03:27.853835 5024 cpu_manager.go:414] "containerMap: removing container" podUID="9a1deed4-a72f-4ac9-9911-9213c507b151" containerName="revoke-keys" Nov 03 14:03:27.857312 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:03:27.853857 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a1deed4-a72f-4ac9-9911-9213c507b151" containerName="revoke-keys" Nov 03 14:03:27.857312 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:03:27.853979 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a1deed4-a72f-4ac9-9911-9213c507b151" containerName="revoke-keys" Nov 03 14:03:27.925342 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:03:27.925298 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-s79rx"] Nov 03 14:03:27.935410 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:03:27.926422 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-s79rx" Nov 03 14:03:27.942695 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:03:27.942663 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 14:03:27.948317 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:03:27.943009 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 14:03:27.975306 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:03:27.949210 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w6vqv\"" Nov 03 14:03:27.976149 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:03:27.962426 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 14:03:28.145367 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:03:28.136182 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99703a90-80bd-4e1f-87f6-60541fa89cb2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-s79rx\" (UID: \"99703a90-80bd-4e1f-87f6-60541fa89cb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-s79rx" Nov 03 14:03:28.145855 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:03:28.145819 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99703a90-80bd-4e1f-87f6-60541fa89cb2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-s79rx\" (UID: \"99703a90-80bd-4e1f-87f6-60541fa89cb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-s79rx" Nov 03 14:03:28.152312 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:03:28.146506 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7kcqn\" (UniqueName: \"kubernetes.io/projected/99703a90-80bd-4e1f-87f6-60541fa89cb2-kube-api-access-7kcqn\") pod \"maas-api-revoke-sub-0b7e8edb-s79rx\" (UID: \"99703a90-80bd-4e1f-87f6-60541fa89cb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-s79rx" Nov 03 14:03:28.261307 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:03:28.260851 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7kcqn\" (UniqueName: \"kubernetes.io/projected/99703a90-80bd-4e1f-87f6-60541fa89cb2-kube-api-access-7kcqn\") pod \"maas-api-revoke-sub-0b7e8edb-s79rx\" (UID: \"99703a90-80bd-4e1f-87f6-60541fa89cb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-s79rx" Nov 03 14:03:28.261307 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:03:28.260954 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99703a90-80bd-4e1f-87f6-60541fa89cb2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-s79rx\" (UID: \"99703a90-80bd-4e1f-87f6-60541fa89cb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-s79rx" Nov 03 14:03:28.261307 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:03:28.261079 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99703a90-80bd-4e1f-87f6-60541fa89cb2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-s79rx\" (UID: \"99703a90-80bd-4e1f-87f6-60541fa89cb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-s79rx" Nov 03 14:03:28.265305 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:03:28.261910 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99703a90-80bd-4e1f-87f6-60541fa89cb2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-s79rx\" (UID: \"99703a90-80bd-4e1f-87f6-60541fa89cb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-s79rx" Nov 03 14:03:28.341566 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:03:28.341519 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99703a90-80bd-4e1f-87f6-60541fa89cb2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-s79rx\" (UID: \"99703a90-80bd-4e1f-87f6-60541fa89cb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-s79rx" Nov 03 14:03:28.351260 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:03:28.351219 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7kcqn\" (UniqueName: \"kubernetes.io/projected/99703a90-80bd-4e1f-87f6-60541fa89cb2-kube-api-access-7kcqn\") pod \"maas-api-revoke-sub-0b7e8edb-s79rx\" (UID: \"99703a90-80bd-4e1f-87f6-60541fa89cb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-s79rx" Nov 03 14:03:28.641485 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:03:28.641446 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-s79rx" Nov 03 14:03:29.229371 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:03:29.229326 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-s79rx"] Nov 03 14:03:29.623479 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:03:29.623445 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-s79rx" event={"ID":"99703a90-80bd-4e1f-87f6-60541fa89cb2","Type":"ContainerStarted","Data":"768ef937bfe02a72c017e6e7efe1090563ef520f51fce78c152c3c1a4ed8f824"} Nov 03 14:03:29.623639 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:03:29.623498 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-s79rx" event={"ID":"99703a90-80bd-4e1f-87f6-60541fa89cb2","Type":"ContainerStarted","Data":"e5642f83d50c9404e570a1bc1127c9db3b431014d0a466130630b651794acec7"} Nov 03 14:03:29.644341 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:03:29.644268 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-s79rx" podStartSLOduration=2.644255314 podStartE2EDuration="2.644255314s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:03:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:03:29.644110827 +0000 UTC m=+3289.652302107" watchObservedRunningTime="2026-11-03 14:03:29.644255314 +0000 UTC m=+3289.652446591" Nov 03 14:03:30.629598 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:03:30.629561 5024 generic.go:356] "Generic (PLEG): container finished" podID="99703a90-80bd-4e1f-87f6-60541fa89cb2" containerID="768ef937bfe02a72c017e6e7efe1090563ef520f51fce78c152c3c1a4ed8f824" exitCode=0 Nov 03 14:03:30.630020 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:03:30.629659 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-s79rx" event={"ID":"99703a90-80bd-4e1f-87f6-60541fa89cb2","Type":"ContainerDied","Data":"768ef937bfe02a72c017e6e7efe1090563ef520f51fce78c152c3c1a4ed8f824"} Nov 03 14:03:31.809086 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:03:31.809051 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-s79rx" Nov 03 14:03:31.858840 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:03:31.858798 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/99703a90-80bd-4e1f-87f6-60541fa89cb2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99703a90-80bd-4e1f-87f6-60541fa89cb2-maas-api-openshift-ca\") pod \"99703a90-80bd-4e1f-87f6-60541fa89cb2\" (UID: \"99703a90-80bd-4e1f-87f6-60541fa89cb2\") " Nov 03 14:03:31.859038 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:03:31.858903 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99703a90-80bd-4e1f-87f6-60541fa89cb2-kube-api-access-7kcqn\" (UniqueName: \"kubernetes.io/projected/99703a90-80bd-4e1f-87f6-60541fa89cb2-kube-api-access-7kcqn\") pod \"99703a90-80bd-4e1f-87f6-60541fa89cb2\" (UID: \"99703a90-80bd-4e1f-87f6-60541fa89cb2\") " Nov 03 14:03:31.859038 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:03:31.858953 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/99703a90-80bd-4e1f-87f6-60541fa89cb2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99703a90-80bd-4e1f-87f6-60541fa89cb2-maas-api-xks-ca\") pod \"99703a90-80bd-4e1f-87f6-60541fa89cb2\" (UID: \"99703a90-80bd-4e1f-87f6-60541fa89cb2\") " Nov 03 14:03:31.859250 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:03:31.859227 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99703a90-80bd-4e1f-87f6-60541fa89cb2-maas-api-openshift-ca" pod "99703a90-80bd-4e1f-87f6-60541fa89cb2" (UID: "99703a90-80bd-4e1f-87f6-60541fa89cb2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:03:31.859373 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:03:31.859350 5024 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99703a90-80bd-4e1f-87f6-60541fa89cb2-maas-api-openshift-ca\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 14:03:31.861147 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:03:31.861115 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99703a90-80bd-4e1f-87f6-60541fa89cb2-kube-api-access-7kcqn" pod "99703a90-80bd-4e1f-87f6-60541fa89cb2" (UID: "99703a90-80bd-4e1f-87f6-60541fa89cb2"). InnerVolumeSpecName "kube-api-access-7kcqn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:03:31.861563 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:03:31.861543 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/99703a90-80bd-4e1f-87f6-60541fa89cb2-maas-api-xks-ca" pod "99703a90-80bd-4e1f-87f6-60541fa89cb2" (UID: "99703a90-80bd-4e1f-87f6-60541fa89cb2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:03:31.960013 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:03:31.959941 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7kcqn\" (UniqueName: \"kubernetes.io/projected/99703a90-80bd-4e1f-87f6-60541fa89cb2-kube-api-access-7kcqn\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 14:03:31.960013 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:03:31.959971 5024 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99703a90-80bd-4e1f-87f6-60541fa89cb2-maas-api-xks-ca\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 14:03:32.641465 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:03:32.641435 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-s79rx" Nov 03 14:03:32.641641 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:03:32.641435 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-s79rx" event={"ID":"99703a90-80bd-4e1f-87f6-60541fa89cb2","Type":"ContainerDied","Data":"e5642f83d50c9404e570a1bc1127c9db3b431014d0a466130630b651794acec7"} Nov 03 14:03:32.641641 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:03:32.641586 5024 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e5642f83d50c9404e570a1bc1127c9db3b431014d0a466130630b651794acec7" Nov 03 14:03:38.492490 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:03:38.492456 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wqrj2"] Nov 03 14:03:38.492954 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:03:38.492940 5024 cpu_manager.go:414] "containerMap: removing container" podUID="99703a90-80bd-4e1f-87f6-60541fa89cb2" containerName="revoke-keys" Nov 03 14:03:38.493008 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:03:38.492955 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99703a90-80bd-4e1f-87f6-60541fa89cb2" containerName="revoke-keys" Nov 03 14:03:38.493056 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:03:38.493024 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="99703a90-80bd-4e1f-87f6-60541fa89cb2" containerName="revoke-keys" Nov 03 14:03:38.498350 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:03:38.498322 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wqrj2" Nov 03 14:03:38.504558 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:03:38.504525 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 14:03:38.506741 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:03:38.506717 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 14:03:38.506972 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:03:38.506954 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 14:03:38.507092 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:03:38.506958 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w6vqv\"" Nov 03 14:03:38.508594 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:03:38.508569 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wqrj2"] Nov 03 14:03:38.623631 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:03:38.623592 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mdf2z\" (UniqueName: \"kubernetes.io/projected/bf69b7d3-f6a6-4fe7-9134-8e882ef94375-kube-api-access-mdf2z\") pod \"maas-api-revoke-sub-3d4156f1-wqrj2\" (UID: \"bf69b7d3-f6a6-4fe7-9134-8e882ef94375\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wqrj2" Nov 03 14:03:38.623799 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:03:38.623721 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf69b7d3-f6a6-4fe7-9134-8e882ef94375-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-wqrj2\" (UID: \"bf69b7d3-f6a6-4fe7-9134-8e882ef94375\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wqrj2" Nov 03 14:03:38.623858 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:03:38.623811 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf69b7d3-f6a6-4fe7-9134-8e882ef94375-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-wqrj2\" (UID: \"bf69b7d3-f6a6-4fe7-9134-8e882ef94375\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wqrj2" Nov 03 14:03:38.724570 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:03:38.724534 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf69b7d3-f6a6-4fe7-9134-8e882ef94375-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-wqrj2\" (UID: \"bf69b7d3-f6a6-4fe7-9134-8e882ef94375\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wqrj2" Nov 03 14:03:38.724768 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:03:38.724708 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf69b7d3-f6a6-4fe7-9134-8e882ef94375-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-wqrj2\" (UID: \"bf69b7d3-f6a6-4fe7-9134-8e882ef94375\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wqrj2" Nov 03 14:03:38.724768 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:03:38.724764 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdf2z\" (UniqueName: \"kubernetes.io/projected/bf69b7d3-f6a6-4fe7-9134-8e882ef94375-kube-api-access-mdf2z\") pod \"maas-api-revoke-sub-3d4156f1-wqrj2\" (UID: \"bf69b7d3-f6a6-4fe7-9134-8e882ef94375\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wqrj2" Nov 03 14:03:38.725252 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:03:38.725227 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf69b7d3-f6a6-4fe7-9134-8e882ef94375-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-wqrj2\" (UID: \"bf69b7d3-f6a6-4fe7-9134-8e882ef94375\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wqrj2" Nov 03 14:03:38.727043 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:03:38.727023 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf69b7d3-f6a6-4fe7-9134-8e882ef94375-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-wqrj2\" (UID: \"bf69b7d3-f6a6-4fe7-9134-8e882ef94375\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wqrj2" Nov 03 14:03:38.733610 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:03:38.733589 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdf2z\" (UniqueName: \"kubernetes.io/projected/bf69b7d3-f6a6-4fe7-9134-8e882ef94375-kube-api-access-mdf2z\") pod \"maas-api-revoke-sub-3d4156f1-wqrj2\" (UID: \"bf69b7d3-f6a6-4fe7-9134-8e882ef94375\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wqrj2" Nov 03 14:03:38.819314 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:03:38.819210 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wqrj2" Nov 03 14:03:39.094601 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:03:39.094570 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wqrj2"] Nov 03 14:03:39.096344 ip-10-0-131-36 kubenswrapper[5024]: W1103 14:03:39.096278 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf69b7d3_f6a6_4fe7_9134_8e882ef94375.slice/crio-ac3d062a0d44f90a73656b77e54c15e9638ea8aa72d24301cfa5cf1aead95698 WatchSource:0}: Error finding container ac3d062a0d44f90a73656b77e54c15e9638ea8aa72d24301cfa5cf1aead95698: Status 404 returned error can't find the container with id ac3d062a0d44f90a73656b77e54c15e9638ea8aa72d24301cfa5cf1aead95698 Nov 03 14:03:39.506549 ip-10-0-131-36 kubenswrapper[5024]: E1103 14:03:39.506501 5024 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d7mwz" containerName="extractor" Nov 03 14:03:39.678391 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:03:39.678302 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wqrj2" event={"ID":"bf69b7d3-f6a6-4fe7-9134-8e882ef94375","Type":"ContainerStarted","Data":"a7c61b0a6e258cf284b2069a319d7c7c8e059fc7ebf65ee6e09371172ea5cfcb"} Nov 03 14:03:39.678391 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:03:39.678361 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wqrj2" event={"ID":"bf69b7d3-f6a6-4fe7-9134-8e882ef94375","Type":"ContainerStarted","Data":"ac3d062a0d44f90a73656b77e54c15e9638ea8aa72d24301cfa5cf1aead95698"} Nov 03 14:03:39.698548 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:03:39.698483 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wqrj2" podStartSLOduration=1.6984704069999998 podStartE2EDuration="1.698470407s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:03:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:03:39.696519875 +0000 UTC m=+3299.704711153" watchObservedRunningTime="2026-11-03 14:03:39.698470407 +0000 UTC m=+3299.706661684" Nov 03 14:03:40.685757 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:03:40.685720 5024 generic.go:356] "Generic (PLEG): container finished" podID="bf69b7d3-f6a6-4fe7-9134-8e882ef94375" containerID="a7c61b0a6e258cf284b2069a319d7c7c8e059fc7ebf65ee6e09371172ea5cfcb" exitCode=0 Nov 03 14:03:40.686206 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:03:40.685768 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wqrj2" event={"ID":"bf69b7d3-f6a6-4fe7-9134-8e882ef94375","Type":"ContainerDied","Data":"a7c61b0a6e258cf284b2069a319d7c7c8e059fc7ebf65ee6e09371172ea5cfcb"} Nov 03 14:03:41.521272 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:03:41.521240 5024 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6htht_cd91de08-084e-4667-b13f-2e07b3cff624/console-operator/1.log" Nov 03 14:03:41.538203 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:03:41.538175 5024 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6htht_cd91de08-084e-4667-b13f-2e07b3cff624/console-operator/1.log" Nov 03 14:03:41.825726 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:03:41.825702 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wqrj2" Nov 03 14:03:41.957811 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:03:41.957766 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf69b7d3-f6a6-4fe7-9134-8e882ef94375-kube-api-access-mdf2z\" (UniqueName: \"kubernetes.io/projected/bf69b7d3-f6a6-4fe7-9134-8e882ef94375-kube-api-access-mdf2z\") pod \"bf69b7d3-f6a6-4fe7-9134-8e882ef94375\" (UID: \"bf69b7d3-f6a6-4fe7-9134-8e882ef94375\") " Nov 03 14:03:41.957991 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:03:41.957836 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bf69b7d3-f6a6-4fe7-9134-8e882ef94375-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf69b7d3-f6a6-4fe7-9134-8e882ef94375-maas-api-xks-ca\") pod \"bf69b7d3-f6a6-4fe7-9134-8e882ef94375\" (UID: \"bf69b7d3-f6a6-4fe7-9134-8e882ef94375\") " Nov 03 14:03:41.957991 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:03:41.957924 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf69b7d3-f6a6-4fe7-9134-8e882ef94375-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf69b7d3-f6a6-4fe7-9134-8e882ef94375-maas-api-openshift-ca\") pod \"bf69b7d3-f6a6-4fe7-9134-8e882ef94375\" (UID: \"bf69b7d3-f6a6-4fe7-9134-8e882ef94375\") " Nov 03 14:03:41.958318 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:03:41.958274 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf69b7d3-f6a6-4fe7-9134-8e882ef94375-maas-api-openshift-ca" pod "bf69b7d3-f6a6-4fe7-9134-8e882ef94375" (UID: "bf69b7d3-f6a6-4fe7-9134-8e882ef94375"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:03:41.960104 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:03:41.960080 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bf69b7d3-f6a6-4fe7-9134-8e882ef94375-maas-api-xks-ca" pod "bf69b7d3-f6a6-4fe7-9134-8e882ef94375" (UID: "bf69b7d3-f6a6-4fe7-9134-8e882ef94375"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:03:41.960205 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:03:41.960084 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf69b7d3-f6a6-4fe7-9134-8e882ef94375-kube-api-access-mdf2z" pod "bf69b7d3-f6a6-4fe7-9134-8e882ef94375" (UID: "bf69b7d3-f6a6-4fe7-9134-8e882ef94375"). InnerVolumeSpecName "kube-api-access-mdf2z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:03:42.058558 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:03:42.058489 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mdf2z\" (UniqueName: \"kubernetes.io/projected/bf69b7d3-f6a6-4fe7-9134-8e882ef94375-kube-api-access-mdf2z\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 14:03:42.058558 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:03:42.058522 5024 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf69b7d3-f6a6-4fe7-9134-8e882ef94375-maas-api-xks-ca\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 14:03:42.058558 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:03:42.058532 5024 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf69b7d3-f6a6-4fe7-9134-8e882ef94375-maas-api-openshift-ca\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 14:03:42.696512 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:03:42.696474 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wqrj2" Nov 03 14:03:42.696698 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:03:42.696508 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wqrj2" event={"ID":"bf69b7d3-f6a6-4fe7-9134-8e882ef94375","Type":"ContainerDied","Data":"ac3d062a0d44f90a73656b77e54c15e9638ea8aa72d24301cfa5cf1aead95698"} Nov 03 14:03:42.696698 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:03:42.696594 5024 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ac3d062a0d44f90a73656b77e54c15e9638ea8aa72d24301cfa5cf1aead95698" Nov 03 14:03:53.392058 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:03:53.392016 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wqrj2"] Nov 03 14:03:53.394485 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:03:53.394453 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wqrj2"] Nov 03 14:03:53.413966 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:03:53.413924 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ql29v"] Nov 03 14:03:53.414413 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:03:53.414398 5024 cpu_manager.go:414] "containerMap: removing container" podUID="bf69b7d3-f6a6-4fe7-9134-8e882ef94375" containerName="revoke-keys" Nov 03 14:03:53.414510 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:03:53.414414 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf69b7d3-f6a6-4fe7-9134-8e882ef94375" containerName="revoke-keys" Nov 03 14:03:53.414510 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:03:53.414491 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf69b7d3-f6a6-4fe7-9134-8e882ef94375" containerName="revoke-keys" Nov 03 14:03:53.417972 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:03:53.417949 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ql29v" Nov 03 14:03:53.425630 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:03:53.425599 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 14:03:53.425909 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:03:53.425886 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w6vqv\"" Nov 03 14:03:53.426012 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:03:53.425994 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 14:03:53.426081 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:03:53.426063 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 14:03:53.428563 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:03:53.428538 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ql29v"] Nov 03 14:03:53.527421 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:03:53.527386 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0714ebce-b2b9-48c8-b921-402c03e4fb7e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-ql29v\" (UID: \"0714ebce-b2b9-48c8-b921-402c03e4fb7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ql29v" Nov 03 14:03:53.527631 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:03:53.527438 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ng2rx\" (UniqueName: \"kubernetes.io/projected/0714ebce-b2b9-48c8-b921-402c03e4fb7e-kube-api-access-ng2rx\") pod \"maas-api-revoke-sub-3d4156f1-ql29v\" (UID: \"0714ebce-b2b9-48c8-b921-402c03e4fb7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ql29v" Nov 03 14:03:53.527631 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:03:53.527457 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0714ebce-b2b9-48c8-b921-402c03e4fb7e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-ql29v\" (UID: \"0714ebce-b2b9-48c8-b921-402c03e4fb7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ql29v" Nov 03 14:03:53.629318 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:03:53.629241 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0714ebce-b2b9-48c8-b921-402c03e4fb7e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-ql29v\" (UID: \"0714ebce-b2b9-48c8-b921-402c03e4fb7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ql29v" Nov 03 14:03:53.629516 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:03:53.629359 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ng2rx\" (UniqueName: \"kubernetes.io/projected/0714ebce-b2b9-48c8-b921-402c03e4fb7e-kube-api-access-ng2rx\") pod \"maas-api-revoke-sub-3d4156f1-ql29v\" (UID: \"0714ebce-b2b9-48c8-b921-402c03e4fb7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ql29v" Nov 03 14:03:53.629516 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:03:53.629391 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0714ebce-b2b9-48c8-b921-402c03e4fb7e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-ql29v\" (UID: \"0714ebce-b2b9-48c8-b921-402c03e4fb7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ql29v" Nov 03 14:03:53.629995 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:03:53.629965 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0714ebce-b2b9-48c8-b921-402c03e4fb7e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-ql29v\" (UID: \"0714ebce-b2b9-48c8-b921-402c03e4fb7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ql29v" Nov 03 14:03:53.631966 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:03:53.631925 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0714ebce-b2b9-48c8-b921-402c03e4fb7e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-ql29v\" (UID: \"0714ebce-b2b9-48c8-b921-402c03e4fb7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ql29v" Nov 03 14:03:53.641353 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:03:53.641329 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ng2rx\" (UniqueName: \"kubernetes.io/projected/0714ebce-b2b9-48c8-b921-402c03e4fb7e-kube-api-access-ng2rx\") pod \"maas-api-revoke-sub-3d4156f1-ql29v\" (UID: \"0714ebce-b2b9-48c8-b921-402c03e4fb7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ql29v" Nov 03 14:03:53.736789 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:03:53.736714 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ql29v" Nov 03 14:03:54.020998 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:03:54.020968 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ql29v"] Nov 03 14:03:54.022515 ip-10-0-131-36 kubenswrapper[5024]: W1103 14:03:54.022475 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0714ebce_b2b9_48c8_b921_402c03e4fb7e.slice/crio-f0b2aefd8cd7c309cac6019881343d1a9839d5fa5c921bbbe6b7104a93f2ec55 WatchSource:0}: Error finding container f0b2aefd8cd7c309cac6019881343d1a9839d5fa5c921bbbe6b7104a93f2ec55: Status 404 returned error can't find the container with id f0b2aefd8cd7c309cac6019881343d1a9839d5fa5c921bbbe6b7104a93f2ec55 Nov 03 14:03:54.107169 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:03:54.107136 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf69b7d3-f6a6-4fe7-9134-8e882ef94375" path="/var/lib/kubelet/pods/bf69b7d3-f6a6-4fe7-9134-8e882ef94375/volumes" Nov 03 14:03:54.798510 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:03:54.798463 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ql29v" event={"ID":"0714ebce-b2b9-48c8-b921-402c03e4fb7e","Type":"ContainerStarted","Data":"75bdbbbd68f417e0264e4af6dfcccf3e824a654fe429b7b6465a8e102c07bfb0"} Nov 03 14:03:54.798965 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:03:54.798539 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ql29v" event={"ID":"0714ebce-b2b9-48c8-b921-402c03e4fb7e","Type":"ContainerStarted","Data":"f0b2aefd8cd7c309cac6019881343d1a9839d5fa5c921bbbe6b7104a93f2ec55"} Nov 03 14:03:54.817229 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:03:54.817168 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ql29v" podStartSLOduration=1.8171507569999998 podStartE2EDuration="1.817150757s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:03:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:03:54.816357893 +0000 UTC m=+3314.824549184" watchObservedRunningTime="2026-11-03 14:03:54.817150757 +0000 UTC m=+3314.825342036" Nov 03 14:03:55.806157 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:03:55.806122 5024 generic.go:356] "Generic (PLEG): container finished" podID="0714ebce-b2b9-48c8-b921-402c03e4fb7e" containerID="75bdbbbd68f417e0264e4af6dfcccf3e824a654fe429b7b6465a8e102c07bfb0" exitCode=0 Nov 03 14:03:55.806600 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:03:55.806190 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ql29v" event={"ID":"0714ebce-b2b9-48c8-b921-402c03e4fb7e","Type":"ContainerDied","Data":"75bdbbbd68f417e0264e4af6dfcccf3e824a654fe429b7b6465a8e102c07bfb0"} Nov 03 14:03:56.938456 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:03:56.938430 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ql29v" Nov 03 14:03:57.063896 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:03:57.063804 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0714ebce-b2b9-48c8-b921-402c03e4fb7e-kube-api-access-ng2rx\" (UniqueName: \"kubernetes.io/projected/0714ebce-b2b9-48c8-b921-402c03e4fb7e-kube-api-access-ng2rx\") pod \"0714ebce-b2b9-48c8-b921-402c03e4fb7e\" (UID: \"0714ebce-b2b9-48c8-b921-402c03e4fb7e\") " Nov 03 14:03:57.064062 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:03:57.063945 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0714ebce-b2b9-48c8-b921-402c03e4fb7e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0714ebce-b2b9-48c8-b921-402c03e4fb7e-maas-api-openshift-ca\") pod \"0714ebce-b2b9-48c8-b921-402c03e4fb7e\" (UID: \"0714ebce-b2b9-48c8-b921-402c03e4fb7e\") " Nov 03 14:03:57.064062 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:03:57.063983 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0714ebce-b2b9-48c8-b921-402c03e4fb7e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0714ebce-b2b9-48c8-b921-402c03e4fb7e-maas-api-xks-ca\") pod \"0714ebce-b2b9-48c8-b921-402c03e4fb7e\" (UID: \"0714ebce-b2b9-48c8-b921-402c03e4fb7e\") " Nov 03 14:03:57.064347 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:03:57.064320 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0714ebce-b2b9-48c8-b921-402c03e4fb7e-maas-api-openshift-ca" pod "0714ebce-b2b9-48c8-b921-402c03e4fb7e" (UID: "0714ebce-b2b9-48c8-b921-402c03e4fb7e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:03:57.066121 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:03:57.066089 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0714ebce-b2b9-48c8-b921-402c03e4fb7e-kube-api-access-ng2rx" pod "0714ebce-b2b9-48c8-b921-402c03e4fb7e" (UID: "0714ebce-b2b9-48c8-b921-402c03e4fb7e"). InnerVolumeSpecName "kube-api-access-ng2rx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:03:57.066244 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:03:57.066138 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0714ebce-b2b9-48c8-b921-402c03e4fb7e-maas-api-xks-ca" pod "0714ebce-b2b9-48c8-b921-402c03e4fb7e" (UID: "0714ebce-b2b9-48c8-b921-402c03e4fb7e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:03:57.165137 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:03:57.165107 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ng2rx\" (UniqueName: \"kubernetes.io/projected/0714ebce-b2b9-48c8-b921-402c03e4fb7e-kube-api-access-ng2rx\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 14:03:57.165137 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:03:57.165135 5024 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0714ebce-b2b9-48c8-b921-402c03e4fb7e-maas-api-openshift-ca\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 14:03:57.165313 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:03:57.165146 5024 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0714ebce-b2b9-48c8-b921-402c03e4fb7e-maas-api-xks-ca\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 14:03:57.817983 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:03:57.817951 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ql29v" Nov 03 14:03:57.818176 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:03:57.817975 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ql29v" event={"ID":"0714ebce-b2b9-48c8-b921-402c03e4fb7e","Type":"ContainerDied","Data":"f0b2aefd8cd7c309cac6019881343d1a9839d5fa5c921bbbe6b7104a93f2ec55"} Nov 03 14:03:57.818176 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:03:57.818030 5024 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f0b2aefd8cd7c309cac6019881343d1a9839d5fa5c921bbbe6b7104a93f2ec55" Nov 03 14:04:09.507525 ip-10-0-131-36 kubenswrapper[5024]: E1103 14:04:09.507482 5024 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d7mwz" containerName="extractor" Nov 03 14:04:18.830276 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:04:18.830238 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-v4dgr"] Nov 03 14:04:18.830747 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:04:18.830731 5024 cpu_manager.go:414] "containerMap: removing container" podUID="0714ebce-b2b9-48c8-b921-402c03e4fb7e" containerName="revoke-keys" Nov 03 14:04:18.830793 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:04:18.830748 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0714ebce-b2b9-48c8-b921-402c03e4fb7e" containerName="revoke-keys" Nov 03 14:04:18.830829 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:04:18.830823 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="0714ebce-b2b9-48c8-b921-402c03e4fb7e" containerName="revoke-keys" Nov 03 14:04:18.834172 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:04:18.834148 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-v4dgr" Nov 03 14:04:18.842701 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:04:18.842660 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 14:04:18.842840 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:04:18.842734 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w6vqv\"" Nov 03 14:04:18.842915 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:04:18.842820 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 14:04:18.842915 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:04:18.842711 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 14:04:18.845279 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:04:18.845257 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-v4dgr"] Nov 03 14:04:18.918335 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:04:18.918299 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x9662\" (UniqueName: \"kubernetes.io/projected/9dc93a95-b62a-4887-8b2b-3ff84104a3ea-kube-api-access-x9662\") pod \"maas-api-revoke-sub-e2fc9c5b-v4dgr\" (UID: \"9dc93a95-b62a-4887-8b2b-3ff84104a3ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-v4dgr" Nov 03 14:04:18.918517 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:04:18.918347 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9dc93a95-b62a-4887-8b2b-3ff84104a3ea-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-v4dgr\" (UID: \"9dc93a95-b62a-4887-8b2b-3ff84104a3ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-v4dgr" Nov 03 14:04:18.918517 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:04:18.918374 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9dc93a95-b62a-4887-8b2b-3ff84104a3ea-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-v4dgr\" (UID: \"9dc93a95-b62a-4887-8b2b-3ff84104a3ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-v4dgr" Nov 03 14:04:19.019028 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:04:19.018990 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9dc93a95-b62a-4887-8b2b-3ff84104a3ea-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-v4dgr\" (UID: \"9dc93a95-b62a-4887-8b2b-3ff84104a3ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-v4dgr" Nov 03 14:04:19.019196 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:04:19.019151 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9662\" (UniqueName: \"kubernetes.io/projected/9dc93a95-b62a-4887-8b2b-3ff84104a3ea-kube-api-access-x9662\") pod \"maas-api-revoke-sub-e2fc9c5b-v4dgr\" (UID: \"9dc93a95-b62a-4887-8b2b-3ff84104a3ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-v4dgr" Nov 03 14:04:19.019246 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:04:19.019194 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9dc93a95-b62a-4887-8b2b-3ff84104a3ea-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-v4dgr\" (UID: \"9dc93a95-b62a-4887-8b2b-3ff84104a3ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-v4dgr" Nov 03 14:04:19.019868 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:04:19.019843 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9dc93a95-b62a-4887-8b2b-3ff84104a3ea-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-v4dgr\" (UID: \"9dc93a95-b62a-4887-8b2b-3ff84104a3ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-v4dgr" Nov 03 14:04:19.021549 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:04:19.021526 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9dc93a95-b62a-4887-8b2b-3ff84104a3ea-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-v4dgr\" (UID: \"9dc93a95-b62a-4887-8b2b-3ff84104a3ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-v4dgr" Nov 03 14:04:19.029026 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:04:19.028995 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9662\" (UniqueName: \"kubernetes.io/projected/9dc93a95-b62a-4887-8b2b-3ff84104a3ea-kube-api-access-x9662\") pod \"maas-api-revoke-sub-e2fc9c5b-v4dgr\" (UID: \"9dc93a95-b62a-4887-8b2b-3ff84104a3ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-v4dgr" Nov 03 14:04:19.149706 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:04:19.149660 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-v4dgr" Nov 03 14:04:19.426576 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:04:19.426531 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-v4dgr"] Nov 03 14:04:19.428305 ip-10-0-131-36 kubenswrapper[5024]: W1103 14:04:19.428263 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9dc93a95_b62a_4887_8b2b_3ff84104a3ea.slice/crio-8ef097c15d800c0711beb864202fb7b2f437634522246453d6fb24fda211cf1b WatchSource:0}: Error finding container 8ef097c15d800c0711beb864202fb7b2f437634522246453d6fb24fda211cf1b: Status 404 returned error can't find the container with id 8ef097c15d800c0711beb864202fb7b2f437634522246453d6fb24fda211cf1b Nov 03 14:04:19.968013 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:04:19.967920 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-v4dgr" event={"ID":"9dc93a95-b62a-4887-8b2b-3ff84104a3ea","Type":"ContainerStarted","Data":"a37f2f00ca631d1290d5eace140d67b46db2c820465cc1bf75f4aa4797421285"} Nov 03 14:04:19.968521 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:04:19.968018 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-v4dgr" event={"ID":"9dc93a95-b62a-4887-8b2b-3ff84104a3ea","Type":"ContainerStarted","Data":"8ef097c15d800c0711beb864202fb7b2f437634522246453d6fb24fda211cf1b"} Nov 03 14:04:19.987469 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:04:19.987413 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-v4dgr" podStartSLOduration=1.9873997289999998 podStartE2EDuration="1.987399729s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:04:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:04:19.984987738 +0000 UTC m=+3339.993179013" watchObservedRunningTime="2026-11-03 14:04:19.987399729 +0000 UTC m=+3339.995591005" Nov 03 14:04:20.974163 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:04:20.974127 5024 generic.go:356] "Generic (PLEG): container finished" podID="9dc93a95-b62a-4887-8b2b-3ff84104a3ea" containerID="a37f2f00ca631d1290d5eace140d67b46db2c820465cc1bf75f4aa4797421285" exitCode=0 Nov 03 14:04:20.974599 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:04:20.974170 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-v4dgr" event={"ID":"9dc93a95-b62a-4887-8b2b-3ff84104a3ea","Type":"ContainerDied","Data":"a37f2f00ca631d1290d5eace140d67b46db2c820465cc1bf75f4aa4797421285"} Nov 03 14:04:22.104208 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:04:22.104183 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-v4dgr" Nov 03 14:04:22.250579 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:04:22.250466 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9dc93a95-b62a-4887-8b2b-3ff84104a3ea-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9dc93a95-b62a-4887-8b2b-3ff84104a3ea-maas-api-openshift-ca\") pod \"9dc93a95-b62a-4887-8b2b-3ff84104a3ea\" (UID: \"9dc93a95-b62a-4887-8b2b-3ff84104a3ea\") " Nov 03 14:04:22.250579 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:04:22.250528 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9dc93a95-b62a-4887-8b2b-3ff84104a3ea-kube-api-access-x9662\" (UniqueName: \"kubernetes.io/projected/9dc93a95-b62a-4887-8b2b-3ff84104a3ea-kube-api-access-x9662\") pod \"9dc93a95-b62a-4887-8b2b-3ff84104a3ea\" (UID: \"9dc93a95-b62a-4887-8b2b-3ff84104a3ea\") " Nov 03 14:04:22.250836 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:04:22.250598 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9dc93a95-b62a-4887-8b2b-3ff84104a3ea-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9dc93a95-b62a-4887-8b2b-3ff84104a3ea-maas-api-xks-ca\") pod \"9dc93a95-b62a-4887-8b2b-3ff84104a3ea\" (UID: \"9dc93a95-b62a-4887-8b2b-3ff84104a3ea\") " Nov 03 14:04:22.250997 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:04:22.250973 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9dc93a95-b62a-4887-8b2b-3ff84104a3ea-maas-api-openshift-ca" pod "9dc93a95-b62a-4887-8b2b-3ff84104a3ea" (UID: "9dc93a95-b62a-4887-8b2b-3ff84104a3ea"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:04:22.252642 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:04:22.252618 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9dc93a95-b62a-4887-8b2b-3ff84104a3ea-maas-api-xks-ca" pod "9dc93a95-b62a-4887-8b2b-3ff84104a3ea" (UID: "9dc93a95-b62a-4887-8b2b-3ff84104a3ea"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:04:22.252729 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:04:22.252708 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9dc93a95-b62a-4887-8b2b-3ff84104a3ea-kube-api-access-x9662" pod "9dc93a95-b62a-4887-8b2b-3ff84104a3ea" (UID: "9dc93a95-b62a-4887-8b2b-3ff84104a3ea"). InnerVolumeSpecName "kube-api-access-x9662". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:04:22.352142 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:04:22.352118 5024 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9dc93a95-b62a-4887-8b2b-3ff84104a3ea-maas-api-openshift-ca\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 14:04:22.352142 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:04:22.352141 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x9662\" (UniqueName: \"kubernetes.io/projected/9dc93a95-b62a-4887-8b2b-3ff84104a3ea-kube-api-access-x9662\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 14:04:22.352278 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:04:22.352151 5024 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9dc93a95-b62a-4887-8b2b-3ff84104a3ea-maas-api-xks-ca\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 14:04:22.986756 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:04:22.986725 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-v4dgr" Nov 03 14:04:22.986756 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:04:22.986742 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-v4dgr" event={"ID":"9dc93a95-b62a-4887-8b2b-3ff84104a3ea","Type":"ContainerDied","Data":"8ef097c15d800c0711beb864202fb7b2f437634522246453d6fb24fda211cf1b"} Nov 03 14:04:22.987016 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:04:22.986785 5024 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8ef097c15d800c0711beb864202fb7b2f437634522246453d6fb24fda211cf1b" Nov 03 14:04:29.054685 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:04:29.054622 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-2ncfv"] Nov 03 14:04:29.055206 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:04:29.055096 5024 cpu_manager.go:414] "containerMap: removing container" podUID="9dc93a95-b62a-4887-8b2b-3ff84104a3ea" containerName="revoke-keys" Nov 03 14:04:29.055206 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:04:29.055113 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9dc93a95-b62a-4887-8b2b-3ff84104a3ea" containerName="revoke-keys" Nov 03 14:04:29.055346 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:04:29.055225 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="9dc93a95-b62a-4887-8b2b-3ff84104a3ea" containerName="revoke-keys" Nov 03 14:04:29.059318 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:04:29.059279 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-2ncfv" Nov 03 14:04:29.065193 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:04:29.065065 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 14:04:29.065193 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:04:29.065109 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 14:04:29.065193 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:04:29.065180 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 14:04:29.066587 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:04:29.066566 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w6vqv\"" Nov 03 14:04:29.066750 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:04:29.066725 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-2ncfv"] Nov 03 14:04:29.217613 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:04:29.217579 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lqfpw\" (UniqueName: \"kubernetes.io/projected/b39ad908-ed87-45cc-9658-d5a2151a1c3e-kube-api-access-lqfpw\") pod \"maas-api-revoke-sub-5a29efb3-2ncfv\" (UID: \"b39ad908-ed87-45cc-9658-d5a2151a1c3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-2ncfv" Nov 03 14:04:29.217789 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:04:29.217628 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b39ad908-ed87-45cc-9658-d5a2151a1c3e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-2ncfv\" (UID: \"b39ad908-ed87-45cc-9658-d5a2151a1c3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-2ncfv" Nov 03 14:04:29.218616 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:04:29.218579 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b39ad908-ed87-45cc-9658-d5a2151a1c3e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-2ncfv\" (UID: \"b39ad908-ed87-45cc-9658-d5a2151a1c3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-2ncfv" Nov 03 14:04:29.321835 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:04:29.321009 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqfpw\" (UniqueName: \"kubernetes.io/projected/b39ad908-ed87-45cc-9658-d5a2151a1c3e-kube-api-access-lqfpw\") pod \"maas-api-revoke-sub-5a29efb3-2ncfv\" (UID: \"b39ad908-ed87-45cc-9658-d5a2151a1c3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-2ncfv" Nov 03 14:04:29.321835 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:04:29.321075 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b39ad908-ed87-45cc-9658-d5a2151a1c3e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-2ncfv\" (UID: \"b39ad908-ed87-45cc-9658-d5a2151a1c3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-2ncfv" Nov 03 14:04:29.321835 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:04:29.321131 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b39ad908-ed87-45cc-9658-d5a2151a1c3e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-2ncfv\" (UID: \"b39ad908-ed87-45cc-9658-d5a2151a1c3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-2ncfv" Nov 03 14:04:29.327013 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:04:29.324810 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b39ad908-ed87-45cc-9658-d5a2151a1c3e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-2ncfv\" (UID: \"b39ad908-ed87-45cc-9658-d5a2151a1c3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-2ncfv" Nov 03 14:04:29.327013 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:04:29.325798 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b39ad908-ed87-45cc-9658-d5a2151a1c3e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-2ncfv\" (UID: \"b39ad908-ed87-45cc-9658-d5a2151a1c3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-2ncfv" Nov 03 14:04:29.334640 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:04:29.334615 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lqfpw\" (UniqueName: \"kubernetes.io/projected/b39ad908-ed87-45cc-9658-d5a2151a1c3e-kube-api-access-lqfpw\") pod \"maas-api-revoke-sub-5a29efb3-2ncfv\" (UID: \"b39ad908-ed87-45cc-9658-d5a2151a1c3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-2ncfv" Nov 03 14:04:29.378680 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:04:29.378651 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-2ncfv" Nov 03 14:04:29.660947 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:04:29.660917 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-2ncfv"] Nov 03 14:04:29.661967 ip-10-0-131-36 kubenswrapper[5024]: W1103 14:04:29.661938 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb39ad908_ed87_45cc_9658_d5a2151a1c3e.slice/crio-56d9d717cc0fe1a668a7d9cb3b325b88f0edb70b546bcd38524565e020974a38 WatchSource:0}: Error finding container 56d9d717cc0fe1a668a7d9cb3b325b88f0edb70b546bcd38524565e020974a38: Status 404 returned error can't find the container with id 56d9d717cc0fe1a668a7d9cb3b325b88f0edb70b546bcd38524565e020974a38 Nov 03 14:04:30.025315 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:04:30.025257 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-2ncfv" event={"ID":"b39ad908-ed87-45cc-9658-d5a2151a1c3e","Type":"ContainerStarted","Data":"e2e4e596c0abde1f96a61125780b8b7e85496545012767a9a71e87f67eebe3d2"} Nov 03 14:04:30.025511 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:04:30.025344 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-2ncfv" event={"ID":"b39ad908-ed87-45cc-9658-d5a2151a1c3e","Type":"ContainerStarted","Data":"56d9d717cc0fe1a668a7d9cb3b325b88f0edb70b546bcd38524565e020974a38"} Nov 03 14:04:30.044471 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:04:30.044415 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-2ncfv" podStartSLOduration=1.044401258 podStartE2EDuration="1.044401258s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:04:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:04:30.043996002 +0000 UTC m=+3350.052187279" watchObservedRunningTime="2026-11-03 14:04:30.044401258 +0000 UTC m=+3350.052592809" Nov 03 14:04:31.037033 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:04:31.036989 5024 generic.go:356] "Generic (PLEG): container finished" podID="b39ad908-ed87-45cc-9658-d5a2151a1c3e" containerID="e2e4e596c0abde1f96a61125780b8b7e85496545012767a9a71e87f67eebe3d2" exitCode=0 Nov 03 14:04:31.037777 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:04:31.037745 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-2ncfv" event={"ID":"b39ad908-ed87-45cc-9658-d5a2151a1c3e","Type":"ContainerDied","Data":"e2e4e596c0abde1f96a61125780b8b7e85496545012767a9a71e87f67eebe3d2"} Nov 03 14:04:32.965308 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:04:32.965207 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-2ncfv" Nov 03 14:04:33.069326 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:04:33.069261 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b39ad908-ed87-45cc-9658-d5a2151a1c3e-kube-api-access-lqfpw\" (UniqueName: \"kubernetes.io/projected/b39ad908-ed87-45cc-9658-d5a2151a1c3e-kube-api-access-lqfpw\") pod \"b39ad908-ed87-45cc-9658-d5a2151a1c3e\" (UID: \"b39ad908-ed87-45cc-9658-d5a2151a1c3e\") " Nov 03 14:04:33.069668 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:04:33.069648 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b39ad908-ed87-45cc-9658-d5a2151a1c3e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b39ad908-ed87-45cc-9658-d5a2151a1c3e-maas-api-openshift-ca\") pod \"b39ad908-ed87-45cc-9658-d5a2151a1c3e\" (UID: \"b39ad908-ed87-45cc-9658-d5a2151a1c3e\") " Nov 03 14:04:33.069926 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:04:33.069910 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b39ad908-ed87-45cc-9658-d5a2151a1c3e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b39ad908-ed87-45cc-9658-d5a2151a1c3e-maas-api-xks-ca\") pod \"b39ad908-ed87-45cc-9658-d5a2151a1c3e\" (UID: \"b39ad908-ed87-45cc-9658-d5a2151a1c3e\") " Nov 03 14:04:33.070882 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:04:33.070840 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b39ad908-ed87-45cc-9658-d5a2151a1c3e-maas-api-openshift-ca" pod "b39ad908-ed87-45cc-9658-d5a2151a1c3e" (UID: "b39ad908-ed87-45cc-9658-d5a2151a1c3e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:04:33.084404 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:04:33.084087 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-2ncfv" event={"ID":"b39ad908-ed87-45cc-9658-d5a2151a1c3e","Type":"ContainerDied","Data":"56d9d717cc0fe1a668a7d9cb3b325b88f0edb70b546bcd38524565e020974a38"} Nov 03 14:04:33.084404 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:04:33.084149 5024 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="56d9d717cc0fe1a668a7d9cb3b325b88f0edb70b546bcd38524565e020974a38" Nov 03 14:04:33.084404 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:04:33.084223 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-2ncfv" Nov 03 14:04:33.085165 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:04:33.085127 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b39ad908-ed87-45cc-9658-d5a2151a1c3e-kube-api-access-lqfpw" pod "b39ad908-ed87-45cc-9658-d5a2151a1c3e" (UID: "b39ad908-ed87-45cc-9658-d5a2151a1c3e"). InnerVolumeSpecName "kube-api-access-lqfpw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:04:33.089931 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:04:33.089908 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b39ad908-ed87-45cc-9658-d5a2151a1c3e-maas-api-xks-ca" pod "b39ad908-ed87-45cc-9658-d5a2151a1c3e" (UID: "b39ad908-ed87-45cc-9658-d5a2151a1c3e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:04:33.172013 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:04:33.171246 5024 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b39ad908-ed87-45cc-9658-d5a2151a1c3e-maas-api-xks-ca\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 14:04:33.172013 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:04:33.171303 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lqfpw\" (UniqueName: \"kubernetes.io/projected/b39ad908-ed87-45cc-9658-d5a2151a1c3e-kube-api-access-lqfpw\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 14:04:33.172013 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:04:33.171321 5024 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b39ad908-ed87-45cc-9658-d5a2151a1c3e-maas-api-openshift-ca\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 14:04:39.511315 ip-10-0-131-36 kubenswrapper[5024]: E1103 14:04:39.511208 5024 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d7mwz" containerName="extractor" Nov 03 14:04:44.984873 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:04:44.984836 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ql29v"] Nov 03 14:04:44.987069 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:04:44.987045 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ql29v"] Nov 03 14:04:45.011912 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:04:45.011879 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cqm22"] Nov 03 14:04:45.012350 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:04:45.012336 5024 cpu_manager.go:414] "containerMap: removing container" podUID="b39ad908-ed87-45cc-9658-d5a2151a1c3e" containerName="revoke-keys" Nov 03 14:04:45.012412 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:04:45.012352 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b39ad908-ed87-45cc-9658-d5a2151a1c3e" containerName="revoke-keys" Nov 03 14:04:45.012454 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:04:45.012428 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="b39ad908-ed87-45cc-9658-d5a2151a1c3e" containerName="revoke-keys" Nov 03 14:04:45.015444 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:04:45.015424 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cqm22" Nov 03 14:04:45.020079 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:04:45.020048 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 14:04:45.020079 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:04:45.020061 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 14:04:45.022167 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:04:45.022140 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 14:04:45.022346 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:04:45.022212 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w6vqv\"" Nov 03 14:04:45.025106 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:04:45.025080 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cqm22"] Nov 03 14:04:45.091498 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:04:45.091457 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8bb15cc1-1bd0-4182-9162-a8ebf916bba7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-cqm22\" (UID: \"8bb15cc1-1bd0-4182-9162-a8ebf916bba7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cqm22" Nov 03 14:04:45.091498 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:04:45.091500 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5flrz\" (UniqueName: \"kubernetes.io/projected/8bb15cc1-1bd0-4182-9162-a8ebf916bba7-kube-api-access-5flrz\") pod \"maas-api-revoke-sub-3d4156f1-cqm22\" (UID: \"8bb15cc1-1bd0-4182-9162-a8ebf916bba7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cqm22" Nov 03 14:04:45.091766 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:04:45.091521 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8bb15cc1-1bd0-4182-9162-a8ebf916bba7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-cqm22\" (UID: \"8bb15cc1-1bd0-4182-9162-a8ebf916bba7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cqm22" Nov 03 14:04:45.192612 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:04:45.192569 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8bb15cc1-1bd0-4182-9162-a8ebf916bba7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-cqm22\" (UID: \"8bb15cc1-1bd0-4182-9162-a8ebf916bba7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cqm22" Nov 03 14:04:45.192844 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:04:45.192631 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5flrz\" (UniqueName: \"kubernetes.io/projected/8bb15cc1-1bd0-4182-9162-a8ebf916bba7-kube-api-access-5flrz\") pod \"maas-api-revoke-sub-3d4156f1-cqm22\" (UID: \"8bb15cc1-1bd0-4182-9162-a8ebf916bba7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cqm22" Nov 03 14:04:45.192844 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:04:45.192653 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8bb15cc1-1bd0-4182-9162-a8ebf916bba7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-cqm22\" (UID: \"8bb15cc1-1bd0-4182-9162-a8ebf916bba7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cqm22" Nov 03 14:04:45.193479 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:04:45.193454 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8bb15cc1-1bd0-4182-9162-a8ebf916bba7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-cqm22\" (UID: \"8bb15cc1-1bd0-4182-9162-a8ebf916bba7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cqm22" Nov 03 14:04:45.195089 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:04:45.195063 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8bb15cc1-1bd0-4182-9162-a8ebf916bba7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-cqm22\" (UID: \"8bb15cc1-1bd0-4182-9162-a8ebf916bba7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cqm22" Nov 03 14:04:45.203579 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:04:45.203548 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5flrz\" (UniqueName: \"kubernetes.io/projected/8bb15cc1-1bd0-4182-9162-a8ebf916bba7-kube-api-access-5flrz\") pod \"maas-api-revoke-sub-3d4156f1-cqm22\" (UID: \"8bb15cc1-1bd0-4182-9162-a8ebf916bba7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cqm22" Nov 03 14:04:45.335933 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:04:45.335837 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cqm22" Nov 03 14:04:45.614010 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:04:45.613984 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cqm22"] Nov 03 14:04:45.615742 ip-10-0-131-36 kubenswrapper[5024]: W1103 14:04:45.615712 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8bb15cc1_1bd0_4182_9162_a8ebf916bba7.slice/crio-ff3de1cd937ec238f1fd571eb9324fcf94592c36bde497ee2863c26b950a4114 WatchSource:0}: Error finding container ff3de1cd937ec238f1fd571eb9324fcf94592c36bde497ee2863c26b950a4114: Status 404 returned error can't find the container with id ff3de1cd937ec238f1fd571eb9324fcf94592c36bde497ee2863c26b950a4114 Nov 03 14:04:46.107181 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:04:46.107141 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0714ebce-b2b9-48c8-b921-402c03e4fb7e" path="/var/lib/kubelet/pods/0714ebce-b2b9-48c8-b921-402c03e4fb7e/volumes" Nov 03 14:04:46.153504 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:04:46.153413 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cqm22" event={"ID":"8bb15cc1-1bd0-4182-9162-a8ebf916bba7","Type":"ContainerStarted","Data":"de2f02d4a7e670630a2c5e7c247283884d282128375428d9956fd026fb70693a"} Nov 03 14:04:46.153504 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:04:46.153473 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cqm22" event={"ID":"8bb15cc1-1bd0-4182-9162-a8ebf916bba7","Type":"ContainerStarted","Data":"ff3de1cd937ec238f1fd571eb9324fcf94592c36bde497ee2863c26b950a4114"} Nov 03 14:04:46.177263 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:04:46.177194 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cqm22" podStartSLOduration=2.177179724 podStartE2EDuration="2.177179724s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:04:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:04:46.173697129 +0000 UTC m=+3366.181888410" watchObservedRunningTime="2026-11-03 14:04:46.177179724 +0000 UTC m=+3366.185371000" Nov 03 14:04:47.160140 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:04:47.160104 5024 generic.go:356] "Generic (PLEG): container finished" podID="8bb15cc1-1bd0-4182-9162-a8ebf916bba7" containerID="de2f02d4a7e670630a2c5e7c247283884d282128375428d9956fd026fb70693a" exitCode=0 Nov 03 14:04:47.160559 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:04:47.160184 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cqm22" event={"ID":"8bb15cc1-1bd0-4182-9162-a8ebf916bba7","Type":"ContainerDied","Data":"de2f02d4a7e670630a2c5e7c247283884d282128375428d9956fd026fb70693a"} Nov 03 14:04:48.302877 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:04:48.302849 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cqm22" Nov 03 14:04:48.424503 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:04:48.424412 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8bb15cc1-1bd0-4182-9162-a8ebf916bba7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8bb15cc1-1bd0-4182-9162-a8ebf916bba7-maas-api-xks-ca\") pod \"8bb15cc1-1bd0-4182-9162-a8ebf916bba7\" (UID: \"8bb15cc1-1bd0-4182-9162-a8ebf916bba7\") " Nov 03 14:04:48.424687 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:04:48.424551 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8bb15cc1-1bd0-4182-9162-a8ebf916bba7-kube-api-access-5flrz\" (UniqueName: \"kubernetes.io/projected/8bb15cc1-1bd0-4182-9162-a8ebf916bba7-kube-api-access-5flrz\") pod \"8bb15cc1-1bd0-4182-9162-a8ebf916bba7\" (UID: \"8bb15cc1-1bd0-4182-9162-a8ebf916bba7\") " Nov 03 14:04:48.424687 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:04:48.424590 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8bb15cc1-1bd0-4182-9162-a8ebf916bba7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8bb15cc1-1bd0-4182-9162-a8ebf916bba7-maas-api-openshift-ca\") pod \"8bb15cc1-1bd0-4182-9162-a8ebf916bba7\" (UID: \"8bb15cc1-1bd0-4182-9162-a8ebf916bba7\") " Nov 03 14:04:48.425003 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:04:48.424981 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8bb15cc1-1bd0-4182-9162-a8ebf916bba7-maas-api-openshift-ca" pod "8bb15cc1-1bd0-4182-9162-a8ebf916bba7" (UID: "8bb15cc1-1bd0-4182-9162-a8ebf916bba7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:04:48.426652 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:04:48.426624 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8bb15cc1-1bd0-4182-9162-a8ebf916bba7-maas-api-xks-ca" pod "8bb15cc1-1bd0-4182-9162-a8ebf916bba7" (UID: "8bb15cc1-1bd0-4182-9162-a8ebf916bba7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:04:48.426755 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:04:48.426659 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8bb15cc1-1bd0-4182-9162-a8ebf916bba7-kube-api-access-5flrz" pod "8bb15cc1-1bd0-4182-9162-a8ebf916bba7" (UID: "8bb15cc1-1bd0-4182-9162-a8ebf916bba7"). InnerVolumeSpecName "kube-api-access-5flrz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:04:48.526142 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:04:48.526098 5024 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8bb15cc1-1bd0-4182-9162-a8ebf916bba7-maas-api-xks-ca\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 14:04:48.526142 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:04:48.526135 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5flrz\" (UniqueName: \"kubernetes.io/projected/8bb15cc1-1bd0-4182-9162-a8ebf916bba7-kube-api-access-5flrz\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 14:04:48.526142 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:04:48.526147 5024 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8bb15cc1-1bd0-4182-9162-a8ebf916bba7-maas-api-openshift-ca\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 14:04:49.172182 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:04:49.172148 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cqm22" Nov 03 14:04:49.172408 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:04:49.172195 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cqm22" event={"ID":"8bb15cc1-1bd0-4182-9162-a8ebf916bba7","Type":"ContainerDied","Data":"ff3de1cd937ec238f1fd571eb9324fcf94592c36bde497ee2863c26b950a4114"} Nov 03 14:04:49.172408 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:04:49.172255 5024 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ff3de1cd937ec238f1fd571eb9324fcf94592c36bde497ee2863c26b950a4114" Nov 03 14:05:00.467027 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:05:00.466984 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-64455b-openshift-default-6477df7745pk7lm"] Nov 03 14:05:00.467672 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:05:00.467647 5024 cpu_manager.go:414] "containerMap: removing container" podUID="8bb15cc1-1bd0-4182-9162-a8ebf916bba7" containerName="revoke-keys" Nov 03 14:05:00.467767 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:05:00.467674 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8bb15cc1-1bd0-4182-9162-a8ebf916bba7" containerName="revoke-keys" Nov 03 14:05:00.467827 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:05:00.467811 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="8bb15cc1-1bd0-4182-9162-a8ebf916bba7" containerName="revoke-keys" Nov 03 14:05:00.471084 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:05:00.471060 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-64455b-openshift-default-6477df7745pk7lm" Nov 03 14:05:00.481529 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:05:00.481495 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-64455b-openshift-default-dockercfg-lbt6c\"" Nov 03 14:05:00.486783 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:05:00.486744 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-64455b-openshift-default-6477df7745pk7lm"] Nov 03 14:05:00.572877 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:05:00.572831 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6cb1131d-f786-4f4c-bd32-05ce26bb5411-istio-podinfo\") pod \"e2e-worker-single-main-64455b-openshift-default-6477df7745pk7lm\" (UID: \"6cb1131d-f786-4f4c-bd32-05ce26bb5411\") " pod="openshift-ingress/e2e-worker-single-main-64455b-openshift-default-6477df7745pk7lm" Nov 03 14:05:00.573101 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:05:00.572966 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6cb1131d-f786-4f4c-bd32-05ce26bb5411-workload-certs\") pod \"e2e-worker-single-main-64455b-openshift-default-6477df7745pk7lm\" (UID: \"6cb1131d-f786-4f4c-bd32-05ce26bb5411\") " pod="openshift-ingress/e2e-worker-single-main-64455b-openshift-default-6477df7745pk7lm" Nov 03 14:05:00.573101 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:05:00.573039 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6cb1131d-f786-4f4c-bd32-05ce26bb5411-credential-socket\") pod \"e2e-worker-single-main-64455b-openshift-default-6477df7745pk7lm\" (UID: \"6cb1131d-f786-4f4c-bd32-05ce26bb5411\") " pod="openshift-ingress/e2e-worker-single-main-64455b-openshift-default-6477df7745pk7lm" Nov 03 14:05:00.573101 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:05:00.573080 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6cb1131d-f786-4f4c-bd32-05ce26bb5411-istio-data\") pod \"e2e-worker-single-main-64455b-openshift-default-6477df7745pk7lm\" (UID: \"6cb1131d-f786-4f4c-bd32-05ce26bb5411\") " pod="openshift-ingress/e2e-worker-single-main-64455b-openshift-default-6477df7745pk7lm" Nov 03 14:05:00.573312 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:05:00.573150 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6cb1131d-f786-4f4c-bd32-05ce26bb5411-istio-token\") pod \"e2e-worker-single-main-64455b-openshift-default-6477df7745pk7lm\" (UID: \"6cb1131d-f786-4f4c-bd32-05ce26bb5411\") " pod="openshift-ingress/e2e-worker-single-main-64455b-openshift-default-6477df7745pk7lm" Nov 03 14:05:00.573312 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:05:00.573175 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6cb1131d-f786-4f4c-bd32-05ce26bb5411-workload-socket\") pod \"e2e-worker-single-main-64455b-openshift-default-6477df7745pk7lm\" (UID: \"6cb1131d-f786-4f4c-bd32-05ce26bb5411\") " pod="openshift-ingress/e2e-worker-single-main-64455b-openshift-default-6477df7745pk7lm" Nov 03 14:05:00.573312 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:05:00.573207 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9dwh\" (UniqueName: \"kubernetes.io/projected/6cb1131d-f786-4f4c-bd32-05ce26bb5411-kube-api-access-p9dwh\") pod \"e2e-worker-single-main-64455b-openshift-default-6477df7745pk7lm\" (UID: \"6cb1131d-f786-4f4c-bd32-05ce26bb5411\") " pod="openshift-ingress/e2e-worker-single-main-64455b-openshift-default-6477df7745pk7lm" Nov 03 14:05:00.573503 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:05:00.573332 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6cb1131d-f786-4f4c-bd32-05ce26bb5411-istio-envoy\") pod \"e2e-worker-single-main-64455b-openshift-default-6477df7745pk7lm\" (UID: \"6cb1131d-f786-4f4c-bd32-05ce26bb5411\") " pod="openshift-ingress/e2e-worker-single-main-64455b-openshift-default-6477df7745pk7lm" Nov 03 14:05:00.573503 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:05:00.573365 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6cb1131d-f786-4f4c-bd32-05ce26bb5411-istiod-ca-cert\") pod \"e2e-worker-single-main-64455b-openshift-default-6477df7745pk7lm\" (UID: \"6cb1131d-f786-4f4c-bd32-05ce26bb5411\") " pod="openshift-ingress/e2e-worker-single-main-64455b-openshift-default-6477df7745pk7lm" Nov 03 14:05:00.678863 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:05:00.678806 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6cb1131d-f786-4f4c-bd32-05ce26bb5411-workload-certs\") pod \"e2e-worker-single-main-64455b-openshift-default-6477df7745pk7lm\" (UID: \"6cb1131d-f786-4f4c-bd32-05ce26bb5411\") " pod="openshift-ingress/e2e-worker-single-main-64455b-openshift-default-6477df7745pk7lm" Nov 03 14:05:00.678863 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:05:00.678864 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6cb1131d-f786-4f4c-bd32-05ce26bb5411-credential-socket\") pod \"e2e-worker-single-main-64455b-openshift-default-6477df7745pk7lm\" (UID: \"6cb1131d-f786-4f4c-bd32-05ce26bb5411\") " pod="openshift-ingress/e2e-worker-single-main-64455b-openshift-default-6477df7745pk7lm" Nov 03 14:05:00.679223 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:05:00.678910 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6cb1131d-f786-4f4c-bd32-05ce26bb5411-istio-data\") pod \"e2e-worker-single-main-64455b-openshift-default-6477df7745pk7lm\" (UID: \"6cb1131d-f786-4f4c-bd32-05ce26bb5411\") " pod="openshift-ingress/e2e-worker-single-main-64455b-openshift-default-6477df7745pk7lm" Nov 03 14:05:00.679223 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:05:00.678947 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6cb1131d-f786-4f4c-bd32-05ce26bb5411-istio-token\") pod \"e2e-worker-single-main-64455b-openshift-default-6477df7745pk7lm\" (UID: \"6cb1131d-f786-4f4c-bd32-05ce26bb5411\") " pod="openshift-ingress/e2e-worker-single-main-64455b-openshift-default-6477df7745pk7lm" Nov 03 14:05:00.679223 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:05:00.678982 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6cb1131d-f786-4f4c-bd32-05ce26bb5411-workload-socket\") pod \"e2e-worker-single-main-64455b-openshift-default-6477df7745pk7lm\" (UID: \"6cb1131d-f786-4f4c-bd32-05ce26bb5411\") " pod="openshift-ingress/e2e-worker-single-main-64455b-openshift-default-6477df7745pk7lm" Nov 03 14:05:00.679223 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:05:00.679020 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9dwh\" (UniqueName: \"kubernetes.io/projected/6cb1131d-f786-4f4c-bd32-05ce26bb5411-kube-api-access-p9dwh\") pod \"e2e-worker-single-main-64455b-openshift-default-6477df7745pk7lm\" (UID: \"6cb1131d-f786-4f4c-bd32-05ce26bb5411\") " pod="openshift-ingress/e2e-worker-single-main-64455b-openshift-default-6477df7745pk7lm" Nov 03 14:05:00.679223 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:05:00.679069 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6cb1131d-f786-4f4c-bd32-05ce26bb5411-istio-envoy\") pod \"e2e-worker-single-main-64455b-openshift-default-6477df7745pk7lm\" (UID: \"6cb1131d-f786-4f4c-bd32-05ce26bb5411\") " pod="openshift-ingress/e2e-worker-single-main-64455b-openshift-default-6477df7745pk7lm" Nov 03 14:05:00.679223 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:05:00.679098 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6cb1131d-f786-4f4c-bd32-05ce26bb5411-istiod-ca-cert\") pod \"e2e-worker-single-main-64455b-openshift-default-6477df7745pk7lm\" (UID: \"6cb1131d-f786-4f4c-bd32-05ce26bb5411\") " pod="openshift-ingress/e2e-worker-single-main-64455b-openshift-default-6477df7745pk7lm" Nov 03 14:05:00.679223 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:05:00.679142 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6cb1131d-f786-4f4c-bd32-05ce26bb5411-istio-podinfo\") pod \"e2e-worker-single-main-64455b-openshift-default-6477df7745pk7lm\" (UID: \"6cb1131d-f786-4f4c-bd32-05ce26bb5411\") " pod="openshift-ingress/e2e-worker-single-main-64455b-openshift-default-6477df7745pk7lm" Nov 03 14:05:00.679775 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:05:00.679264 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6cb1131d-f786-4f4c-bd32-05ce26bb5411-workload-certs\") pod \"e2e-worker-single-main-64455b-openshift-default-6477df7745pk7lm\" (UID: \"6cb1131d-f786-4f4c-bd32-05ce26bb5411\") " pod="openshift-ingress/e2e-worker-single-main-64455b-openshift-default-6477df7745pk7lm" Nov 03 14:05:00.679775 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:05:00.679582 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6cb1131d-f786-4f4c-bd32-05ce26bb5411-workload-socket\") pod \"e2e-worker-single-main-64455b-openshift-default-6477df7745pk7lm\" (UID: \"6cb1131d-f786-4f4c-bd32-05ce26bb5411\") " pod="openshift-ingress/e2e-worker-single-main-64455b-openshift-default-6477df7745pk7lm" Nov 03 14:05:00.679897 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:05:00.679832 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6cb1131d-f786-4f4c-bd32-05ce26bb5411-credential-socket\") pod \"e2e-worker-single-main-64455b-openshift-default-6477df7745pk7lm\" (UID: \"6cb1131d-f786-4f4c-bd32-05ce26bb5411\") " pod="openshift-ingress/e2e-worker-single-main-64455b-openshift-default-6477df7745pk7lm" Nov 03 14:05:00.680093 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:05:00.680066 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6cb1131d-f786-4f4c-bd32-05ce26bb5411-istio-data\") pod \"e2e-worker-single-main-64455b-openshift-default-6477df7745pk7lm\" (UID: \"6cb1131d-f786-4f4c-bd32-05ce26bb5411\") " pod="openshift-ingress/e2e-worker-single-main-64455b-openshift-default-6477df7745pk7lm" Nov 03 14:05:00.680845 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:05:00.680819 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6cb1131d-f786-4f4c-bd32-05ce26bb5411-istiod-ca-cert\") pod \"e2e-worker-single-main-64455b-openshift-default-6477df7745pk7lm\" (UID: \"6cb1131d-f786-4f4c-bd32-05ce26bb5411\") " pod="openshift-ingress/e2e-worker-single-main-64455b-openshift-default-6477df7745pk7lm" Nov 03 14:05:00.682616 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:05:00.682584 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6cb1131d-f786-4f4c-bd32-05ce26bb5411-istio-envoy\") pod \"e2e-worker-single-main-64455b-openshift-default-6477df7745pk7lm\" (UID: \"6cb1131d-f786-4f4c-bd32-05ce26bb5411\") " pod="openshift-ingress/e2e-worker-single-main-64455b-openshift-default-6477df7745pk7lm" Nov 03 14:05:00.692168 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:05:00.692136 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6cb1131d-f786-4f4c-bd32-05ce26bb5411-istio-podinfo\") pod \"e2e-worker-single-main-64455b-openshift-default-6477df7745pk7lm\" (UID: \"6cb1131d-f786-4f4c-bd32-05ce26bb5411\") " pod="openshift-ingress/e2e-worker-single-main-64455b-openshift-default-6477df7745pk7lm" Nov 03 14:05:00.692352 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:05:00.692224 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6cb1131d-f786-4f4c-bd32-05ce26bb5411-istio-token\") pod \"e2e-worker-single-main-64455b-openshift-default-6477df7745pk7lm\" (UID: \"6cb1131d-f786-4f4c-bd32-05ce26bb5411\") " pod="openshift-ingress/e2e-worker-single-main-64455b-openshift-default-6477df7745pk7lm" Nov 03 14:05:00.692436 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:05:00.692404 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9dwh\" (UniqueName: \"kubernetes.io/projected/6cb1131d-f786-4f4c-bd32-05ce26bb5411-kube-api-access-p9dwh\") pod \"e2e-worker-single-main-64455b-openshift-default-6477df7745pk7lm\" (UID: \"6cb1131d-f786-4f4c-bd32-05ce26bb5411\") " pod="openshift-ingress/e2e-worker-single-main-64455b-openshift-default-6477df7745pk7lm" Nov 03 14:05:00.789083 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:05:00.788992 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-64455b-openshift-default-6477df7745pk7lm" Nov 03 14:05:01.071592 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:05:01.071563 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-64455b-openshift-default-6477df7745pk7lm"] Nov 03 14:05:01.073307 ip-10-0-131-36 kubenswrapper[5024]: W1103 14:05:01.073263 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6cb1131d_f786_4f4c_bd32_05ce26bb5411.slice/crio-25bb6828c8da09aec14fa7f2ddf7da3f486f2007e9990520c1e2b82806d761a1 WatchSource:0}: Error finding container 25bb6828c8da09aec14fa7f2ddf7da3f486f2007e9990520c1e2b82806d761a1: Status 404 returned error can't find the container with id 25bb6828c8da09aec14fa7f2ddf7da3f486f2007e9990520c1e2b82806d761a1 Nov 03 14:05:01.075899 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:05:01.075864 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 14:05:01.076007 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:05:01.075927 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 14:05:01.076007 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:05:01.075960 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 14:05:01.259259 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:05:01.259218 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-64455b-openshift-default-6477df7745pk7lm" event={"ID":"6cb1131d-f786-4f4c-bd32-05ce26bb5411","Type":"ContainerStarted","Data":"b539d0ad906549635c01bb1249e3a8c1214abb45868746e0bd50ac32933117ac"} Nov 03 14:05:01.259481 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:05:01.259318 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-64455b-openshift-default-6477df7745pk7lm" event={"ID":"6cb1131d-f786-4f4c-bd32-05ce26bb5411","Type":"ContainerStarted","Data":"25bb6828c8da09aec14fa7f2ddf7da3f486f2007e9990520c1e2b82806d761a1"} Nov 03 14:05:01.285584 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:05:01.285513 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-64455b-openshift-default-6477df7745pk7lm" podStartSLOduration=1.285494721 podStartE2EDuration="1.285494721s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:05:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:05:01.283187756 +0000 UTC m=+3381.291379035" watchObservedRunningTime="2026-11-03 14:05:01.285494721 +0000 UTC m=+3381.293685999" Nov 03 14:05:01.790061 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:05:01.790025 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-64455b-openshift-default-6477df7745pk7lm" Nov 03 14:05:02.795812 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:05:02.795776 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-64455b-openshift-default-6477df7745pk7lm" Nov 03 14:05:03.274038 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:05:03.274005 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-64455b-openshift-default-6477df7745pk7lm" Nov 03 14:05:03.275273 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:05:03.275239 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-64455b-openshift-default-6477df7745pk7lm" Nov 03 14:05:04.673331 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:05:04.673277 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-64455b-6cf489d56b-dmww9"] Nov 03 14:05:04.677957 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:05:04.677928 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-64455b-6cf489d56b-dmww9" Nov 03 14:05:04.687983 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:05:04.687951 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 14:05:04.688139 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:05:04.688084 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 03 14:05:04.688139 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:05:04.687954 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-64455b-serving-cert\"" Nov 03 14:05:04.688267 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:05:04.688204 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-p5qd5\"" Nov 03 14:05:04.688467 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:05:04.688448 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 14:05:04.689906 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:05:04.689879 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-64455b-6cf489d56b-dmww9"] Nov 03 14:05:04.827525 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:05:04.827484 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xr778\" (UniqueName: \"kubernetes.io/projected/3d34cb7d-f2f1-44a6-8144-9d576c61e465-kube-api-access-xr778\") pod \"maas-api-e2e-worker-single-main-64455b-6cf489d56b-dmww9\" (UID: \"3d34cb7d-f2f1-44a6-8144-9d576c61e465\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-64455b-6cf489d56b-dmww9" Nov 03 14:05:04.827689 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:05:04.827552 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3d34cb7d-f2f1-44a6-8144-9d576c61e465-metrics-certs\") pod \"maas-api-e2e-worker-single-main-64455b-6cf489d56b-dmww9\" (UID: \"3d34cb7d-f2f1-44a6-8144-9d576c61e465\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-64455b-6cf489d56b-dmww9" Nov 03 14:05:04.827689 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:05:04.827575 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3d34cb7d-f2f1-44a6-8144-9d576c61e465-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-64455b-6cf489d56b-dmww9\" (UID: \"3d34cb7d-f2f1-44a6-8144-9d576c61e465\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-64455b-6cf489d56b-dmww9" Nov 03 14:05:04.827802 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:05:04.827727 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d34cb7d-f2f1-44a6-8144-9d576c61e465-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-64455b-6cf489d56b-dmww9\" (UID: \"3d34cb7d-f2f1-44a6-8144-9d576c61e465\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-64455b-6cf489d56b-dmww9" Nov 03 14:05:04.851593 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:05:04.851554 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-64455b-6d498qmg7d"] Nov 03 14:05:04.859756 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:05:04.859712 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-64455b-6d498qmg7d" Nov 03 14:05:04.867251 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:05:04.867216 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-64455b-6d498qmg7d"] Nov 03 14:05:04.870907 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:05:04.870869 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-64455b\"" Nov 03 14:05:04.871072 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:05:04.870983 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-abd4ece3-dockercfg-hrr74\"" Nov 03 14:05:04.892692 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:05:04.892660 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-64455b-b6c5ff7c5f7stz"] Nov 03 14:05:04.897166 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:05:04.897142 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-64455b-b6c5ff7c5f7stz" Nov 03 14:05:04.904949 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:05:04.904922 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-64455b-b6c5ff7c5f7stz"] Nov 03 14:05:04.928920 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:05:04.928846 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d34cb7d-f2f1-44a6-8144-9d576c61e465-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-64455b-6cf489d56b-dmww9\" (UID: \"3d34cb7d-f2f1-44a6-8144-9d576c61e465\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-64455b-6cf489d56b-dmww9" Nov 03 14:05:04.928920 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:05:04.928902 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xr778\" (UniqueName: \"kubernetes.io/projected/3d34cb7d-f2f1-44a6-8144-9d576c61e465-kube-api-access-xr778\") pod \"maas-api-e2e-worker-single-main-64455b-6cf489d56b-dmww9\" (UID: \"3d34cb7d-f2f1-44a6-8144-9d576c61e465\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-64455b-6cf489d56b-dmww9" Nov 03 14:05:04.929136 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:05:04.928944 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3d34cb7d-f2f1-44a6-8144-9d576c61e465-metrics-certs\") pod \"maas-api-e2e-worker-single-main-64455b-6cf489d56b-dmww9\" (UID: \"3d34cb7d-f2f1-44a6-8144-9d576c61e465\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-64455b-6cf489d56b-dmww9" Nov 03 14:05:04.929136 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:05:04.928968 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3d34cb7d-f2f1-44a6-8144-9d576c61e465-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-64455b-6cf489d56b-dmww9\" (UID: \"3d34cb7d-f2f1-44a6-8144-9d576c61e465\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-64455b-6cf489d56b-dmww9" Nov 03 14:05:04.929719 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:05:04.929667 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d34cb7d-f2f1-44a6-8144-9d576c61e465-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-64455b-6cf489d56b-dmww9\" (UID: \"3d34cb7d-f2f1-44a6-8144-9d576c61e465\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-64455b-6cf489d56b-dmww9" Nov 03 14:05:04.931598 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:05:04.931573 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3d34cb7d-f2f1-44a6-8144-9d576c61e465-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-64455b-6cf489d56b-dmww9\" (UID: \"3d34cb7d-f2f1-44a6-8144-9d576c61e465\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-64455b-6cf489d56b-dmww9" Nov 03 14:05:04.931817 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:05:04.931787 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3d34cb7d-f2f1-44a6-8144-9d576c61e465-metrics-certs\") pod \"maas-api-e2e-worker-single-main-64455b-6cf489d56b-dmww9\" (UID: \"3d34cb7d-f2f1-44a6-8144-9d576c61e465\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-64455b-6cf489d56b-dmww9" Nov 03 14:05:04.940573 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:05:04.940542 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xr778\" (UniqueName: \"kubernetes.io/projected/3d34cb7d-f2f1-44a6-8144-9d576c61e465-kube-api-access-xr778\") pod \"maas-api-e2e-worker-single-main-64455b-6cf489d56b-dmww9\" (UID: \"3d34cb7d-f2f1-44a6-8144-9d576c61e465\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-64455b-6cf489d56b-dmww9" Nov 03 14:05:05.000802 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:05:05.000764 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-64455b-6cf489d56b-dmww9" Nov 03 14:05:05.030637 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:05:05.030595 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gzrvv\" (UniqueName: \"kubernetes.io/projected/620438d0-7fcc-4b0f-b93d-779691e3f820-kube-api-access-gzrvv\") pod \"payload-processing-e2e-worker-single-main-64455b-b6c5ff7c5f7stz\" (UID: \"620438d0-7fcc-4b0f-b93d-779691e3f820\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-64455b-b6c5ff7c5f7stz" Nov 03 14:05:05.030804 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:05:05.030669 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/620438d0-7fcc-4b0f-b93d-779691e3f820-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-64455b-b6c5ff7c5f7stz\" (UID: \"620438d0-7fcc-4b0f-b93d-779691e3f820\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-64455b-b6c5ff7c5f7stz" Nov 03 14:05:05.030900 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:05:05.030868 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af66fe5d-0c58-4bb2-9089-24d85d371ab1-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-64455b-6d498qmg7d\" (UID: \"af66fe5d-0c58-4bb2-9089-24d85d371ab1\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-64455b-6d498qmg7d" Nov 03 14:05:05.031020 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:05:05.031000 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gcblc\" (UniqueName: \"kubernetes.io/projected/af66fe5d-0c58-4bb2-9089-24d85d371ab1-kube-api-access-gcblc\") pod \"payload-pre-processing-e2e-worker-single-main-64455b-6d498qmg7d\" (UID: \"af66fe5d-0c58-4bb2-9089-24d85d371ab1\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-64455b-6d498qmg7d" Nov 03 14:05:05.131806 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:05:05.131764 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af66fe5d-0c58-4bb2-9089-24d85d371ab1-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-64455b-6d498qmg7d\" (UID: \"af66fe5d-0c58-4bb2-9089-24d85d371ab1\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-64455b-6d498qmg7d" Nov 03 14:05:05.131968 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:05:05.131824 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gcblc\" (UniqueName: \"kubernetes.io/projected/af66fe5d-0c58-4bb2-9089-24d85d371ab1-kube-api-access-gcblc\") pod \"payload-pre-processing-e2e-worker-single-main-64455b-6d498qmg7d\" (UID: \"af66fe5d-0c58-4bb2-9089-24d85d371ab1\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-64455b-6d498qmg7d" Nov 03 14:05:05.131968 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:05:05.131876 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzrvv\" (UniqueName: \"kubernetes.io/projected/620438d0-7fcc-4b0f-b93d-779691e3f820-kube-api-access-gzrvv\") pod \"payload-processing-e2e-worker-single-main-64455b-b6c5ff7c5f7stz\" (UID: \"620438d0-7fcc-4b0f-b93d-779691e3f820\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-64455b-b6c5ff7c5f7stz" Nov 03 14:05:05.132050 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:05:05.131991 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/620438d0-7fcc-4b0f-b93d-779691e3f820-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-64455b-b6c5ff7c5f7stz\" (UID: \"620438d0-7fcc-4b0f-b93d-779691e3f820\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-64455b-b6c5ff7c5f7stz" Nov 03 14:05:05.132673 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:05:05.132649 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af66fe5d-0c58-4bb2-9089-24d85d371ab1-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-64455b-6d498qmg7d\" (UID: \"af66fe5d-0c58-4bb2-9089-24d85d371ab1\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-64455b-6d498qmg7d" Nov 03 14:05:05.132797 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:05:05.132782 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/620438d0-7fcc-4b0f-b93d-779691e3f820-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-64455b-b6c5ff7c5f7stz\" (UID: \"620438d0-7fcc-4b0f-b93d-779691e3f820\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-64455b-b6c5ff7c5f7stz" Nov 03 14:05:05.142153 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:05:05.142117 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gcblc\" (UniqueName: \"kubernetes.io/projected/af66fe5d-0c58-4bb2-9089-24d85d371ab1-kube-api-access-gcblc\") pod \"payload-pre-processing-e2e-worker-single-main-64455b-6d498qmg7d\" (UID: \"af66fe5d-0c58-4bb2-9089-24d85d371ab1\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-64455b-6d498qmg7d" Nov 03 14:05:05.142280 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:05:05.142179 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gzrvv\" (UniqueName: \"kubernetes.io/projected/620438d0-7fcc-4b0f-b93d-779691e3f820-kube-api-access-gzrvv\") pod \"payload-processing-e2e-worker-single-main-64455b-b6c5ff7c5f7stz\" (UID: \"620438d0-7fcc-4b0f-b93d-779691e3f820\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-64455b-b6c5ff7c5f7stz" Nov 03 14:05:05.178581 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:05:05.178554 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-64455b-6d498qmg7d" Nov 03 14:05:05.215134 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:05:05.215019 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-64455b-b6c5ff7c5f7stz" Nov 03 14:05:05.304584 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:05:05.304327 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-64455b-6cf489d56b-dmww9"] Nov 03 14:05:05.308343 ip-10-0-131-36 kubenswrapper[5024]: W1103 14:05:05.308241 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d34cb7d_f2f1_44a6_8144_9d576c61e465.slice/crio-61c2f7fdca510298e6b907c5338c57397f8cf706f05c73c57d1d57110eac509e WatchSource:0}: Error finding container 61c2f7fdca510298e6b907c5338c57397f8cf706f05c73c57d1d57110eac509e: Status 404 returned error can't find the container with id 61c2f7fdca510298e6b907c5338c57397f8cf706f05c73c57d1d57110eac509e Nov 03 14:05:05.310033 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:05:05.310009 5024 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 14:05:05.464076 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:05:05.464052 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-64455b-6d498qmg7d"] Nov 03 14:05:05.465352 ip-10-0-131-36 kubenswrapper[5024]: W1103 14:05:05.465322 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf66fe5d_0c58_4bb2_9089_24d85d371ab1.slice/crio-9415c95542d4f7e44b9a71c258d83b611166d7eb73ec211f080dfb1c2cc04e47 WatchSource:0}: Error finding container 9415c95542d4f7e44b9a71c258d83b611166d7eb73ec211f080dfb1c2cc04e47: Status 404 returned error can't find the container with id 9415c95542d4f7e44b9a71c258d83b611166d7eb73ec211f080dfb1c2cc04e47 Nov 03 14:05:05.519758 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:05:05.519730 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-64455b-b6c5ff7c5f7stz"] Nov 03 14:05:05.523553 ip-10-0-131-36 kubenswrapper[5024]: W1103 14:05:05.523520 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod620438d0_7fcc_4b0f_b93d_779691e3f820.slice/crio-68b444941d7a7d728e81ec52a38318174fd47759fbeb7658b7860de03786c876 WatchSource:0}: Error finding container 68b444941d7a7d728e81ec52a38318174fd47759fbeb7658b7860de03786c876: Status 404 returned error can't find the container with id 68b444941d7a7d728e81ec52a38318174fd47759fbeb7658b7860de03786c876 Nov 03 14:05:06.303558 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:05:06.303458 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-64455b-6cf489d56b-dmww9" event={"ID":"3d34cb7d-f2f1-44a6-8144-9d576c61e465","Type":"ContainerStarted","Data":"8e7871c77c594b30950dca656b8e498155bd556d7361cbe2040c4251251132c2"} Nov 03 14:05:06.303558 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:05:06.303527 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-64455b-6cf489d56b-dmww9" event={"ID":"3d34cb7d-f2f1-44a6-8144-9d576c61e465","Type":"ContainerStarted","Data":"61c2f7fdca510298e6b907c5338c57397f8cf706f05c73c57d1d57110eac509e"} Nov 03 14:05:06.303981 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:05:06.303561 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-64455b-6cf489d56b-dmww9" Nov 03 14:05:06.305371 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:05:06.305342 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-64455b-b6c5ff7c5f7stz" event={"ID":"620438d0-7fcc-4b0f-b93d-779691e3f820","Type":"ContainerStarted","Data":"75deb7bf408b8f5bc0cb9902e09ac22e0e2e7266212a00fcca9349e4ceaa3e4e"} Nov 03 14:05:06.305514 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:05:06.305399 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-64455b-b6c5ff7c5f7stz" Nov 03 14:05:06.305514 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:05:06.305423 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-64455b-b6c5ff7c5f7stz" event={"ID":"620438d0-7fcc-4b0f-b93d-779691e3f820","Type":"ContainerStarted","Data":"68b444941d7a7d728e81ec52a38318174fd47759fbeb7658b7860de03786c876"} Nov 03 14:05:06.306989 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:05:06.306965 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-64455b-6d498qmg7d" event={"ID":"af66fe5d-0c58-4bb2-9089-24d85d371ab1","Type":"ContainerStarted","Data":"c36e223da678bd036e9b56b01d111765ca83c070f9d60d116988662884626b0b"} Nov 03 14:05:06.307085 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:05:06.307015 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-64455b-6d498qmg7d" Nov 03 14:05:06.307085 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:05:06.307033 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-64455b-6d498qmg7d" event={"ID":"af66fe5d-0c58-4bb2-9089-24d85d371ab1","Type":"ContainerStarted","Data":"9415c95542d4f7e44b9a71c258d83b611166d7eb73ec211f080dfb1c2cc04e47"} Nov 03 14:05:06.337639 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:05:06.337581 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-64455b-6cf489d56b-dmww9" podStartSLOduration=1.781089126 podStartE2EDuration="2.337567036s" totalImagesPullingTime="556.47791ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:05:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:05:06.335245893 +0000 UTC m=+3386.343437169" watchObservedRunningTime="2026-11-03 14:05:06.337567036 +0000 UTC m=+3386.345758358" Nov 03 14:05:06.362689 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:05:06.362631 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-64455b-b6c5ff7c5f7stz" podStartSLOduration=2.362616441 podStartE2EDuration="2.362616441s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:05:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:05:06.360328169 +0000 UTC m=+3386.368519447" watchObservedRunningTime="2026-11-03 14:05:06.362616441 +0000 UTC m=+3386.370807717" Nov 03 14:05:06.383316 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:05:06.383255 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-64455b-6d498qmg7d" podStartSLOduration=2.383241432 podStartE2EDuration="2.383241432s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:05:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:05:06.379914394 +0000 UTC m=+3386.388105672" watchObservedRunningTime="2026-11-03 14:05:06.383241432 +0000 UTC m=+3386.391432708" Nov 03 14:05:09.521948 ip-10-0-131-36 kubenswrapper[5024]: E1103 14:05:09.521898 5024 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d7mwz" containerName="extractor" Nov 03 14:05:12.318555 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:05:12.318518 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-64455b-6cf489d56b-dmww9" Nov 03 14:05:17.316118 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:05:17.316038 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-64455b-b6c5ff7c5f7stz" Nov 03 14:05:17.316118 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:05:17.316111 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-64455b-6d498qmg7d" Nov 03 14:05:17.376666 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:05:17.376627 5024 generic.go:356] "Generic (PLEG): container finished" podID="6cb1131d-f786-4f4c-bd32-05ce26bb5411" containerID="b539d0ad906549635c01bb1249e3a8c1214abb45868746e0bd50ac32933117ac" exitCode=0 Nov 03 14:05:17.376936 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:05:17.376713 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-64455b-openshift-default-6477df7745pk7lm" event={"ID":"6cb1131d-f786-4f4c-bd32-05ce26bb5411","Type":"ContainerDied","Data":"b539d0ad906549635c01bb1249e3a8c1214abb45868746e0bd50ac32933117ac"} Nov 03 14:05:17.377295 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:05:17.377261 5024 scope.go:122] "RemoveContainer" containerID="b539d0ad906549635c01bb1249e3a8c1214abb45868746e0bd50ac32933117ac" Nov 03 14:05:17.378788 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:05:17.378755 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 14:05:17.378890 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:05:17.378844 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 14:05:17.378954 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:05:17.378898 5024 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 14:05:18.287310 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:05:18.279396 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-64455b-openshift-default-6477df7745pk7lm" Nov 03 14:05:18.402399 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:05:18.402359 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-64455b-openshift-default-6477df7745pk7lm" event={"ID":"6cb1131d-f786-4f4c-bd32-05ce26bb5411","Type":"ContainerStarted","Data":"224c6b177b16d3d9bc95a8e18d4e6808e80272fc38d9dd42bfd0c81cf916e9a4"} Nov 03 14:05:18.794308 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:05:18.790675 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-64455b-openshift-default-6477df7745pk7lm" Nov 03 14:05:18.794308 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:05:18.793696 5024 patch_prober.go:29] interesting pod/e2e-worker-single-main-64455b-openshift-default-6477df7745pk7lm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.234:15021/healthz/ready\": dial tcp 10.132.0.234:15021: connect: connection refused" start-of-body= Nov 03 14:05:18.794308 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:05:18.793754 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-64455b-openshift-default-6477df7745pk7lm" podUID="6cb1131d-f786-4f4c-bd32-05ce26bb5411" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.234:15021/healthz/ready\": dial tcp 10.132.0.234:15021: connect: connection refused" Nov 03 14:05:19.795776 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:05:19.795731 5024 patch_prober.go:29] interesting pod/e2e-worker-single-main-64455b-openshift-default-6477df7745pk7lm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.234:15021/healthz/ready\": dial tcp 10.132.0.234:15021: connect: connection refused" start-of-body= Nov 03 14:05:19.796226 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:05:19.795816 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-64455b-openshift-default-6477df7745pk7lm" podUID="6cb1131d-f786-4f4c-bd32-05ce26bb5411" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.234:15021/healthz/ready\": dial tcp 10.132.0.234:15021: connect: connection refused" Nov 03 14:05:20.564971 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:05:20.564936 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-64455b/simulator-single-main-64455b-kserve-bf6dbbbc6-nr7mf"] Nov 03 14:05:20.567857 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:05:20.567833 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-64455b/simulator-single-main-64455b-kserve-bf6dbbbc6-nr7mf" Nov 03 14:05:20.607335 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:05:20.603596 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-64455b\"/\"openshift-service-ca.crt\"" Nov 03 14:05:20.608447 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:05:20.608143 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-64455b\"/\"default-dockercfg-2t6ng\"" Nov 03 14:05:20.609847 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:05:20.609192 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-64455b/simulator-single-main-64455b-kserve-bf6dbbbc6-nr7mf"] Nov 03 14:05:20.610033 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:05:20.609492 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-64455b\"/\"simulator-single-main-64455b-kserve-self-signed-certs\"" Nov 03 14:05:20.610127 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:05:20.609789 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-64455b\"/\"kube-root-ca.crt\"" Nov 03 14:05:20.626160 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:05:20.626126 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a526440-5821-45b2-8bae-17c404668fc1-model-cache\") pod \"simulator-single-main-64455b-kserve-bf6dbbbc6-nr7mf\" (UID: \"2a526440-5821-45b2-8bae-17c404668fc1\") " pod="e2e-models-e2e-worker-single-main-64455b/simulator-single-main-64455b-kserve-bf6dbbbc6-nr7mf" Nov 03 14:05:20.626542 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:05:20.626513 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2jnzm\" (UniqueName: \"kubernetes.io/projected/2a526440-5821-45b2-8bae-17c404668fc1-kube-api-access-2jnzm\") pod \"simulator-single-main-64455b-kserve-bf6dbbbc6-nr7mf\" (UID: \"2a526440-5821-45b2-8bae-17c404668fc1\") " pod="e2e-models-e2e-worker-single-main-64455b/simulator-single-main-64455b-kserve-bf6dbbbc6-nr7mf" Nov 03 14:05:20.626680 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:05:20.626556 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a526440-5821-45b2-8bae-17c404668fc1-home\") pod \"simulator-single-main-64455b-kserve-bf6dbbbc6-nr7mf\" (UID: \"2a526440-5821-45b2-8bae-17c404668fc1\") " pod="e2e-models-e2e-worker-single-main-64455b/simulator-single-main-64455b-kserve-bf6dbbbc6-nr7mf" Nov 03 14:05:20.626680 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:05:20.626582 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a526440-5821-45b2-8bae-17c404668fc1-dshm\") pod \"simulator-single-main-64455b-kserve-bf6dbbbc6-nr7mf\" (UID: \"2a526440-5821-45b2-8bae-17c404668fc1\") " pod="e2e-models-e2e-worker-single-main-64455b/simulator-single-main-64455b-kserve-bf6dbbbc6-nr7mf" Nov 03 14:05:20.626680 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:05:20.626652 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a526440-5821-45b2-8bae-17c404668fc1-tmp-dir\") pod \"simulator-single-main-64455b-kserve-bf6dbbbc6-nr7mf\" (UID: \"2a526440-5821-45b2-8bae-17c404668fc1\") " pod="e2e-models-e2e-worker-single-main-64455b/simulator-single-main-64455b-kserve-bf6dbbbc6-nr7mf" Nov 03 14:05:20.626880 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:05:20.626694 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a526440-5821-45b2-8bae-17c404668fc1-tls-certs\") pod \"simulator-single-main-64455b-kserve-bf6dbbbc6-nr7mf\" (UID: \"2a526440-5821-45b2-8bae-17c404668fc1\") " pod="e2e-models-e2e-worker-single-main-64455b/simulator-single-main-64455b-kserve-bf6dbbbc6-nr7mf" Nov 03 14:05:20.727618 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:05:20.727568 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a526440-5821-45b2-8bae-17c404668fc1-tls-certs\") pod \"simulator-single-main-64455b-kserve-bf6dbbbc6-nr7mf\" (UID: \"2a526440-5821-45b2-8bae-17c404668fc1\") " pod="e2e-models-e2e-worker-single-main-64455b/simulator-single-main-64455b-kserve-bf6dbbbc6-nr7mf" Nov 03 14:05:20.727831 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:05:20.727644 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a526440-5821-45b2-8bae-17c404668fc1-model-cache\") pod \"simulator-single-main-64455b-kserve-bf6dbbbc6-nr7mf\" (UID: \"2a526440-5821-45b2-8bae-17c404668fc1\") " pod="e2e-models-e2e-worker-single-main-64455b/simulator-single-main-64455b-kserve-bf6dbbbc6-nr7mf" Nov 03 14:05:20.727831 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:05:20.727737 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jnzm\" (UniqueName: \"kubernetes.io/projected/2a526440-5821-45b2-8bae-17c404668fc1-kube-api-access-2jnzm\") pod \"simulator-single-main-64455b-kserve-bf6dbbbc6-nr7mf\" (UID: \"2a526440-5821-45b2-8bae-17c404668fc1\") " pod="e2e-models-e2e-worker-single-main-64455b/simulator-single-main-64455b-kserve-bf6dbbbc6-nr7mf" Nov 03 14:05:20.727831 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:05:20.727769 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a526440-5821-45b2-8bae-17c404668fc1-home\") pod \"simulator-single-main-64455b-kserve-bf6dbbbc6-nr7mf\" (UID: \"2a526440-5821-45b2-8bae-17c404668fc1\") " pod="e2e-models-e2e-worker-single-main-64455b/simulator-single-main-64455b-kserve-bf6dbbbc6-nr7mf" Nov 03 14:05:20.727831 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:05:20.727795 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a526440-5821-45b2-8bae-17c404668fc1-dshm\") pod \"simulator-single-main-64455b-kserve-bf6dbbbc6-nr7mf\" (UID: \"2a526440-5821-45b2-8bae-17c404668fc1\") " pod="e2e-models-e2e-worker-single-main-64455b/simulator-single-main-64455b-kserve-bf6dbbbc6-nr7mf" Nov 03 14:05:20.728078 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:05:20.727851 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a526440-5821-45b2-8bae-17c404668fc1-tmp-dir\") pod \"simulator-single-main-64455b-kserve-bf6dbbbc6-nr7mf\" (UID: \"2a526440-5821-45b2-8bae-17c404668fc1\") " pod="e2e-models-e2e-worker-single-main-64455b/simulator-single-main-64455b-kserve-bf6dbbbc6-nr7mf" Nov 03 14:05:20.728364 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:05:20.728203 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a526440-5821-45b2-8bae-17c404668fc1-tmp-dir\") pod \"simulator-single-main-64455b-kserve-bf6dbbbc6-nr7mf\" (UID: \"2a526440-5821-45b2-8bae-17c404668fc1\") " pod="e2e-models-e2e-worker-single-main-64455b/simulator-single-main-64455b-kserve-bf6dbbbc6-nr7mf" Nov 03 14:05:20.728616 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:05:20.728589 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a526440-5821-45b2-8bae-17c404668fc1-home\") pod \"simulator-single-main-64455b-kserve-bf6dbbbc6-nr7mf\" (UID: \"2a526440-5821-45b2-8bae-17c404668fc1\") " pod="e2e-models-e2e-worker-single-main-64455b/simulator-single-main-64455b-kserve-bf6dbbbc6-nr7mf" Nov 03 14:05:20.728710 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:05:20.728655 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a526440-5821-45b2-8bae-17c404668fc1-model-cache\") pod \"simulator-single-main-64455b-kserve-bf6dbbbc6-nr7mf\" (UID: \"2a526440-5821-45b2-8bae-17c404668fc1\") " pod="e2e-models-e2e-worker-single-main-64455b/simulator-single-main-64455b-kserve-bf6dbbbc6-nr7mf" Nov 03 14:05:20.732049 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:05:20.731258 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a526440-5821-45b2-8bae-17c404668fc1-dshm\") pod \"simulator-single-main-64455b-kserve-bf6dbbbc6-nr7mf\" (UID: \"2a526440-5821-45b2-8bae-17c404668fc1\") " pod="e2e-models-e2e-worker-single-main-64455b/simulator-single-main-64455b-kserve-bf6dbbbc6-nr7mf" Nov 03 14:05:20.732049 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:05:20.731496 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a526440-5821-45b2-8bae-17c404668fc1-tls-certs\") pod \"simulator-single-main-64455b-kserve-bf6dbbbc6-nr7mf\" (UID: \"2a526440-5821-45b2-8bae-17c404668fc1\") " pod="e2e-models-e2e-worker-single-main-64455b/simulator-single-main-64455b-kserve-bf6dbbbc6-nr7mf" Nov 03 14:05:20.740840 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:05:20.740751 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2jnzm\" (UniqueName: \"kubernetes.io/projected/2a526440-5821-45b2-8bae-17c404668fc1-kube-api-access-2jnzm\") pod \"simulator-single-main-64455b-kserve-bf6dbbbc6-nr7mf\" (UID: \"2a526440-5821-45b2-8bae-17c404668fc1\") " pod="e2e-models-e2e-worker-single-main-64455b/simulator-single-main-64455b-kserve-bf6dbbbc6-nr7mf" Nov 03 14:05:20.794739 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:05:20.794696 5024 patch_prober.go:29] interesting pod/e2e-worker-single-main-64455b-openshift-default-6477df7745pk7lm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.234:15021/healthz/ready\": dial tcp 10.132.0.234:15021: connect: connection refused" start-of-body= Nov 03 14:05:20.794998 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:05:20.794968 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-64455b-openshift-default-6477df7745pk7lm" podUID="6cb1131d-f786-4f4c-bd32-05ce26bb5411" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.234:15021/healthz/ready\": dial tcp 10.132.0.234:15021: connect: connection refused" Nov 03 14:05:20.917377 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:05:20.917274 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-64455b/simulator-single-main-64455b-kserve-bf6dbbbc6-nr7mf" Nov 03 14:05:21.239079 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:05:21.239034 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-64455b/simulator-single-main-64455b-kserve-bf6dbbbc6-nr7mf"] Nov 03 14:05:21.239722 ip-10-0-131-36 kubenswrapper[5024]: W1103 14:05:21.239694 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2a526440_5821_45b2_8bae_17c404668fc1.slice/crio-d5654fe9b57cf8b13c36fd8035f0d6ce7e8b18f428ad0035e54066407d6ae02f WatchSource:0}: Error finding container d5654fe9b57cf8b13c36fd8035f0d6ce7e8b18f428ad0035e54066407d6ae02f: Status 404 returned error can't find the container with id d5654fe9b57cf8b13c36fd8035f0d6ce7e8b18f428ad0035e54066407d6ae02f Nov 03 14:05:21.425160 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:05:21.425119 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-64455b/simulator-single-main-64455b-kserve-bf6dbbbc6-nr7mf" event={"ID":"2a526440-5821-45b2-8bae-17c404668fc1","Type":"ContainerStarted","Data":"fb59ddba63943bbedcbd118c21404bf6cc4d9a56f352ac5396b1b99714b05d6b"} Nov 03 14:05:21.425417 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:05:21.425173 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-64455b/simulator-single-main-64455b-kserve-bf6dbbbc6-nr7mf" event={"ID":"2a526440-5821-45b2-8bae-17c404668fc1","Type":"ContainerStarted","Data":"d5654fe9b57cf8b13c36fd8035f0d6ce7e8b18f428ad0035e54066407d6ae02f"} Nov 03 14:05:21.445248 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:05:21.445195 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-64455b/simulator-single-main-64455b-kserve-bf6dbbbc6-nr7mf" podStartSLOduration=1.445179343 podStartE2EDuration="1.445179343s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:05:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:05:21.444236234 +0000 UTC m=+3401.452427511" watchObservedRunningTime="2026-11-03 14:05:21.445179343 +0000 UTC m=+3401.453370620" Nov 03 14:05:21.797313 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:05:21.792859 5024 patch_prober.go:29] interesting pod/e2e-worker-single-main-64455b-openshift-default-6477df7745pk7lm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.234:15021/healthz/ready\": dial tcp 10.132.0.234:15021: connect: connection refused" start-of-body= Nov 03 14:05:21.797313 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:05:21.792940 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-64455b-openshift-default-6477df7745pk7lm" podUID="6cb1131d-f786-4f4c-bd32-05ce26bb5411" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.234:15021/healthz/ready\": dial tcp 10.132.0.234:15021: connect: connection refused" Nov 03 14:05:21.918413 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:05:21.918349 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-64455b/simulator-single-main-64455b-kserve-bf6dbbbc6-nr7mf" Nov 03 14:05:22.793308 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:05:22.792721 5024 patch_prober.go:29] interesting pod/e2e-worker-single-main-64455b-openshift-default-6477df7745pk7lm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.234:15021/healthz/ready\": dial tcp 10.132.0.234:15021: connect: connection refused" start-of-body= Nov 03 14:05:22.793308 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:05:22.792788 5024 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-64455b-openshift-default-6477df7745pk7lm" podUID="6cb1131d-f786-4f4c-bd32-05ce26bb5411" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.234:15021/healthz/ready\": dial tcp 10.132.0.234:15021: connect: connection refused" Nov 03 14:05:24.030195 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:05:24.030155 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-64455b-openshift-default-6477df7745pk7lm" Nov 03 14:05:24.030732 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:05:24.030698 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-64455b-openshift-default-6477df7745pk7lm" Nov 03 14:05:24.031875 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:05:24.031849 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-64455b-openshift-default-6477df7745pk7lm" Nov 03 14:05:30.917482 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:05:30.917439 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-64455b/simulator-single-main-64455b-kserve-bf6dbbbc6-nr7mf" Nov 03 14:05:30.930279 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:05:30.930249 5024 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-64455b/simulator-single-main-64455b-kserve-bf6dbbbc6-nr7mf" Nov 03 14:05:31.497223 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:05:31.497189 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-64455b/simulator-single-main-64455b-kserve-bf6dbbbc6-nr7mf" Nov 03 14:05:39.514447 ip-10-0-131-36 kubenswrapper[5024]: E1103 14:05:39.514402 5024 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d7mwz" containerName="extractor" Nov 03 14:06:06.468110 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:06:06.468072 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-64455b/simulator-single-main-64455b-kserve-bf6dbbbc6-nr7mf"] Nov 03 14:06:08.130913 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:06:08.130831 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-64455b-b6c5ff7c5f7stz"] Nov 03 14:06:08.142761 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:06:08.142729 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-64455b-6d498qmg7d"] Nov 03 14:06:09.160495 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:06:09.160454 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-64455b-6cf489d56b-dmww9"] Nov 03 14:06:09.161354 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:06:09.161320 5024 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-64455b-6cf489d56b-dmww9" podUID="3d34cb7d-f2f1-44a6-8144-9d576c61e465" containerName="maas-api" containerID="cri-o://8e7871c77c594b30950dca656b8e498155bd556d7361cbe2040c4251251132c2" gracePeriod=30 Nov 03 14:06:09.462704 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:06:09.462677 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-64455b-6cf489d56b-dmww9" Nov 03 14:06:09.516438 ip-10-0-131-36 kubenswrapper[5024]: E1103 14:06:09.516397 5024 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d7mwz" containerName="extractor" Nov 03 14:06:09.552875 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:06:09.552846 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d34cb7d-f2f1-44a6-8144-9d576c61e465-kube-api-access-xr778\" (UniqueName: \"kubernetes.io/projected/3d34cb7d-f2f1-44a6-8144-9d576c61e465-kube-api-access-xr778\") pod \"3d34cb7d-f2f1-44a6-8144-9d576c61e465\" (UID: \"3d34cb7d-f2f1-44a6-8144-9d576c61e465\") " Nov 03 14:06:09.553038 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:06:09.552930 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3d34cb7d-f2f1-44a6-8144-9d576c61e465-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3d34cb7d-f2f1-44a6-8144-9d576c61e465-metrics-certs\") pod \"3d34cb7d-f2f1-44a6-8144-9d576c61e465\" (UID: \"3d34cb7d-f2f1-44a6-8144-9d576c61e465\") " Nov 03 14:06:09.553038 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:06:09.553021 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d34cb7d-f2f1-44a6-8144-9d576c61e465-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d34cb7d-f2f1-44a6-8144-9d576c61e465-openshift-service-ca\") pod \"3d34cb7d-f2f1-44a6-8144-9d576c61e465\" (UID: \"3d34cb7d-f2f1-44a6-8144-9d576c61e465\") " Nov 03 14:06:09.553129 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:06:09.553079 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3d34cb7d-f2f1-44a6-8144-9d576c61e465-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3d34cb7d-f2f1-44a6-8144-9d576c61e465-maas-api-tls\") pod \"3d34cb7d-f2f1-44a6-8144-9d576c61e465\" (UID: \"3d34cb7d-f2f1-44a6-8144-9d576c61e465\") " Nov 03 14:06:09.553411 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:06:09.553384 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d34cb7d-f2f1-44a6-8144-9d576c61e465-openshift-service-ca" pod "3d34cb7d-f2f1-44a6-8144-9d576c61e465" (UID: "3d34cb7d-f2f1-44a6-8144-9d576c61e465"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:06:09.555345 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:06:09.555319 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d34cb7d-f2f1-44a6-8144-9d576c61e465-maas-api-tls" pod "3d34cb7d-f2f1-44a6-8144-9d576c61e465" (UID: "3d34cb7d-f2f1-44a6-8144-9d576c61e465"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:06:09.555578 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:06:09.555555 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d34cb7d-f2f1-44a6-8144-9d576c61e465-metrics-certs" pod "3d34cb7d-f2f1-44a6-8144-9d576c61e465" (UID: "3d34cb7d-f2f1-44a6-8144-9d576c61e465"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:06:09.555630 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:06:09.555578 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d34cb7d-f2f1-44a6-8144-9d576c61e465-kube-api-access-xr778" pod "3d34cb7d-f2f1-44a6-8144-9d576c61e465" (UID: "3d34cb7d-f2f1-44a6-8144-9d576c61e465"). InnerVolumeSpecName "kube-api-access-xr778". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:06:09.654147 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:06:09.654107 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xr778\" (UniqueName: \"kubernetes.io/projected/3d34cb7d-f2f1-44a6-8144-9d576c61e465-kube-api-access-xr778\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 14:06:09.654147 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:06:09.654141 5024 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3d34cb7d-f2f1-44a6-8144-9d576c61e465-metrics-certs\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 14:06:09.654147 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:06:09.654151 5024 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d34cb7d-f2f1-44a6-8144-9d576c61e465-openshift-service-ca\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 14:06:09.654423 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:06:09.654160 5024 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3d34cb7d-f2f1-44a6-8144-9d576c61e465-maas-api-tls\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 14:06:09.700997 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:06:09.700900 5024 generic.go:356] "Generic (PLEG): container finished" podID="3d34cb7d-f2f1-44a6-8144-9d576c61e465" containerID="8e7871c77c594b30950dca656b8e498155bd556d7361cbe2040c4251251132c2" exitCode=0 Nov 03 14:06:09.700997 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:06:09.700945 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-64455b-6cf489d56b-dmww9" Nov 03 14:06:09.701199 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:06:09.700992 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-64455b-6cf489d56b-dmww9" event={"ID":"3d34cb7d-f2f1-44a6-8144-9d576c61e465","Type":"ContainerDied","Data":"8e7871c77c594b30950dca656b8e498155bd556d7361cbe2040c4251251132c2"} Nov 03 14:06:09.701199 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:06:09.701043 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-64455b-6cf489d56b-dmww9" event={"ID":"3d34cb7d-f2f1-44a6-8144-9d576c61e465","Type":"ContainerDied","Data":"61c2f7fdca510298e6b907c5338c57397f8cf706f05c73c57d1d57110eac509e"} Nov 03 14:06:09.701199 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:06:09.701065 5024 scope.go:122] "RemoveContainer" containerID="8e7871c77c594b30950dca656b8e498155bd556d7361cbe2040c4251251132c2" Nov 03 14:06:09.712224 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:06:09.712207 5024 scope.go:122] "RemoveContainer" containerID="8e7871c77c594b30950dca656b8e498155bd556d7361cbe2040c4251251132c2" Nov 03 14:06:09.712475 ip-10-0-131-36 kubenswrapper[5024]: E1103 14:06:09.712454 5024 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e7871c77c594b30950dca656b8e498155bd556d7361cbe2040c4251251132c2\": container with ID starting with 8e7871c77c594b30950dca656b8e498155bd556d7361cbe2040c4251251132c2 not found: ID does not exist" containerID="8e7871c77c594b30950dca656b8e498155bd556d7361cbe2040c4251251132c2" Nov 03 14:06:09.712546 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:06:09.712481 5024 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e7871c77c594b30950dca656b8e498155bd556d7361cbe2040c4251251132c2"} err="failed to get container status \"8e7871c77c594b30950dca656b8e498155bd556d7361cbe2040c4251251132c2\": rpc error: code = NotFound desc = could not find container \"8e7871c77c594b30950dca656b8e498155bd556d7361cbe2040c4251251132c2\": container with ID starting with 8e7871c77c594b30950dca656b8e498155bd556d7361cbe2040c4251251132c2 not found: ID does not exist" Nov 03 14:06:09.731376 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:06:09.731351 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-64455b-6cf489d56b-dmww9"] Nov 03 14:06:09.733736 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:06:09.733711 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-64455b-6cf489d56b-dmww9"] Nov 03 14:06:10.108081 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:06:10.108045 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d34cb7d-f2f1-44a6-8144-9d576c61e465" path="/var/lib/kubelet/pods/3d34cb7d-f2f1-44a6-8144-9d576c61e465/volumes" Nov 03 14:06:13.131859 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:06:13.131808 5024 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-64455b-b6c5ff7c5f7stz" podUID="620438d0-7fcc-4b0f-b93d-779691e3f820" containerName="payload-processing" containerID="cri-o://75deb7bf408b8f5bc0cb9902e09ac22e0e2e7266212a00fcca9349e4ceaa3e4e" gracePeriod=25 Nov 03 14:06:13.143358 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:06:13.143319 5024 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-64455b-6d498qmg7d" podUID="af66fe5d-0c58-4bb2-9089-24d85d371ab1" containerName="payload-pre-processing" containerID="cri-o://c36e223da678bd036e9b56b01d111765ca83c070f9d60d116988662884626b0b" gracePeriod=25 Nov 03 14:06:17.314729 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:06:17.314684 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-64455b-b6c5ff7c5f7stz" podUID="620438d0-7fcc-4b0f-b93d-779691e3f820" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 14:06:17.315154 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:06:17.314690 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-64455b-6d498qmg7d" podUID="af66fe5d-0c58-4bb2-9089-24d85d371ab1" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 14:06:20.024029 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:06:20.023991 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-qxwsd"] Nov 03 14:06:20.026711 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:06:20.026688 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-qxwsd"] Nov 03 14:06:20.109343 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:06:20.109312 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d5f05b0-68e0-486a-babd-20c35127c2e5" path="/var/lib/kubelet/pods/8d5f05b0-68e0-486a-babd-20c35127c2e5/volumes" Nov 03 14:06:21.488414 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:06:21.488370 5024 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-64455b/simulator-single-main-64455b-kserve-bf6dbbbc6-nr7mf" podUID="2a526440-5821-45b2-8bae-17c404668fc1" containerName="main" containerID="cri-o://fb59ddba63943bbedcbd118c21404bf6cc4d9a56f352ac5396b1b99714b05d6b" gracePeriod=45 Nov 03 14:06:21.492428 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:06:21.492395 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-single-main-64455b/simulator-single-main-64455b-kserve-bf6dbbbc6-nr7mf" podUID="2a526440-5821-45b2-8bae-17c404668fc1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.238:8000/ready\": EOF" Nov 03 14:06:21.743870 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:06:21.743816 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-64455b/simulator-single-main-64455b-kserve-bf6dbbbc6-nr7mf" Nov 03 14:06:21.775569 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:06:21.775537 5024 generic.go:356] "Generic (PLEG): container finished" podID="2a526440-5821-45b2-8bae-17c404668fc1" containerID="fb59ddba63943bbedcbd118c21404bf6cc4d9a56f352ac5396b1b99714b05d6b" exitCode=0 Nov 03 14:06:21.775782 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:06:21.775590 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-64455b/simulator-single-main-64455b-kserve-bf6dbbbc6-nr7mf" Nov 03 14:06:21.775782 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:06:21.775603 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-64455b/simulator-single-main-64455b-kserve-bf6dbbbc6-nr7mf" event={"ID":"2a526440-5821-45b2-8bae-17c404668fc1","Type":"ContainerDied","Data":"fb59ddba63943bbedcbd118c21404bf6cc4d9a56f352ac5396b1b99714b05d6b"} Nov 03 14:06:21.775782 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:06:21.775636 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-64455b/simulator-single-main-64455b-kserve-bf6dbbbc6-nr7mf" event={"ID":"2a526440-5821-45b2-8bae-17c404668fc1","Type":"ContainerDied","Data":"d5654fe9b57cf8b13c36fd8035f0d6ce7e8b18f428ad0035e54066407d6ae02f"} Nov 03 14:06:21.775782 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:06:21.775655 5024 scope.go:122] "RemoveContainer" containerID="fb59ddba63943bbedcbd118c21404bf6cc4d9a56f352ac5396b1b99714b05d6b" Nov 03 14:06:21.787599 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:06:21.787565 5024 scope.go:122] "RemoveContainer" containerID="fb59ddba63943bbedcbd118c21404bf6cc4d9a56f352ac5396b1b99714b05d6b" Nov 03 14:06:21.787915 ip-10-0-131-36 kubenswrapper[5024]: E1103 14:06:21.787878 5024 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb59ddba63943bbedcbd118c21404bf6cc4d9a56f352ac5396b1b99714b05d6b\": container with ID starting with fb59ddba63943bbedcbd118c21404bf6cc4d9a56f352ac5396b1b99714b05d6b not found: ID does not exist" containerID="fb59ddba63943bbedcbd118c21404bf6cc4d9a56f352ac5396b1b99714b05d6b" Nov 03 14:06:21.787915 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:06:21.787907 5024 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb59ddba63943bbedcbd118c21404bf6cc4d9a56f352ac5396b1b99714b05d6b"} err="failed to get container status \"fb59ddba63943bbedcbd118c21404bf6cc4d9a56f352ac5396b1b99714b05d6b\": rpc error: code = NotFound desc = could not find container \"fb59ddba63943bbedcbd118c21404bf6cc4d9a56f352ac5396b1b99714b05d6b\": container with ID starting with fb59ddba63943bbedcbd118c21404bf6cc4d9a56f352ac5396b1b99714b05d6b not found: ID does not exist" Nov 03 14:06:21.882619 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:06:21.882588 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2a526440-5821-45b2-8bae-17c404668fc1-tls-certs\" (UniqueName: \"kubernetes.io/secret/2a526440-5821-45b2-8bae-17c404668fc1-tls-certs\") pod \"2a526440-5821-45b2-8bae-17c404668fc1\" (UID: \"2a526440-5821-45b2-8bae-17c404668fc1\") " Nov 03 14:06:21.882769 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:06:21.882658 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a526440-5821-45b2-8bae-17c404668fc1-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a526440-5821-45b2-8bae-17c404668fc1-dshm\") pod \"2a526440-5821-45b2-8bae-17c404668fc1\" (UID: \"2a526440-5821-45b2-8bae-17c404668fc1\") " Nov 03 14:06:21.882769 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:06:21.882699 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a526440-5821-45b2-8bae-17c404668fc1-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a526440-5821-45b2-8bae-17c404668fc1-model-cache\") pod \"2a526440-5821-45b2-8bae-17c404668fc1\" (UID: \"2a526440-5821-45b2-8bae-17c404668fc1\") " Nov 03 14:06:21.882851 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:06:21.882773 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a526440-5821-45b2-8bae-17c404668fc1-kube-api-access-2jnzm\" (UniqueName: \"kubernetes.io/projected/2a526440-5821-45b2-8bae-17c404668fc1-kube-api-access-2jnzm\") pod \"2a526440-5821-45b2-8bae-17c404668fc1\" (UID: \"2a526440-5821-45b2-8bae-17c404668fc1\") " Nov 03 14:06:21.882851 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:06:21.882820 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a526440-5821-45b2-8bae-17c404668fc1-home\" (UniqueName: \"kubernetes.io/empty-dir/2a526440-5821-45b2-8bae-17c404668fc1-home\") pod \"2a526440-5821-45b2-8bae-17c404668fc1\" (UID: \"2a526440-5821-45b2-8bae-17c404668fc1\") " Nov 03 14:06:21.882851 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:06:21.882838 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a526440-5821-45b2-8bae-17c404668fc1-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a526440-5821-45b2-8bae-17c404668fc1-tmp-dir\") pod \"2a526440-5821-45b2-8bae-17c404668fc1\" (UID: \"2a526440-5821-45b2-8bae-17c404668fc1\") " Nov 03 14:06:21.883049 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:06:21.883008 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a526440-5821-45b2-8bae-17c404668fc1-model-cache" pod "2a526440-5821-45b2-8bae-17c404668fc1" (UID: "2a526440-5821-45b2-8bae-17c404668fc1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:06:21.883163 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:06:21.883120 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a526440-5821-45b2-8bae-17c404668fc1-home" pod "2a526440-5821-45b2-8bae-17c404668fc1" (UID: "2a526440-5821-45b2-8bae-17c404668fc1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:06:21.883232 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:06:21.883171 5024 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a526440-5821-45b2-8bae-17c404668fc1-model-cache\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 14:06:21.883232 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:06:21.883195 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a526440-5821-45b2-8bae-17c404668fc1-tmp-dir" pod "2a526440-5821-45b2-8bae-17c404668fc1" (UID: "2a526440-5821-45b2-8bae-17c404668fc1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:06:21.884855 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:06:21.884831 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a526440-5821-45b2-8bae-17c404668fc1-dshm" pod "2a526440-5821-45b2-8bae-17c404668fc1" (UID: "2a526440-5821-45b2-8bae-17c404668fc1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:06:21.885155 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:06:21.885131 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a526440-5821-45b2-8bae-17c404668fc1-tls-certs" pod "2a526440-5821-45b2-8bae-17c404668fc1" (UID: "2a526440-5821-45b2-8bae-17c404668fc1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:06:21.885237 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:06:21.885138 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a526440-5821-45b2-8bae-17c404668fc1-kube-api-access-2jnzm" pod "2a526440-5821-45b2-8bae-17c404668fc1" (UID: "2a526440-5821-45b2-8bae-17c404668fc1"). InnerVolumeSpecName "kube-api-access-2jnzm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:06:21.984211 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:06:21.984178 5024 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a526440-5821-45b2-8bae-17c404668fc1-dshm\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 14:06:21.984211 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:06:21.984208 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2jnzm\" (UniqueName: \"kubernetes.io/projected/2a526440-5821-45b2-8bae-17c404668fc1-kube-api-access-2jnzm\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 14:06:21.984416 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:06:21.984220 5024 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a526440-5821-45b2-8bae-17c404668fc1-home\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 14:06:21.984416 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:06:21.984229 5024 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a526440-5821-45b2-8bae-17c404668fc1-tmp-dir\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 14:06:21.984416 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:06:21.984249 5024 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a526440-5821-45b2-8bae-17c404668fc1-tls-certs\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 14:06:22.100814 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:06:22.100780 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-64455b/simulator-single-main-64455b-kserve-bf6dbbbc6-nr7mf"] Nov 03 14:06:22.107261 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:06:22.107237 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-64455b/simulator-single-main-64455b-kserve-bf6dbbbc6-nr7mf"] Nov 03 14:06:24.107275 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:06:24.107238 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a526440-5821-45b2-8bae-17c404668fc1" path="/var/lib/kubelet/pods/2a526440-5821-45b2-8bae-17c404668fc1/volumes" Nov 03 14:06:27.314340 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:06:27.314293 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-64455b-b6c5ff7c5f7stz" podUID="620438d0-7fcc-4b0f-b93d-779691e3f820" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 14:06:27.314752 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:06:27.314335 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-64455b-6d498qmg7d" podUID="af66fe5d-0c58-4bb2-9089-24d85d371ab1" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 14:06:33.383831 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:06:33.383801 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-64455b-b6c5ff7c5f7stz" Nov 03 14:06:33.441977 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:06:33.441952 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-64455b-6d498qmg7d" Nov 03 14:06:33.494663 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:06:33.494631 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/620438d0-7fcc-4b0f-b93d-779691e3f820-kube-api-access-gzrvv\" (UniqueName: \"kubernetes.io/projected/620438d0-7fcc-4b0f-b93d-779691e3f820-kube-api-access-gzrvv\") pod \"620438d0-7fcc-4b0f-b93d-779691e3f820\" (UID: \"620438d0-7fcc-4b0f-b93d-779691e3f820\") " Nov 03 14:06:33.494663 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:06:33.494664 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/af66fe5d-0c58-4bb2-9089-24d85d371ab1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af66fe5d-0c58-4bb2-9089-24d85d371ab1-plugins-config-volume\") pod \"af66fe5d-0c58-4bb2-9089-24d85d371ab1\" (UID: \"af66fe5d-0c58-4bb2-9089-24d85d371ab1\") " Nov 03 14:06:33.494826 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:06:33.494744 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/620438d0-7fcc-4b0f-b93d-779691e3f820-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/620438d0-7fcc-4b0f-b93d-779691e3f820-plugins-config-volume\") pod \"620438d0-7fcc-4b0f-b93d-779691e3f820\" (UID: \"620438d0-7fcc-4b0f-b93d-779691e3f820\") " Nov 03 14:06:33.494826 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:06:33.494810 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af66fe5d-0c58-4bb2-9089-24d85d371ab1-kube-api-access-gcblc\" (UniqueName: \"kubernetes.io/projected/af66fe5d-0c58-4bb2-9089-24d85d371ab1-kube-api-access-gcblc\") pod \"af66fe5d-0c58-4bb2-9089-24d85d371ab1\" (UID: \"af66fe5d-0c58-4bb2-9089-24d85d371ab1\") " Nov 03 14:06:33.495168 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:06:33.495143 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/620438d0-7fcc-4b0f-b93d-779691e3f820-plugins-config-volume" pod "620438d0-7fcc-4b0f-b93d-779691e3f820" (UID: "620438d0-7fcc-4b0f-b93d-779691e3f820"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:06:33.495261 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:06:33.495180 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af66fe5d-0c58-4bb2-9089-24d85d371ab1-plugins-config-volume" pod "af66fe5d-0c58-4bb2-9089-24d85d371ab1" (UID: "af66fe5d-0c58-4bb2-9089-24d85d371ab1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:06:33.496768 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:06:33.496744 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/620438d0-7fcc-4b0f-b93d-779691e3f820-kube-api-access-gzrvv" pod "620438d0-7fcc-4b0f-b93d-779691e3f820" (UID: "620438d0-7fcc-4b0f-b93d-779691e3f820"). InnerVolumeSpecName "kube-api-access-gzrvv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:06:33.496834 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:06:33.496801 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af66fe5d-0c58-4bb2-9089-24d85d371ab1-kube-api-access-gcblc" pod "af66fe5d-0c58-4bb2-9089-24d85d371ab1" (UID: "af66fe5d-0c58-4bb2-9089-24d85d371ab1"). InnerVolumeSpecName "kube-api-access-gcblc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:06:33.596440 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:06:33.596412 5024 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/620438d0-7fcc-4b0f-b93d-779691e3f820-plugins-config-volume\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 14:06:33.596440 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:06:33.596436 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gcblc\" (UniqueName: \"kubernetes.io/projected/af66fe5d-0c58-4bb2-9089-24d85d371ab1-kube-api-access-gcblc\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 14:06:33.596602 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:06:33.596447 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gzrvv\" (UniqueName: \"kubernetes.io/projected/620438d0-7fcc-4b0f-b93d-779691e3f820-kube-api-access-gzrvv\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 14:06:33.596602 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:06:33.596455 5024 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af66fe5d-0c58-4bb2-9089-24d85d371ab1-plugins-config-volume\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 14:06:33.841017 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:06:33.840921 5024 generic.go:356] "Generic (PLEG): container finished" podID="af66fe5d-0c58-4bb2-9089-24d85d371ab1" containerID="c36e223da678bd036e9b56b01d111765ca83c070f9d60d116988662884626b0b" exitCode=0 Nov 03 14:06:33.841017 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:06:33.840971 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-64455b-6d498qmg7d" Nov 03 14:06:33.841017 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:06:33.841005 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-64455b-6d498qmg7d" event={"ID":"af66fe5d-0c58-4bb2-9089-24d85d371ab1","Type":"ContainerDied","Data":"c36e223da678bd036e9b56b01d111765ca83c070f9d60d116988662884626b0b"} Nov 03 14:06:33.841308 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:06:33.841058 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-64455b-6d498qmg7d" event={"ID":"af66fe5d-0c58-4bb2-9089-24d85d371ab1","Type":"ContainerDied","Data":"9415c95542d4f7e44b9a71c258d83b611166d7eb73ec211f080dfb1c2cc04e47"} Nov 03 14:06:33.841308 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:06:33.841086 5024 scope.go:122] "RemoveContainer" containerID="c36e223da678bd036e9b56b01d111765ca83c070f9d60d116988662884626b0b" Nov 03 14:06:33.842587 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:06:33.842566 5024 generic.go:356] "Generic (PLEG): container finished" podID="620438d0-7fcc-4b0f-b93d-779691e3f820" containerID="75deb7bf408b8f5bc0cb9902e09ac22e0e2e7266212a00fcca9349e4ceaa3e4e" exitCode=0 Nov 03 14:06:33.842703 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:06:33.842607 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-64455b-b6c5ff7c5f7stz" Nov 03 14:06:33.842703 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:06:33.842606 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-64455b-b6c5ff7c5f7stz" event={"ID":"620438d0-7fcc-4b0f-b93d-779691e3f820","Type":"ContainerDied","Data":"75deb7bf408b8f5bc0cb9902e09ac22e0e2e7266212a00fcca9349e4ceaa3e4e"} Nov 03 14:06:33.842831 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:06:33.842722 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-64455b-b6c5ff7c5f7stz" event={"ID":"620438d0-7fcc-4b0f-b93d-779691e3f820","Type":"ContainerDied","Data":"68b444941d7a7d728e81ec52a38318174fd47759fbeb7658b7860de03786c876"} Nov 03 14:06:33.852191 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:06:33.852164 5024 scope.go:122] "RemoveContainer" containerID="c36e223da678bd036e9b56b01d111765ca83c070f9d60d116988662884626b0b" Nov 03 14:06:33.852619 ip-10-0-131-36 kubenswrapper[5024]: E1103 14:06:33.852542 5024 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c36e223da678bd036e9b56b01d111765ca83c070f9d60d116988662884626b0b\": container with ID starting with c36e223da678bd036e9b56b01d111765ca83c070f9d60d116988662884626b0b not found: ID does not exist" containerID="c36e223da678bd036e9b56b01d111765ca83c070f9d60d116988662884626b0b" Nov 03 14:06:33.852619 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:06:33.852582 5024 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c36e223da678bd036e9b56b01d111765ca83c070f9d60d116988662884626b0b"} err="failed to get container status \"c36e223da678bd036e9b56b01d111765ca83c070f9d60d116988662884626b0b\": rpc error: code = NotFound desc = could not find container \"c36e223da678bd036e9b56b01d111765ca83c070f9d60d116988662884626b0b\": container with ID starting with c36e223da678bd036e9b56b01d111765ca83c070f9d60d116988662884626b0b not found: ID does not exist" Nov 03 14:06:33.852619 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:06:33.852608 5024 scope.go:122] "RemoveContainer" containerID="75deb7bf408b8f5bc0cb9902e09ac22e0e2e7266212a00fcca9349e4ceaa3e4e" Nov 03 14:06:33.863518 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:06:33.863498 5024 scope.go:122] "RemoveContainer" containerID="75deb7bf408b8f5bc0cb9902e09ac22e0e2e7266212a00fcca9349e4ceaa3e4e" Nov 03 14:06:33.863774 ip-10-0-131-36 kubenswrapper[5024]: E1103 14:06:33.863754 5024 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75deb7bf408b8f5bc0cb9902e09ac22e0e2e7266212a00fcca9349e4ceaa3e4e\": container with ID starting with 75deb7bf408b8f5bc0cb9902e09ac22e0e2e7266212a00fcca9349e4ceaa3e4e not found: ID does not exist" containerID="75deb7bf408b8f5bc0cb9902e09ac22e0e2e7266212a00fcca9349e4ceaa3e4e" Nov 03 14:06:33.863838 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:06:33.863781 5024 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75deb7bf408b8f5bc0cb9902e09ac22e0e2e7266212a00fcca9349e4ceaa3e4e"} err="failed to get container status \"75deb7bf408b8f5bc0cb9902e09ac22e0e2e7266212a00fcca9349e4ceaa3e4e\": rpc error: code = NotFound desc = could not find container \"75deb7bf408b8f5bc0cb9902e09ac22e0e2e7266212a00fcca9349e4ceaa3e4e\": container with ID starting with 75deb7bf408b8f5bc0cb9902e09ac22e0e2e7266212a00fcca9349e4ceaa3e4e not found: ID does not exist" Nov 03 14:06:33.875621 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:06:33.875600 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-64455b-6d498qmg7d"] Nov 03 14:06:33.879350 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:06:33.879331 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-64455b-6d498qmg7d"] Nov 03 14:06:33.891894 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:06:33.891873 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-64455b-b6c5ff7c5f7stz"] Nov 03 14:06:33.896200 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:06:33.896181 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-64455b-b6c5ff7c5f7stz"] Nov 03 14:06:34.107397 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:06:34.107364 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="620438d0-7fcc-4b0f-b93d-779691e3f820" path="/var/lib/kubelet/pods/620438d0-7fcc-4b0f-b93d-779691e3f820/volumes" Nov 03 14:06:34.107821 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:06:34.107800 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af66fe5d-0c58-4bb2-9089-24d85d371ab1" path="/var/lib/kubelet/pods/af66fe5d-0c58-4bb2-9089-24d85d371ab1/volumes" Nov 03 14:06:37.023298 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:06:37.023227 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-8ggxb"] Nov 03 14:06:37.026305 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:06:37.026271 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-8ggxb"] Nov 03 14:06:38.107712 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:06:38.107682 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="16a37f2f-c747-4493-bc05-a78846706028" path="/var/lib/kubelet/pods/16a37f2f-c747-4493-bc05-a78846706028/volumes" Nov 03 14:06:39.506849 ip-10-0-131-36 kubenswrapper[5024]: E1103 14:06:39.506805 5024 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d7mwz" containerName="extractor" Nov 03 14:06:40.558931 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:06:40.558888 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-64455b-6cf489d56b-hrkml"] Nov 03 14:06:40.559513 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:06:40.559338 5024 cpu_manager.go:414] "containerMap: removing container" podUID="620438d0-7fcc-4b0f-b93d-779691e3f820" containerName="payload-processing" Nov 03 14:06:40.559513 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:06:40.559352 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="620438d0-7fcc-4b0f-b93d-779691e3f820" containerName="payload-processing" Nov 03 14:06:40.559513 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:06:40.559376 5024 cpu_manager.go:414] "containerMap: removing container" podUID="af66fe5d-0c58-4bb2-9089-24d85d371ab1" containerName="payload-pre-processing" Nov 03 14:06:40.559513 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:06:40.559382 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af66fe5d-0c58-4bb2-9089-24d85d371ab1" containerName="payload-pre-processing" Nov 03 14:06:40.559513 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:06:40.559392 5024 cpu_manager.go:414] "containerMap: removing container" podUID="2a526440-5821-45b2-8bae-17c404668fc1" containerName="main" Nov 03 14:06:40.559513 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:06:40.559398 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a526440-5821-45b2-8bae-17c404668fc1" containerName="main" Nov 03 14:06:40.559513 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:06:40.559419 5024 cpu_manager.go:414] "containerMap: removing container" podUID="3d34cb7d-f2f1-44a6-8144-9d576c61e465" containerName="maas-api" Nov 03 14:06:40.559513 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:06:40.559424 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d34cb7d-f2f1-44a6-8144-9d576c61e465" containerName="maas-api" Nov 03 14:06:40.559513 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:06:40.559490 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="af66fe5d-0c58-4bb2-9089-24d85d371ab1" containerName="payload-pre-processing" Nov 03 14:06:40.559513 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:06:40.559500 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a526440-5821-45b2-8bae-17c404668fc1" containerName="main" Nov 03 14:06:40.559513 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:06:40.559510 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d34cb7d-f2f1-44a6-8144-9d576c61e465" containerName="maas-api" Nov 03 14:06:40.559513 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:06:40.559520 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="620438d0-7fcc-4b0f-b93d-779691e3f820" containerName="payload-processing" Nov 03 14:06:40.564633 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:06:40.564605 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-64455b-6cf489d56b-hrkml" Nov 03 14:06:40.573441 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:06:40.573415 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-64455b-serving-cert\"" Nov 03 14:06:40.573616 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:06:40.573457 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 14:06:40.574994 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:06:40.574970 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-64455b-6cf489d56b-hrkml"] Nov 03 14:06:40.575733 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:06:40.575713 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 14:06:40.576602 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:06:40.576509 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-p5qd5\"" Nov 03 14:06:40.576602 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:06:40.576573 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 03 14:06:40.666212 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:06:40.666179 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e99d1465-90a2-4f40-b7a6-0194e1d17d33-metrics-certs\") pod \"maas-api-e2e-worker-single-main-64455b-6cf489d56b-hrkml\" (UID: \"e99d1465-90a2-4f40-b7a6-0194e1d17d33\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-64455b-6cf489d56b-hrkml" Nov 03 14:06:40.666398 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:06:40.666243 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r2sbq\" (UniqueName: \"kubernetes.io/projected/e99d1465-90a2-4f40-b7a6-0194e1d17d33-kube-api-access-r2sbq\") pod \"maas-api-e2e-worker-single-main-64455b-6cf489d56b-hrkml\" (UID: \"e99d1465-90a2-4f40-b7a6-0194e1d17d33\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-64455b-6cf489d56b-hrkml" Nov 03 14:06:40.666398 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:06:40.666340 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e99d1465-90a2-4f40-b7a6-0194e1d17d33-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-64455b-6cf489d56b-hrkml\" (UID: \"e99d1465-90a2-4f40-b7a6-0194e1d17d33\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-64455b-6cf489d56b-hrkml" Nov 03 14:06:40.666476 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:06:40.666452 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e99d1465-90a2-4f40-b7a6-0194e1d17d33-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-64455b-6cf489d56b-hrkml\" (UID: \"e99d1465-90a2-4f40-b7a6-0194e1d17d33\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-64455b-6cf489d56b-hrkml" Nov 03 14:06:40.739777 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:06:40.739751 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-64455b-b6c5ff7c5xbjdl"] Nov 03 14:06:40.743260 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:06:40.743242 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-64455b-6d498zdbkw"] Nov 03 14:06:40.743387 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:06:40.743368 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-64455b-b6c5ff7c5xbjdl" Nov 03 14:06:40.746618 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:06:40.746597 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-64455b-6d498zdbkw" Nov 03 14:06:40.751562 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:06:40.751540 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-64455b\"" Nov 03 14:06:40.752673 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:06:40.752653 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-abd4ece3-dockercfg-fw9qs\"" Nov 03 14:06:40.754405 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:06:40.754386 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-64455b-b6c5ff7c5xbjdl"] Nov 03 14:06:40.767237 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:06:40.767213 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e99d1465-90a2-4f40-b7a6-0194e1d17d33-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-64455b-6cf489d56b-hrkml\" (UID: \"e99d1465-90a2-4f40-b7a6-0194e1d17d33\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-64455b-6cf489d56b-hrkml" Nov 03 14:06:40.767385 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:06:40.767278 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e99d1465-90a2-4f40-b7a6-0194e1d17d33-metrics-certs\") pod \"maas-api-e2e-worker-single-main-64455b-6cf489d56b-hrkml\" (UID: \"e99d1465-90a2-4f40-b7a6-0194e1d17d33\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-64455b-6cf489d56b-hrkml" Nov 03 14:06:40.767468 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:06:40.767387 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r2sbq\" (UniqueName: \"kubernetes.io/projected/e99d1465-90a2-4f40-b7a6-0194e1d17d33-kube-api-access-r2sbq\") pod \"maas-api-e2e-worker-single-main-64455b-6cf489d56b-hrkml\" (UID: \"e99d1465-90a2-4f40-b7a6-0194e1d17d33\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-64455b-6cf489d56b-hrkml" Nov 03 14:06:40.767468 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:06:40.767425 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e99d1465-90a2-4f40-b7a6-0194e1d17d33-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-64455b-6cf489d56b-hrkml\" (UID: \"e99d1465-90a2-4f40-b7a6-0194e1d17d33\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-64455b-6cf489d56b-hrkml" Nov 03 14:06:40.768039 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:06:40.768012 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e99d1465-90a2-4f40-b7a6-0194e1d17d33-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-64455b-6cf489d56b-hrkml\" (UID: \"e99d1465-90a2-4f40-b7a6-0194e1d17d33\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-64455b-6cf489d56b-hrkml" Nov 03 14:06:40.769696 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:06:40.768330 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-64455b-6d498zdbkw"] Nov 03 14:06:40.771785 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:06:40.771756 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e99d1465-90a2-4f40-b7a6-0194e1d17d33-metrics-certs\") pod \"maas-api-e2e-worker-single-main-64455b-6cf489d56b-hrkml\" (UID: \"e99d1465-90a2-4f40-b7a6-0194e1d17d33\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-64455b-6cf489d56b-hrkml" Nov 03 14:06:40.774828 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:06:40.774802 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e99d1465-90a2-4f40-b7a6-0194e1d17d33-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-64455b-6cf489d56b-hrkml\" (UID: \"e99d1465-90a2-4f40-b7a6-0194e1d17d33\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-64455b-6cf489d56b-hrkml" Nov 03 14:06:40.777769 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:06:40.777750 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r2sbq\" (UniqueName: \"kubernetes.io/projected/e99d1465-90a2-4f40-b7a6-0194e1d17d33-kube-api-access-r2sbq\") pod \"maas-api-e2e-worker-single-main-64455b-6cf489d56b-hrkml\" (UID: \"e99d1465-90a2-4f40-b7a6-0194e1d17d33\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-64455b-6cf489d56b-hrkml" Nov 03 14:06:40.868184 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:06:40.868132 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrp2w\" (UniqueName: \"kubernetes.io/projected/6de81601-2dee-47c5-ad49-33911cf69583-kube-api-access-vrp2w\") pod \"payload-pre-processing-e2e-worker-single-main-64455b-6d498zdbkw\" (UID: \"6de81601-2dee-47c5-ad49-33911cf69583\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-64455b-6d498zdbkw" Nov 03 14:06:40.868434 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:06:40.868403 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6de81601-2dee-47c5-ad49-33911cf69583-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-64455b-6d498zdbkw\" (UID: \"6de81601-2dee-47c5-ad49-33911cf69583\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-64455b-6d498zdbkw" Nov 03 14:06:40.868514 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:06:40.868479 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-plqpj\" (UniqueName: \"kubernetes.io/projected/3bc4d94c-4e26-4231-9c28-9cf047f9b601-kube-api-access-plqpj\") pod \"payload-processing-e2e-worker-single-main-64455b-b6c5ff7c5xbjdl\" (UID: \"3bc4d94c-4e26-4231-9c28-9cf047f9b601\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-64455b-b6c5ff7c5xbjdl" Nov 03 14:06:40.868598 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:06:40.868579 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3bc4d94c-4e26-4231-9c28-9cf047f9b601-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-64455b-b6c5ff7c5xbjdl\" (UID: \"3bc4d94c-4e26-4231-9c28-9cf047f9b601\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-64455b-b6c5ff7c5xbjdl" Nov 03 14:06:40.884363 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:06:40.884260 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-64455b-6cf489d56b-hrkml" Nov 03 14:06:40.970262 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:06:40.970193 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrp2w\" (UniqueName: \"kubernetes.io/projected/6de81601-2dee-47c5-ad49-33911cf69583-kube-api-access-vrp2w\") pod \"payload-pre-processing-e2e-worker-single-main-64455b-6d498zdbkw\" (UID: \"6de81601-2dee-47c5-ad49-33911cf69583\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-64455b-6d498zdbkw" Nov 03 14:06:40.970898 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:06:40.970872 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6de81601-2dee-47c5-ad49-33911cf69583-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-64455b-6d498zdbkw\" (UID: \"6de81601-2dee-47c5-ad49-33911cf69583\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-64455b-6d498zdbkw" Nov 03 14:06:40.970960 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:06:40.970947 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-plqpj\" (UniqueName: \"kubernetes.io/projected/3bc4d94c-4e26-4231-9c28-9cf047f9b601-kube-api-access-plqpj\") pod \"payload-processing-e2e-worker-single-main-64455b-b6c5ff7c5xbjdl\" (UID: \"3bc4d94c-4e26-4231-9c28-9cf047f9b601\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-64455b-b6c5ff7c5xbjdl" Nov 03 14:06:40.971008 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:06:40.970992 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3bc4d94c-4e26-4231-9c28-9cf047f9b601-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-64455b-b6c5ff7c5xbjdl\" (UID: \"3bc4d94c-4e26-4231-9c28-9cf047f9b601\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-64455b-b6c5ff7c5xbjdl" Nov 03 14:06:40.971979 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:06:40.971953 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3bc4d94c-4e26-4231-9c28-9cf047f9b601-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-64455b-b6c5ff7c5xbjdl\" (UID: \"3bc4d94c-4e26-4231-9c28-9cf047f9b601\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-64455b-b6c5ff7c5xbjdl" Nov 03 14:06:40.972759 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:06:40.972738 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6de81601-2dee-47c5-ad49-33911cf69583-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-64455b-6d498zdbkw\" (UID: \"6de81601-2dee-47c5-ad49-33911cf69583\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-64455b-6d498zdbkw" Nov 03 14:06:40.980129 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:06:40.980101 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrp2w\" (UniqueName: \"kubernetes.io/projected/6de81601-2dee-47c5-ad49-33911cf69583-kube-api-access-vrp2w\") pod \"payload-pre-processing-e2e-worker-single-main-64455b-6d498zdbkw\" (UID: \"6de81601-2dee-47c5-ad49-33911cf69583\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-64455b-6d498zdbkw" Nov 03 14:06:40.981598 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:06:40.981576 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-plqpj\" (UniqueName: \"kubernetes.io/projected/3bc4d94c-4e26-4231-9c28-9cf047f9b601-kube-api-access-plqpj\") pod \"payload-processing-e2e-worker-single-main-64455b-b6c5ff7c5xbjdl\" (UID: \"3bc4d94c-4e26-4231-9c28-9cf047f9b601\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-64455b-b6c5ff7c5xbjdl" Nov 03 14:06:41.059865 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:06:41.059834 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-64455b-b6c5ff7c5xbjdl" Nov 03 14:06:41.069338 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:06:41.069309 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-64455b-6d498zdbkw" Nov 03 14:06:41.166994 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:06:41.166968 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-64455b-6cf489d56b-hrkml"] Nov 03 14:06:41.170051 ip-10-0-131-36 kubenswrapper[5024]: W1103 14:06:41.170018 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode99d1465_90a2_4f40_b7a6_0194e1d17d33.slice/crio-2999adbb93cbd0c13bca1d45f0fe004e3b0d040894e15b94bcb5c62e7399b5f3 WatchSource:0}: Error finding container 2999adbb93cbd0c13bca1d45f0fe004e3b0d040894e15b94bcb5c62e7399b5f3: Status 404 returned error can't find the container with id 2999adbb93cbd0c13bca1d45f0fe004e3b0d040894e15b94bcb5c62e7399b5f3 Nov 03 14:06:41.342177 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:06:41.342144 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-64455b-b6c5ff7c5xbjdl"] Nov 03 14:06:41.342902 ip-10-0-131-36 kubenswrapper[5024]: W1103 14:06:41.342872 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3bc4d94c_4e26_4231_9c28_9cf047f9b601.slice/crio-2fc0dae2a88f9bbc31952d2c310147053db30eb5232bca52ce4ef729a87895c2 WatchSource:0}: Error finding container 2fc0dae2a88f9bbc31952d2c310147053db30eb5232bca52ce4ef729a87895c2: Status 404 returned error can't find the container with id 2fc0dae2a88f9bbc31952d2c310147053db30eb5232bca52ce4ef729a87895c2 Nov 03 14:06:41.364399 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:06:41.364378 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-64455b-6d498zdbkw"] Nov 03 14:06:41.889526 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:06:41.889485 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-64455b-b6c5ff7c5xbjdl" event={"ID":"3bc4d94c-4e26-4231-9c28-9cf047f9b601","Type":"ContainerStarted","Data":"c2a878de891b7e0385ab66e9236594394ea0c9341fe0772f26e68424932f10c4"} Nov 03 14:06:41.890005 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:06:41.889669 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-64455b-b6c5ff7c5xbjdl" Nov 03 14:06:41.890005 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:06:41.889717 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-64455b-b6c5ff7c5xbjdl" event={"ID":"3bc4d94c-4e26-4231-9c28-9cf047f9b601","Type":"ContainerStarted","Data":"2fc0dae2a88f9bbc31952d2c310147053db30eb5232bca52ce4ef729a87895c2"} Nov 03 14:06:41.891661 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:06:41.891426 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-64455b-6d498zdbkw" event={"ID":"6de81601-2dee-47c5-ad49-33911cf69583","Type":"ContainerStarted","Data":"e3c2ba3c82e8ec008dd63e9c2032c87644132e883166e70d317ac58fef558619"} Nov 03 14:06:41.891661 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:06:41.891477 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-64455b-6d498zdbkw" Nov 03 14:06:41.891661 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:06:41.891499 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-64455b-6d498zdbkw" event={"ID":"6de81601-2dee-47c5-ad49-33911cf69583","Type":"ContainerStarted","Data":"d8ebb208d461243abf721a15d017c35941e3d6d24e25477ff335dc343a846589"} Nov 03 14:06:41.893204 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:06:41.893178 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-64455b-6cf489d56b-hrkml" event={"ID":"e99d1465-90a2-4f40-b7a6-0194e1d17d33","Type":"ContainerStarted","Data":"c47585b568d57aec6c7d3418bfacfa76d0d27f44599e9ccf6464ae63c5d69604"} Nov 03 14:06:41.893436 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:06:41.893224 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-64455b-6cf489d56b-hrkml" event={"ID":"e99d1465-90a2-4f40-b7a6-0194e1d17d33","Type":"ContainerStarted","Data":"2999adbb93cbd0c13bca1d45f0fe004e3b0d040894e15b94bcb5c62e7399b5f3"} Nov 03 14:06:41.893436 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:06:41.893254 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-64455b-6cf489d56b-hrkml" Nov 03 14:06:41.930623 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:06:41.930538 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-64455b-b6c5ff7c5xbjdl" podStartSLOduration=1.930519696 podStartE2EDuration="1.930519696s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:06:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:06:41.909420616 +0000 UTC m=+3481.917611891" watchObservedRunningTime="2026-11-03 14:06:41.930519696 +0000 UTC m=+3481.938710953" Nov 03 14:06:41.930780 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:06:41.930674 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-64455b-6d498zdbkw" podStartSLOduration=1.930664824 podStartE2EDuration="1.930664824s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:06:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:06:41.92901512 +0000 UTC m=+3481.937206397" watchObservedRunningTime="2026-11-03 14:06:41.930664824 +0000 UTC m=+3481.938856102" Nov 03 14:06:41.950605 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:06:41.950477 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-64455b-6cf489d56b-hrkml" podStartSLOduration=1.421606504 podStartE2EDuration="1.950462067s" totalImagesPullingTime="528.855563ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:06:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:06:41.947833041 +0000 UTC m=+3481.956024322" watchObservedRunningTime="2026-11-03 14:06:41.950462067 +0000 UTC m=+3481.958653342" Nov 03 14:06:47.025366 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:06:47.025102 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-8wc9s"] Nov 03 14:06:47.028876 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:06:47.028843 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-8wc9s"] Nov 03 14:06:47.908063 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:06:47.908027 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-64455b-6cf489d56b-hrkml" Nov 03 14:06:48.107538 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:06:48.107497 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b216c9c7-9352-4585-94b5-c6852ac7429d" path="/var/lib/kubelet/pods/b216c9c7-9352-4585-94b5-c6852ac7429d/volumes" Nov 03 14:06:52.905507 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:06:52.905474 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-64455b-6d498zdbkw" Nov 03 14:06:52.905899 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:06:52.905640 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-64455b-b6c5ff7c5xbjdl" Nov 03 14:06:59.441272 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:06:59.441239 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-64455b-b6c5ff7c5xbjdl"] Nov 03 14:06:59.459822 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:06:59.459784 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-64455b-6d498zdbkw"] Nov 03 14:07:00.701726 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:00.701686 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-64455b-6cf489d56b-hrkml"] Nov 03 14:07:00.702257 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:00.701900 5024 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-64455b-6cf489d56b-hrkml" podUID="e99d1465-90a2-4f40-b7a6-0194e1d17d33" containerName="maas-api" containerID="cri-o://c47585b568d57aec6c7d3418bfacfa76d0d27f44599e9ccf6464ae63c5d69604" gracePeriod=30 Nov 03 14:07:00.992610 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:00.992580 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-64455b-6cf489d56b-hrkml" Nov 03 14:07:01.004500 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:01.004466 5024 generic.go:356] "Generic (PLEG): container finished" podID="e99d1465-90a2-4f40-b7a6-0194e1d17d33" containerID="c47585b568d57aec6c7d3418bfacfa76d0d27f44599e9ccf6464ae63c5d69604" exitCode=0 Nov 03 14:07:01.004648 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:01.004508 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-64455b-6cf489d56b-hrkml" Nov 03 14:07:01.004648 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:01.004549 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-64455b-6cf489d56b-hrkml" event={"ID":"e99d1465-90a2-4f40-b7a6-0194e1d17d33","Type":"ContainerDied","Data":"c47585b568d57aec6c7d3418bfacfa76d0d27f44599e9ccf6464ae63c5d69604"} Nov 03 14:07:01.004648 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:01.004609 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-64455b-6cf489d56b-hrkml" event={"ID":"e99d1465-90a2-4f40-b7a6-0194e1d17d33","Type":"ContainerDied","Data":"2999adbb93cbd0c13bca1d45f0fe004e3b0d040894e15b94bcb5c62e7399b5f3"} Nov 03 14:07:01.004648 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:01.004641 5024 scope.go:122] "RemoveContainer" containerID="c47585b568d57aec6c7d3418bfacfa76d0d27f44599e9ccf6464ae63c5d69604" Nov 03 14:07:01.016276 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:01.016213 5024 scope.go:122] "RemoveContainer" containerID="c47585b568d57aec6c7d3418bfacfa76d0d27f44599e9ccf6464ae63c5d69604" Nov 03 14:07:01.016937 ip-10-0-131-36 kubenswrapper[5024]: E1103 14:07:01.016868 5024 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c47585b568d57aec6c7d3418bfacfa76d0d27f44599e9ccf6464ae63c5d69604\": container with ID starting with c47585b568d57aec6c7d3418bfacfa76d0d27f44599e9ccf6464ae63c5d69604 not found: ID does not exist" containerID="c47585b568d57aec6c7d3418bfacfa76d0d27f44599e9ccf6464ae63c5d69604" Nov 03 14:07:01.016937 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:01.016904 5024 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c47585b568d57aec6c7d3418bfacfa76d0d27f44599e9ccf6464ae63c5d69604"} err="failed to get container status \"c47585b568d57aec6c7d3418bfacfa76d0d27f44599e9ccf6464ae63c5d69604\": rpc error: code = NotFound desc = could not find container \"c47585b568d57aec6c7d3418bfacfa76d0d27f44599e9ccf6464ae63c5d69604\": container with ID starting with c47585b568d57aec6c7d3418bfacfa76d0d27f44599e9ccf6464ae63c5d69604 not found: ID does not exist" Nov 03 14:07:01.079826 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:01.079789 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e99d1465-90a2-4f40-b7a6-0194e1d17d33-kube-api-access-r2sbq\" (UniqueName: \"kubernetes.io/projected/e99d1465-90a2-4f40-b7a6-0194e1d17d33-kube-api-access-r2sbq\") pod \"e99d1465-90a2-4f40-b7a6-0194e1d17d33\" (UID: \"e99d1465-90a2-4f40-b7a6-0194e1d17d33\") " Nov 03 14:07:01.080005 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:01.079889 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e99d1465-90a2-4f40-b7a6-0194e1d17d33-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e99d1465-90a2-4f40-b7a6-0194e1d17d33-maas-api-tls\") pod \"e99d1465-90a2-4f40-b7a6-0194e1d17d33\" (UID: \"e99d1465-90a2-4f40-b7a6-0194e1d17d33\") " Nov 03 14:07:01.080005 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:01.079944 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e99d1465-90a2-4f40-b7a6-0194e1d17d33-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e99d1465-90a2-4f40-b7a6-0194e1d17d33-openshift-service-ca\") pod \"e99d1465-90a2-4f40-b7a6-0194e1d17d33\" (UID: \"e99d1465-90a2-4f40-b7a6-0194e1d17d33\") " Nov 03 14:07:01.080005 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:01.079988 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e99d1465-90a2-4f40-b7a6-0194e1d17d33-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e99d1465-90a2-4f40-b7a6-0194e1d17d33-metrics-certs\") pod \"e99d1465-90a2-4f40-b7a6-0194e1d17d33\" (UID: \"e99d1465-90a2-4f40-b7a6-0194e1d17d33\") " Nov 03 14:07:01.080422 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:01.080392 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e99d1465-90a2-4f40-b7a6-0194e1d17d33-openshift-service-ca" pod "e99d1465-90a2-4f40-b7a6-0194e1d17d33" (UID: "e99d1465-90a2-4f40-b7a6-0194e1d17d33"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:07:01.082327 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:01.082268 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e99d1465-90a2-4f40-b7a6-0194e1d17d33-kube-api-access-r2sbq" pod "e99d1465-90a2-4f40-b7a6-0194e1d17d33" (UID: "e99d1465-90a2-4f40-b7a6-0194e1d17d33"). InnerVolumeSpecName "kube-api-access-r2sbq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:07:01.082588 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:01.082567 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e99d1465-90a2-4f40-b7a6-0194e1d17d33-metrics-certs" pod "e99d1465-90a2-4f40-b7a6-0194e1d17d33" (UID: "e99d1465-90a2-4f40-b7a6-0194e1d17d33"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:07:01.082652 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:01.082589 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e99d1465-90a2-4f40-b7a6-0194e1d17d33-maas-api-tls" pod "e99d1465-90a2-4f40-b7a6-0194e1d17d33" (UID: "e99d1465-90a2-4f40-b7a6-0194e1d17d33"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:07:01.180997 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:01.180961 5024 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e99d1465-90a2-4f40-b7a6-0194e1d17d33-openshift-service-ca\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 14:07:01.180997 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:01.180994 5024 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e99d1465-90a2-4f40-b7a6-0194e1d17d33-metrics-certs\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 14:07:01.180997 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:01.181004 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r2sbq\" (UniqueName: \"kubernetes.io/projected/e99d1465-90a2-4f40-b7a6-0194e1d17d33-kube-api-access-r2sbq\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 14:07:01.181239 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:01.181014 5024 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e99d1465-90a2-4f40-b7a6-0194e1d17d33-maas-api-tls\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 14:07:01.339741 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:01.339700 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-64455b-6cf489d56b-hrkml"] Nov 03 14:07:01.343589 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:01.343561 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-64455b-6cf489d56b-hrkml"] Nov 03 14:07:01.347204 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:01.347168 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-64455b-openshift-default-6477df7745pk7lm"] Nov 03 14:07:01.347480 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:01.347430 5024 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-64455b-openshift-default-6477df7745pk7lm" podUID="6cb1131d-f786-4f4c-bd32-05ce26bb5411" containerName="istio-proxy" containerID="cri-o://224c6b177b16d3d9bc95a8e18d4e6808e80272fc38d9dd42bfd0c81cf916e9a4" gracePeriod=30 Nov 03 14:07:02.108624 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:02.108580 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e99d1465-90a2-4f40-b7a6-0194e1d17d33" path="/var/lib/kubelet/pods/e99d1465-90a2-4f40-b7a6-0194e1d17d33/volumes" Nov 03 14:07:02.559422 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:02.559296 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cqm22"] Nov 03 14:07:02.562084 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:02.562061 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cqm22"] Nov 03 14:07:02.586016 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:02.585977 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fmzgf"] Nov 03 14:07:02.586681 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:02.586654 5024 cpu_manager.go:414] "containerMap: removing container" podUID="e99d1465-90a2-4f40-b7a6-0194e1d17d33" containerName="maas-api" Nov 03 14:07:02.586681 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:02.586680 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e99d1465-90a2-4f40-b7a6-0194e1d17d33" containerName="maas-api" Nov 03 14:07:02.586874 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:02.586819 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="e99d1465-90a2-4f40-b7a6-0194e1d17d33" containerName="maas-api" Nov 03 14:07:02.591995 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:02.591972 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fmzgf" Nov 03 14:07:02.596977 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:02.596954 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 14:07:02.597129 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:02.597093 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w6vqv\"" Nov 03 14:07:02.597215 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:02.597175 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 14:07:02.597326 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:02.597277 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 14:07:02.597880 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:02.597855 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fmzgf"] Nov 03 14:07:02.697939 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:02.697900 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26dd913b-c146-4745-9c18-3dbafa15c763-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-fmzgf\" (UID: \"26dd913b-c146-4745-9c18-3dbafa15c763\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fmzgf" Nov 03 14:07:02.698124 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:02.697955 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gmw6q\" (UniqueName: \"kubernetes.io/projected/26dd913b-c146-4745-9c18-3dbafa15c763-kube-api-access-gmw6q\") pod \"maas-api-revoke-sub-3d4156f1-fmzgf\" (UID: \"26dd913b-c146-4745-9c18-3dbafa15c763\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fmzgf" Nov 03 14:07:02.698124 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:02.698077 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26dd913b-c146-4745-9c18-3dbafa15c763-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-fmzgf\" (UID: \"26dd913b-c146-4745-9c18-3dbafa15c763\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fmzgf" Nov 03 14:07:02.799797 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:02.799749 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26dd913b-c146-4745-9c18-3dbafa15c763-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-fmzgf\" (UID: \"26dd913b-c146-4745-9c18-3dbafa15c763\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fmzgf" Nov 03 14:07:02.800003 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:02.799900 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26dd913b-c146-4745-9c18-3dbafa15c763-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-fmzgf\" (UID: \"26dd913b-c146-4745-9c18-3dbafa15c763\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fmzgf" Nov 03 14:07:02.800003 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:02.799978 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmw6q\" (UniqueName: \"kubernetes.io/projected/26dd913b-c146-4745-9c18-3dbafa15c763-kube-api-access-gmw6q\") pod \"maas-api-revoke-sub-3d4156f1-fmzgf\" (UID: \"26dd913b-c146-4745-9c18-3dbafa15c763\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fmzgf" Nov 03 14:07:02.800744 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:02.800717 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26dd913b-c146-4745-9c18-3dbafa15c763-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-fmzgf\" (UID: \"26dd913b-c146-4745-9c18-3dbafa15c763\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fmzgf" Nov 03 14:07:02.802262 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:02.802237 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26dd913b-c146-4745-9c18-3dbafa15c763-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-fmzgf\" (UID: \"26dd913b-c146-4745-9c18-3dbafa15c763\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fmzgf" Nov 03 14:07:02.808881 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:02.808850 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmw6q\" (UniqueName: \"kubernetes.io/projected/26dd913b-c146-4745-9c18-3dbafa15c763-kube-api-access-gmw6q\") pod \"maas-api-revoke-sub-3d4156f1-fmzgf\" (UID: \"26dd913b-c146-4745-9c18-3dbafa15c763\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fmzgf" Nov 03 14:07:02.911367 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:02.911335 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fmzgf" Nov 03 14:07:03.192182 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:03.192146 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fmzgf"] Nov 03 14:07:03.193699 ip-10-0-131-36 kubenswrapper[5024]: W1103 14:07:03.193663 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26dd913b_c146_4745_9c18_3dbafa15c763.slice/crio-1486e0f31c86f2c138bf82dfdc63c24193d3cc3d60c784df25cb884c0b8b092c WatchSource:0}: Error finding container 1486e0f31c86f2c138bf82dfdc63c24193d3cc3d60c784df25cb884c0b8b092c: Status 404 returned error can't find the container with id 1486e0f31c86f2c138bf82dfdc63c24193d3cc3d60c784df25cb884c0b8b092c Nov 03 14:07:04.027030 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:04.026992 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fmzgf" event={"ID":"26dd913b-c146-4745-9c18-3dbafa15c763","Type":"ContainerStarted","Data":"c4ce8bbd3cfb4f9304c5b2c0da2fbcb1b4a6ad4d8c5ebb0d8a255a80e83a781e"} Nov 03 14:07:04.027208 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:04.027053 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fmzgf" event={"ID":"26dd913b-c146-4745-9c18-3dbafa15c763","Type":"ContainerStarted","Data":"1486e0f31c86f2c138bf82dfdc63c24193d3cc3d60c784df25cb884c0b8b092c"} Nov 03 14:07:04.049165 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:04.049109 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fmzgf" podStartSLOduration=2.049093275 podStartE2EDuration="2.049093275s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:07:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:07:04.047754576 +0000 UTC m=+3504.055945854" watchObservedRunningTime="2026-11-03 14:07:04.049093275 +0000 UTC m=+3504.057284552" Nov 03 14:07:04.108919 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:04.108885 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8bb15cc1-1bd0-4182-9162-a8ebf916bba7" path="/var/lib/kubelet/pods/8bb15cc1-1bd0-4182-9162-a8ebf916bba7/volumes" Nov 03 14:07:04.441703 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:04.441653 5024 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-64455b-b6c5ff7c5xbjdl" podUID="3bc4d94c-4e26-4231-9c28-9cf047f9b601" containerName="payload-processing" containerID="cri-o://c2a878de891b7e0385ab66e9236594394ea0c9341fe0772f26e68424932f10c4" gracePeriod=25 Nov 03 14:07:04.460346 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:04.460312 5024 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-64455b-6d498zdbkw" podUID="6de81601-2dee-47c5-ad49-33911cf69583" containerName="payload-pre-processing" containerID="cri-o://e3c2ba3c82e8ec008dd63e9c2032c87644132e883166e70d317ac58fef558619" gracePeriod=25 Nov 03 14:07:04.710825 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:04.710796 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-64455b-b6c5ff7c5xbjdl" Nov 03 14:07:04.756671 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:04.756641 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-64455b-6d498zdbkw" Nov 03 14:07:04.825231 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:04.825198 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6de81601-2dee-47c5-ad49-33911cf69583-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6de81601-2dee-47c5-ad49-33911cf69583-plugins-config-volume\") pod \"6de81601-2dee-47c5-ad49-33911cf69583\" (UID: \"6de81601-2dee-47c5-ad49-33911cf69583\") " Nov 03 14:07:04.825405 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:04.825339 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6de81601-2dee-47c5-ad49-33911cf69583-kube-api-access-vrp2w\" (UniqueName: \"kubernetes.io/projected/6de81601-2dee-47c5-ad49-33911cf69583-kube-api-access-vrp2w\") pod \"6de81601-2dee-47c5-ad49-33911cf69583\" (UID: \"6de81601-2dee-47c5-ad49-33911cf69583\") " Nov 03 14:07:04.825488 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:04.825403 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3bc4d94c-4e26-4231-9c28-9cf047f9b601-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3bc4d94c-4e26-4231-9c28-9cf047f9b601-plugins-config-volume\") pod \"3bc4d94c-4e26-4231-9c28-9cf047f9b601\" (UID: \"3bc4d94c-4e26-4231-9c28-9cf047f9b601\") " Nov 03 14:07:04.825553 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:04.825527 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3bc4d94c-4e26-4231-9c28-9cf047f9b601-kube-api-access-plqpj\" (UniqueName: \"kubernetes.io/projected/3bc4d94c-4e26-4231-9c28-9cf047f9b601-kube-api-access-plqpj\") pod \"3bc4d94c-4e26-4231-9c28-9cf047f9b601\" (UID: \"3bc4d94c-4e26-4231-9c28-9cf047f9b601\") " Nov 03 14:07:04.825729 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:04.825699 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6de81601-2dee-47c5-ad49-33911cf69583-plugins-config-volume" pod "6de81601-2dee-47c5-ad49-33911cf69583" (UID: "6de81601-2dee-47c5-ad49-33911cf69583"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:07:04.825810 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:04.825760 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3bc4d94c-4e26-4231-9c28-9cf047f9b601-plugins-config-volume" pod "3bc4d94c-4e26-4231-9c28-9cf047f9b601" (UID: "3bc4d94c-4e26-4231-9c28-9cf047f9b601"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:07:04.825872 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:04.825856 5024 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6de81601-2dee-47c5-ad49-33911cf69583-plugins-config-volume\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 14:07:04.825911 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:04.825875 5024 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3bc4d94c-4e26-4231-9c28-9cf047f9b601-plugins-config-volume\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 14:07:04.827777 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:04.827752 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3bc4d94c-4e26-4231-9c28-9cf047f9b601-kube-api-access-plqpj" pod "3bc4d94c-4e26-4231-9c28-9cf047f9b601" (UID: "3bc4d94c-4e26-4231-9c28-9cf047f9b601"). InnerVolumeSpecName "kube-api-access-plqpj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:07:04.827864 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:04.827787 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6de81601-2dee-47c5-ad49-33911cf69583-kube-api-access-vrp2w" pod "6de81601-2dee-47c5-ad49-33911cf69583" (UID: "6de81601-2dee-47c5-ad49-33911cf69583"). InnerVolumeSpecName "kube-api-access-vrp2w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:07:04.927411 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:04.927375 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vrp2w\" (UniqueName: \"kubernetes.io/projected/6de81601-2dee-47c5-ad49-33911cf69583-kube-api-access-vrp2w\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 14:07:04.927411 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:04.927409 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-plqpj\" (UniqueName: \"kubernetes.io/projected/3bc4d94c-4e26-4231-9c28-9cf047f9b601-kube-api-access-plqpj\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 14:07:05.033447 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:05.033357 5024 generic.go:356] "Generic (PLEG): container finished" podID="6de81601-2dee-47c5-ad49-33911cf69583" containerID="e3c2ba3c82e8ec008dd63e9c2032c87644132e883166e70d317ac58fef558619" exitCode=0 Nov 03 14:07:05.033447 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:05.033407 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-64455b-6d498zdbkw" Nov 03 14:07:05.033663 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:05.033441 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-64455b-6d498zdbkw" event={"ID":"6de81601-2dee-47c5-ad49-33911cf69583","Type":"ContainerDied","Data":"e3c2ba3c82e8ec008dd63e9c2032c87644132e883166e70d317ac58fef558619"} Nov 03 14:07:05.033663 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:05.033521 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-64455b-6d498zdbkw" event={"ID":"6de81601-2dee-47c5-ad49-33911cf69583","Type":"ContainerDied","Data":"d8ebb208d461243abf721a15d017c35941e3d6d24e25477ff335dc343a846589"} Nov 03 14:07:05.033663 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:05.033547 5024 scope.go:122] "RemoveContainer" containerID="e3c2ba3c82e8ec008dd63e9c2032c87644132e883166e70d317ac58fef558619" Nov 03 14:07:05.036204 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:05.035362 5024 generic.go:356] "Generic (PLEG): container finished" podID="26dd913b-c146-4745-9c18-3dbafa15c763" containerID="c4ce8bbd3cfb4f9304c5b2c0da2fbcb1b4a6ad4d8c5ebb0d8a255a80e83a781e" exitCode=0 Nov 03 14:07:05.036204 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:05.035396 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fmzgf" event={"ID":"26dd913b-c146-4745-9c18-3dbafa15c763","Type":"ContainerDied","Data":"c4ce8bbd3cfb4f9304c5b2c0da2fbcb1b4a6ad4d8c5ebb0d8a255a80e83a781e"} Nov 03 14:07:05.036956 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:05.036931 5024 generic.go:356] "Generic (PLEG): container finished" podID="3bc4d94c-4e26-4231-9c28-9cf047f9b601" containerID="c2a878de891b7e0385ab66e9236594394ea0c9341fe0772f26e68424932f10c4" exitCode=0 Nov 03 14:07:05.037085 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:05.036960 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-64455b-b6c5ff7c5xbjdl" event={"ID":"3bc4d94c-4e26-4231-9c28-9cf047f9b601","Type":"ContainerDied","Data":"c2a878de891b7e0385ab66e9236594394ea0c9341fe0772f26e68424932f10c4"} Nov 03 14:07:05.037085 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:05.036975 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-64455b-b6c5ff7c5xbjdl" Nov 03 14:07:05.037085 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:05.037004 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-64455b-b6c5ff7c5xbjdl" event={"ID":"3bc4d94c-4e26-4231-9c28-9cf047f9b601","Type":"ContainerDied","Data":"2fc0dae2a88f9bbc31952d2c310147053db30eb5232bca52ce4ef729a87895c2"} Nov 03 14:07:05.045781 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:05.045752 5024 scope.go:122] "RemoveContainer" containerID="e3c2ba3c82e8ec008dd63e9c2032c87644132e883166e70d317ac58fef558619" Nov 03 14:07:05.046082 ip-10-0-131-36 kubenswrapper[5024]: E1103 14:07:05.046057 5024 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e3c2ba3c82e8ec008dd63e9c2032c87644132e883166e70d317ac58fef558619\": container with ID starting with e3c2ba3c82e8ec008dd63e9c2032c87644132e883166e70d317ac58fef558619 not found: ID does not exist" containerID="e3c2ba3c82e8ec008dd63e9c2032c87644132e883166e70d317ac58fef558619" Nov 03 14:07:05.046151 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:05.046094 5024 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3c2ba3c82e8ec008dd63e9c2032c87644132e883166e70d317ac58fef558619"} err="failed to get container status \"e3c2ba3c82e8ec008dd63e9c2032c87644132e883166e70d317ac58fef558619\": rpc error: code = NotFound desc = could not find container \"e3c2ba3c82e8ec008dd63e9c2032c87644132e883166e70d317ac58fef558619\": container with ID starting with e3c2ba3c82e8ec008dd63e9c2032c87644132e883166e70d317ac58fef558619 not found: ID does not exist" Nov 03 14:07:05.046151 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:05.046119 5024 scope.go:122] "RemoveContainer" containerID="c2a878de891b7e0385ab66e9236594394ea0c9341fe0772f26e68424932f10c4" Nov 03 14:07:05.054652 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:05.054633 5024 scope.go:122] "RemoveContainer" containerID="c2a878de891b7e0385ab66e9236594394ea0c9341fe0772f26e68424932f10c4" Nov 03 14:07:05.054928 ip-10-0-131-36 kubenswrapper[5024]: E1103 14:07:05.054901 5024 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2a878de891b7e0385ab66e9236594394ea0c9341fe0772f26e68424932f10c4\": container with ID starting with c2a878de891b7e0385ab66e9236594394ea0c9341fe0772f26e68424932f10c4 not found: ID does not exist" containerID="c2a878de891b7e0385ab66e9236594394ea0c9341fe0772f26e68424932f10c4" Nov 03 14:07:05.055023 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:05.054935 5024 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2a878de891b7e0385ab66e9236594394ea0c9341fe0772f26e68424932f10c4"} err="failed to get container status \"c2a878de891b7e0385ab66e9236594394ea0c9341fe0772f26e68424932f10c4\": rpc error: code = NotFound desc = could not find container \"c2a878de891b7e0385ab66e9236594394ea0c9341fe0772f26e68424932f10c4\": container with ID starting with c2a878de891b7e0385ab66e9236594394ea0c9341fe0772f26e68424932f10c4 not found: ID does not exist" Nov 03 14:07:05.076253 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:05.076208 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-64455b-6d498zdbkw"] Nov 03 14:07:05.079881 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:05.079852 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-64455b-6d498zdbkw"] Nov 03 14:07:05.091506 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:05.091477 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-64455b-b6c5ff7c5xbjdl"] Nov 03 14:07:05.097921 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:05.097891 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-64455b-b6c5ff7c5xbjdl"] Nov 03 14:07:06.114037 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:06.114004 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3bc4d94c-4e26-4231-9c28-9cf047f9b601" path="/var/lib/kubelet/pods/3bc4d94c-4e26-4231-9c28-9cf047f9b601/volumes" Nov 03 14:07:06.114628 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:06.114606 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6de81601-2dee-47c5-ad49-33911cf69583" path="/var/lib/kubelet/pods/6de81601-2dee-47c5-ad49-33911cf69583/volumes" Nov 03 14:07:06.185797 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:06.185772 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fmzgf" Nov 03 14:07:06.239648 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:06.239617 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/26dd913b-c146-4745-9c18-3dbafa15c763-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26dd913b-c146-4745-9c18-3dbafa15c763-maas-api-openshift-ca\") pod \"26dd913b-c146-4745-9c18-3dbafa15c763\" (UID: \"26dd913b-c146-4745-9c18-3dbafa15c763\") " Nov 03 14:07:06.239813 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:06.239666 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26dd913b-c146-4745-9c18-3dbafa15c763-kube-api-access-gmw6q\" (UniqueName: \"kubernetes.io/projected/26dd913b-c146-4745-9c18-3dbafa15c763-kube-api-access-gmw6q\") pod \"26dd913b-c146-4745-9c18-3dbafa15c763\" (UID: \"26dd913b-c146-4745-9c18-3dbafa15c763\") " Nov 03 14:07:06.239813 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:06.239697 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/26dd913b-c146-4745-9c18-3dbafa15c763-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26dd913b-c146-4745-9c18-3dbafa15c763-maas-api-xks-ca\") pod \"26dd913b-c146-4745-9c18-3dbafa15c763\" (UID: \"26dd913b-c146-4745-9c18-3dbafa15c763\") " Nov 03 14:07:06.240019 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:06.239992 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26dd913b-c146-4745-9c18-3dbafa15c763-maas-api-openshift-ca" pod "26dd913b-c146-4745-9c18-3dbafa15c763" (UID: "26dd913b-c146-4745-9c18-3dbafa15c763"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:07:06.240149 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:06.240114 5024 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26dd913b-c146-4745-9c18-3dbafa15c763-maas-api-openshift-ca\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 14:07:06.241845 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:06.241821 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26dd913b-c146-4745-9c18-3dbafa15c763-kube-api-access-gmw6q" pod "26dd913b-c146-4745-9c18-3dbafa15c763" (UID: "26dd913b-c146-4745-9c18-3dbafa15c763"). InnerVolumeSpecName "kube-api-access-gmw6q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:07:06.241845 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:06.241834 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/26dd913b-c146-4745-9c18-3dbafa15c763-maas-api-xks-ca" pod "26dd913b-c146-4745-9c18-3dbafa15c763" (UID: "26dd913b-c146-4745-9c18-3dbafa15c763"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:07:06.340824 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:06.340738 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gmw6q\" (UniqueName: \"kubernetes.io/projected/26dd913b-c146-4745-9c18-3dbafa15c763-kube-api-access-gmw6q\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 14:07:06.340824 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:06.340779 5024 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26dd913b-c146-4745-9c18-3dbafa15c763-maas-api-xks-ca\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 14:07:06.585906 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:06.585879 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-64455b-openshift-default-6477df7745pk7lm" Nov 03 14:07:06.644080 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:06.644041 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6cb1131d-f786-4f4c-bd32-05ce26bb5411-istio-token\" (UniqueName: \"kubernetes.io/projected/6cb1131d-f786-4f4c-bd32-05ce26bb5411-istio-token\") pod \"6cb1131d-f786-4f4c-bd32-05ce26bb5411\" (UID: \"6cb1131d-f786-4f4c-bd32-05ce26bb5411\") " Nov 03 14:07:06.644340 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:06.644095 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6cb1131d-f786-4f4c-bd32-05ce26bb5411-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6cb1131d-f786-4f4c-bd32-05ce26bb5411-istio-podinfo\") pod \"6cb1131d-f786-4f4c-bd32-05ce26bb5411\" (UID: \"6cb1131d-f786-4f4c-bd32-05ce26bb5411\") " Nov 03 14:07:06.644340 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:06.644273 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6cb1131d-f786-4f4c-bd32-05ce26bb5411-kube-api-access-p9dwh\" (UniqueName: \"kubernetes.io/projected/6cb1131d-f786-4f4c-bd32-05ce26bb5411-kube-api-access-p9dwh\") pod \"6cb1131d-f786-4f4c-bd32-05ce26bb5411\" (UID: \"6cb1131d-f786-4f4c-bd32-05ce26bb5411\") " Nov 03 14:07:06.644453 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:06.644354 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6cb1131d-f786-4f4c-bd32-05ce26bb5411-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6cb1131d-f786-4f4c-bd32-05ce26bb5411-workload-certs\") pod \"6cb1131d-f786-4f4c-bd32-05ce26bb5411\" (UID: \"6cb1131d-f786-4f4c-bd32-05ce26bb5411\") " Nov 03 14:07:06.644453 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:06.644395 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6cb1131d-f786-4f4c-bd32-05ce26bb5411-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6cb1131d-f786-4f4c-bd32-05ce26bb5411-istio-envoy\") pod \"6cb1131d-f786-4f4c-bd32-05ce26bb5411\" (UID: \"6cb1131d-f786-4f4c-bd32-05ce26bb5411\") " Nov 03 14:07:06.644570 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:06.644476 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6cb1131d-f786-4f4c-bd32-05ce26bb5411-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6cb1131d-f786-4f4c-bd32-05ce26bb5411-workload-socket\") pod \"6cb1131d-f786-4f4c-bd32-05ce26bb5411\" (UID: \"6cb1131d-f786-4f4c-bd32-05ce26bb5411\") " Nov 03 14:07:06.644570 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:06.644509 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6cb1131d-f786-4f4c-bd32-05ce26bb5411-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6cb1131d-f786-4f4c-bd32-05ce26bb5411-istio-data\") pod \"6cb1131d-f786-4f4c-bd32-05ce26bb5411\" (UID: \"6cb1131d-f786-4f4c-bd32-05ce26bb5411\") " Nov 03 14:07:06.644674 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:06.644576 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6cb1131d-f786-4f4c-bd32-05ce26bb5411-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6cb1131d-f786-4f4c-bd32-05ce26bb5411-istiod-ca-cert\") pod \"6cb1131d-f786-4f4c-bd32-05ce26bb5411\" (UID: \"6cb1131d-f786-4f4c-bd32-05ce26bb5411\") " Nov 03 14:07:06.644674 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:06.644621 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6cb1131d-f786-4f4c-bd32-05ce26bb5411-workload-certs" pod "6cb1131d-f786-4f4c-bd32-05ce26bb5411" (UID: "6cb1131d-f786-4f4c-bd32-05ce26bb5411"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:07:06.644674 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:06.644645 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6cb1131d-f786-4f4c-bd32-05ce26bb5411-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6cb1131d-f786-4f4c-bd32-05ce26bb5411-credential-socket\") pod \"6cb1131d-f786-4f4c-bd32-05ce26bb5411\" (UID: \"6cb1131d-f786-4f4c-bd32-05ce26bb5411\") " Nov 03 14:07:06.644846 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:06.644723 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6cb1131d-f786-4f4c-bd32-05ce26bb5411-workload-socket" pod "6cb1131d-f786-4f4c-bd32-05ce26bb5411" (UID: "6cb1131d-f786-4f4c-bd32-05ce26bb5411"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:07:06.644846 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:06.644793 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6cb1131d-f786-4f4c-bd32-05ce26bb5411-istio-data" pod "6cb1131d-f786-4f4c-bd32-05ce26bb5411" (UID: "6cb1131d-f786-4f4c-bd32-05ce26bb5411"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:07:06.645019 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:06.645001 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6cb1131d-f786-4f4c-bd32-05ce26bb5411-credential-socket" pod "6cb1131d-f786-4f4c-bd32-05ce26bb5411" (UID: "6cb1131d-f786-4f4c-bd32-05ce26bb5411"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:07:06.645095 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:06.645033 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6cb1131d-f786-4f4c-bd32-05ce26bb5411-istiod-ca-cert" pod "6cb1131d-f786-4f4c-bd32-05ce26bb5411" (UID: "6cb1131d-f786-4f4c-bd32-05ce26bb5411"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:07:06.645095 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:06.645089 5024 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6cb1131d-f786-4f4c-bd32-05ce26bb5411-workload-certs\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 14:07:06.645215 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:06.645103 5024 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6cb1131d-f786-4f4c-bd32-05ce26bb5411-workload-socket\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 14:07:06.645215 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:06.645113 5024 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6cb1131d-f786-4f4c-bd32-05ce26bb5411-istio-data\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 14:07:06.645215 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:06.645123 5024 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6cb1131d-f786-4f4c-bd32-05ce26bb5411-credential-socket\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 14:07:06.646549 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:06.646522 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6cb1131d-f786-4f4c-bd32-05ce26bb5411-kube-api-access-p9dwh" pod "6cb1131d-f786-4f4c-bd32-05ce26bb5411" (UID: "6cb1131d-f786-4f4c-bd32-05ce26bb5411"). InnerVolumeSpecName "kube-api-access-p9dwh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:07:06.646647 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:06.646555 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6cb1131d-f786-4f4c-bd32-05ce26bb5411-istio-podinfo" pod "6cb1131d-f786-4f4c-bd32-05ce26bb5411" (UID: "6cb1131d-f786-4f4c-bd32-05ce26bb5411"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 14:07:06.646895 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:06.646879 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6cb1131d-f786-4f4c-bd32-05ce26bb5411-istio-envoy" pod "6cb1131d-f786-4f4c-bd32-05ce26bb5411" (UID: "6cb1131d-f786-4f4c-bd32-05ce26bb5411"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:07:06.646973 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:06.646948 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6cb1131d-f786-4f4c-bd32-05ce26bb5411-istio-token" pod "6cb1131d-f786-4f4c-bd32-05ce26bb5411" (UID: "6cb1131d-f786-4f4c-bd32-05ce26bb5411"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:07:06.745721 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:06.745678 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p9dwh\" (UniqueName: \"kubernetes.io/projected/6cb1131d-f786-4f4c-bd32-05ce26bb5411-kube-api-access-p9dwh\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 14:07:06.745721 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:06.745713 5024 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6cb1131d-f786-4f4c-bd32-05ce26bb5411-istio-envoy\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 14:07:06.745721 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:06.745724 5024 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6cb1131d-f786-4f4c-bd32-05ce26bb5411-istiod-ca-cert\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 14:07:06.745965 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:06.745733 5024 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6cb1131d-f786-4f4c-bd32-05ce26bb5411-istio-token\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 14:07:06.745965 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:06.745742 5024 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6cb1131d-f786-4f4c-bd32-05ce26bb5411-istio-podinfo\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 14:07:07.051032 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:07.050942 5024 generic.go:356] "Generic (PLEG): container finished" podID="6cb1131d-f786-4f4c-bd32-05ce26bb5411" containerID="224c6b177b16d3d9bc95a8e18d4e6808e80272fc38d9dd42bfd0c81cf916e9a4" exitCode=0 Nov 03 14:07:07.051032 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:07.050999 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-64455b-openshift-default-6477df7745pk7lm" Nov 03 14:07:07.051273 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:07.051030 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-64455b-openshift-default-6477df7745pk7lm" event={"ID":"6cb1131d-f786-4f4c-bd32-05ce26bb5411","Type":"ContainerDied","Data":"224c6b177b16d3d9bc95a8e18d4e6808e80272fc38d9dd42bfd0c81cf916e9a4"} Nov 03 14:07:07.051273 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:07.051089 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-64455b-openshift-default-6477df7745pk7lm" event={"ID":"6cb1131d-f786-4f4c-bd32-05ce26bb5411","Type":"ContainerDied","Data":"25bb6828c8da09aec14fa7f2ddf7da3f486f2007e9990520c1e2b82806d761a1"} Nov 03 14:07:07.051273 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:07.051116 5024 scope.go:122] "RemoveContainer" containerID="224c6b177b16d3d9bc95a8e18d4e6808e80272fc38d9dd42bfd0c81cf916e9a4" Nov 03 14:07:07.052593 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:07.052574 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fmzgf" Nov 03 14:07:07.052696 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:07.052596 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fmzgf" event={"ID":"26dd913b-c146-4745-9c18-3dbafa15c763","Type":"ContainerDied","Data":"1486e0f31c86f2c138bf82dfdc63c24193d3cc3d60c784df25cb884c0b8b092c"} Nov 03 14:07:07.052696 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:07.052647 5024 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1486e0f31c86f2c138bf82dfdc63c24193d3cc3d60c784df25cb884c0b8b092c" Nov 03 14:07:07.062359 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:07.062338 5024 scope.go:122] "RemoveContainer" containerID="b539d0ad906549635c01bb1249e3a8c1214abb45868746e0bd50ac32933117ac" Nov 03 14:07:07.071388 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:07.071370 5024 scope.go:122] "RemoveContainer" containerID="224c6b177b16d3d9bc95a8e18d4e6808e80272fc38d9dd42bfd0c81cf916e9a4" Nov 03 14:07:07.071637 ip-10-0-131-36 kubenswrapper[5024]: E1103 14:07:07.071616 5024 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"224c6b177b16d3d9bc95a8e18d4e6808e80272fc38d9dd42bfd0c81cf916e9a4\": container with ID starting with 224c6b177b16d3d9bc95a8e18d4e6808e80272fc38d9dd42bfd0c81cf916e9a4 not found: ID does not exist" containerID="224c6b177b16d3d9bc95a8e18d4e6808e80272fc38d9dd42bfd0c81cf916e9a4" Nov 03 14:07:07.071707 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:07.071642 5024 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"224c6b177b16d3d9bc95a8e18d4e6808e80272fc38d9dd42bfd0c81cf916e9a4"} err="failed to get container status \"224c6b177b16d3d9bc95a8e18d4e6808e80272fc38d9dd42bfd0c81cf916e9a4\": rpc error: code = NotFound desc = could not find container \"224c6b177b16d3d9bc95a8e18d4e6808e80272fc38d9dd42bfd0c81cf916e9a4\": container with ID starting with 224c6b177b16d3d9bc95a8e18d4e6808e80272fc38d9dd42bfd0c81cf916e9a4 not found: ID does not exist" Nov 03 14:07:07.071707 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:07.071659 5024 scope.go:122] "RemoveContainer" containerID="b539d0ad906549635c01bb1249e3a8c1214abb45868746e0bd50ac32933117ac" Nov 03 14:07:07.071925 ip-10-0-131-36 kubenswrapper[5024]: E1103 14:07:07.071904 5024 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b539d0ad906549635c01bb1249e3a8c1214abb45868746e0bd50ac32933117ac\": container with ID starting with b539d0ad906549635c01bb1249e3a8c1214abb45868746e0bd50ac32933117ac not found: ID does not exist" containerID="b539d0ad906549635c01bb1249e3a8c1214abb45868746e0bd50ac32933117ac" Nov 03 14:07:07.071968 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:07.071933 5024 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b539d0ad906549635c01bb1249e3a8c1214abb45868746e0bd50ac32933117ac"} err="failed to get container status \"b539d0ad906549635c01bb1249e3a8c1214abb45868746e0bd50ac32933117ac\": rpc error: code = NotFound desc = could not find container \"b539d0ad906549635c01bb1249e3a8c1214abb45868746e0bd50ac32933117ac\": container with ID starting with b539d0ad906549635c01bb1249e3a8c1214abb45868746e0bd50ac32933117ac not found: ID does not exist" Nov 03 14:07:07.089376 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:07.089354 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-64455b-openshift-default-6477df7745pk7lm"] Nov 03 14:07:07.118654 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:07.118630 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-64455b-openshift-default-6477df7745pk7lm"] Nov 03 14:07:08.107209 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:08.107173 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6cb1131d-f786-4f4c-bd32-05ce26bb5411" path="/var/lib/kubelet/pods/6cb1131d-f786-4f4c-bd32-05ce26bb5411/volumes" Nov 03 14:07:09.505826 ip-10-0-131-36 kubenswrapper[5024]: E1103 14:07:09.505789 5024 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d7mwz" containerName="extractor" Nov 03 14:07:14.022069 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:14.022029 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-dkcsb"] Nov 03 14:07:14.024552 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:14.024528 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-dkcsb"] Nov 03 14:07:14.107668 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:14.107636 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="258ec688-8b0f-4fcf-992c-f539b9532ee5" path="/var/lib/kubelet/pods/258ec688-8b0f-4fcf-992c-f539b9532ee5/volumes" Nov 03 14:07:23.603601 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:23.603563 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-jkjxs"] Nov 03 14:07:23.604217 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:23.604194 5024 cpu_manager.go:414] "containerMap: removing container" podUID="26dd913b-c146-4745-9c18-3dbafa15c763" containerName="revoke-keys" Nov 03 14:07:23.604263 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:23.604221 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26dd913b-c146-4745-9c18-3dbafa15c763" containerName="revoke-keys" Nov 03 14:07:23.604263 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:23.604248 5024 cpu_manager.go:414] "containerMap: removing container" podUID="6cb1131d-f786-4f4c-bd32-05ce26bb5411" containerName="istio-proxy" Nov 03 14:07:23.604263 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:23.604257 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6cb1131d-f786-4f4c-bd32-05ce26bb5411" containerName="istio-proxy" Nov 03 14:07:23.604379 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:23.604272 5024 cpu_manager.go:414] "containerMap: removing container" podUID="3bc4d94c-4e26-4231-9c28-9cf047f9b601" containerName="payload-processing" Nov 03 14:07:23.604379 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:23.604279 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3bc4d94c-4e26-4231-9c28-9cf047f9b601" containerName="payload-processing" Nov 03 14:07:23.604379 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:23.604336 5024 cpu_manager.go:414] "containerMap: removing container" podUID="6cb1131d-f786-4f4c-bd32-05ce26bb5411" containerName="istio-proxy" Nov 03 14:07:23.604379 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:23.604343 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6cb1131d-f786-4f4c-bd32-05ce26bb5411" containerName="istio-proxy" Nov 03 14:07:23.604379 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:23.604356 5024 cpu_manager.go:414] "containerMap: removing container" podUID="6de81601-2dee-47c5-ad49-33911cf69583" containerName="payload-pre-processing" Nov 03 14:07:23.604379 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:23.604364 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6de81601-2dee-47c5-ad49-33911cf69583" containerName="payload-pre-processing" Nov 03 14:07:23.604568 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:23.604474 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="6cb1131d-f786-4f4c-bd32-05ce26bb5411" containerName="istio-proxy" Nov 03 14:07:23.604568 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:23.604486 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="6de81601-2dee-47c5-ad49-33911cf69583" containerName="payload-pre-processing" Nov 03 14:07:23.604568 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:23.604500 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="26dd913b-c146-4745-9c18-3dbafa15c763" containerName="revoke-keys" Nov 03 14:07:23.604568 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:23.604514 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="3bc4d94c-4e26-4231-9c28-9cf047f9b601" containerName="payload-processing" Nov 03 14:07:23.609789 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:23.609766 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-jkjxs" Nov 03 14:07:23.618657 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:23.618587 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 14:07:23.618657 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:23.618595 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w6vqv\"" Nov 03 14:07:23.618870 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:23.618667 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 14:07:23.618870 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:23.618708 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 14:07:23.620207 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:23.620181 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-jkjxs"] Nov 03 14:07:23.662910 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:23.662874 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef741c0f-88eb-4aaf-a1aa-f55329a8695e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-jkjxs\" (UID: \"ef741c0f-88eb-4aaf-a1aa-f55329a8695e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-jkjxs" Nov 03 14:07:23.662910 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:23.662913 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef741c0f-88eb-4aaf-a1aa-f55329a8695e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-jkjxs\" (UID: \"ef741c0f-88eb-4aaf-a1aa-f55329a8695e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-jkjxs" Nov 03 14:07:23.663138 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:23.662934 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bf4hr\" (UniqueName: \"kubernetes.io/projected/ef741c0f-88eb-4aaf-a1aa-f55329a8695e-kube-api-access-bf4hr\") pod \"maas-api-revoke-sub-d2bc93bc-jkjxs\" (UID: \"ef741c0f-88eb-4aaf-a1aa-f55329a8695e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-jkjxs" Nov 03 14:07:23.763747 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:23.763699 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef741c0f-88eb-4aaf-a1aa-f55329a8695e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-jkjxs\" (UID: \"ef741c0f-88eb-4aaf-a1aa-f55329a8695e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-jkjxs" Nov 03 14:07:23.763971 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:23.763751 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef741c0f-88eb-4aaf-a1aa-f55329a8695e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-jkjxs\" (UID: \"ef741c0f-88eb-4aaf-a1aa-f55329a8695e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-jkjxs" Nov 03 14:07:23.763971 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:23.763785 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bf4hr\" (UniqueName: \"kubernetes.io/projected/ef741c0f-88eb-4aaf-a1aa-f55329a8695e-kube-api-access-bf4hr\") pod \"maas-api-revoke-sub-d2bc93bc-jkjxs\" (UID: \"ef741c0f-88eb-4aaf-a1aa-f55329a8695e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-jkjxs" Nov 03 14:07:23.764465 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:23.764441 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef741c0f-88eb-4aaf-a1aa-f55329a8695e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-jkjxs\" (UID: \"ef741c0f-88eb-4aaf-a1aa-f55329a8695e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-jkjxs" Nov 03 14:07:23.766096 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:23.766073 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef741c0f-88eb-4aaf-a1aa-f55329a8695e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-jkjxs\" (UID: \"ef741c0f-88eb-4aaf-a1aa-f55329a8695e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-jkjxs" Nov 03 14:07:23.773159 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:23.773130 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bf4hr\" (UniqueName: \"kubernetes.io/projected/ef741c0f-88eb-4aaf-a1aa-f55329a8695e-kube-api-access-bf4hr\") pod \"maas-api-revoke-sub-d2bc93bc-jkjxs\" (UID: \"ef741c0f-88eb-4aaf-a1aa-f55329a8695e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-jkjxs" Nov 03 14:07:23.926126 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:23.926043 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-jkjxs" Nov 03 14:07:24.031229 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:24.031191 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-5g7xc"] Nov 03 14:07:24.034860 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:24.034833 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-5g7xc"] Nov 03 14:07:24.107583 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:24.107550 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9dcdd272-6b65-4e42-8eb0-a072d3d60992" path="/var/lib/kubelet/pods/9dcdd272-6b65-4e42-8eb0-a072d3d60992/volumes" Nov 03 14:07:24.199130 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:24.199091 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-jkjxs"] Nov 03 14:07:24.200495 ip-10-0-131-36 kubenswrapper[5024]: W1103 14:07:24.200473 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef741c0f_88eb_4aaf_a1aa_f55329a8695e.slice/crio-7fcf59f01f56030d4b82f3a0c046dbfd87ecadd9fb285c51bbe838eb85398f10 WatchSource:0}: Error finding container 7fcf59f01f56030d4b82f3a0c046dbfd87ecadd9fb285c51bbe838eb85398f10: Status 404 returned error can't find the container with id 7fcf59f01f56030d4b82f3a0c046dbfd87ecadd9fb285c51bbe838eb85398f10 Nov 03 14:07:24.280637 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:24.280607 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-jkjxs" event={"ID":"ef741c0f-88eb-4aaf-a1aa-f55329a8695e","Type":"ContainerStarted","Data":"7fcf59f01f56030d4b82f3a0c046dbfd87ecadd9fb285c51bbe838eb85398f10"} Nov 03 14:07:25.287568 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:25.287522 5024 generic.go:356] "Generic (PLEG): container finished" podID="ef741c0f-88eb-4aaf-a1aa-f55329a8695e" containerID="544fede72341b96ce68d93c195908b724ed64037e3276ad9ebcf631c00a9aa3b" exitCode=0 Nov 03 14:07:25.287964 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:25.287576 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-jkjxs" event={"ID":"ef741c0f-88eb-4aaf-a1aa-f55329a8695e","Type":"ContainerDied","Data":"544fede72341b96ce68d93c195908b724ed64037e3276ad9ebcf631c00a9aa3b"} Nov 03 14:07:26.436775 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:26.436746 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-jkjxs" Nov 03 14:07:26.487559 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:26.487523 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef741c0f-88eb-4aaf-a1aa-f55329a8695e-kube-api-access-bf4hr\" (UniqueName: \"kubernetes.io/projected/ef741c0f-88eb-4aaf-a1aa-f55329a8695e-kube-api-access-bf4hr\") pod \"ef741c0f-88eb-4aaf-a1aa-f55329a8695e\" (UID: \"ef741c0f-88eb-4aaf-a1aa-f55329a8695e\") " Nov 03 14:07:26.487776 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:26.487581 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef741c0f-88eb-4aaf-a1aa-f55329a8695e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef741c0f-88eb-4aaf-a1aa-f55329a8695e-maas-api-openshift-ca\") pod \"ef741c0f-88eb-4aaf-a1aa-f55329a8695e\" (UID: \"ef741c0f-88eb-4aaf-a1aa-f55329a8695e\") " Nov 03 14:07:26.487776 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:26.487652 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ef741c0f-88eb-4aaf-a1aa-f55329a8695e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef741c0f-88eb-4aaf-a1aa-f55329a8695e-maas-api-xks-ca\") pod \"ef741c0f-88eb-4aaf-a1aa-f55329a8695e\" (UID: \"ef741c0f-88eb-4aaf-a1aa-f55329a8695e\") " Nov 03 14:07:26.488028 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:26.488003 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef741c0f-88eb-4aaf-a1aa-f55329a8695e-maas-api-openshift-ca" pod "ef741c0f-88eb-4aaf-a1aa-f55329a8695e" (UID: "ef741c0f-88eb-4aaf-a1aa-f55329a8695e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:07:26.489795 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:26.489764 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef741c0f-88eb-4aaf-a1aa-f55329a8695e-maas-api-xks-ca" pod "ef741c0f-88eb-4aaf-a1aa-f55329a8695e" (UID: "ef741c0f-88eb-4aaf-a1aa-f55329a8695e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:07:26.489942 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:26.489924 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef741c0f-88eb-4aaf-a1aa-f55329a8695e-kube-api-access-bf4hr" pod "ef741c0f-88eb-4aaf-a1aa-f55329a8695e" (UID: "ef741c0f-88eb-4aaf-a1aa-f55329a8695e"). InnerVolumeSpecName "kube-api-access-bf4hr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:07:26.588896 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:26.588823 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bf4hr\" (UniqueName: \"kubernetes.io/projected/ef741c0f-88eb-4aaf-a1aa-f55329a8695e-kube-api-access-bf4hr\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 14:07:26.588896 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:26.588850 5024 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef741c0f-88eb-4aaf-a1aa-f55329a8695e-maas-api-openshift-ca\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 14:07:26.588896 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:26.588861 5024 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef741c0f-88eb-4aaf-a1aa-f55329a8695e-maas-api-xks-ca\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 14:07:27.299783 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:27.299743 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-jkjxs" Nov 03 14:07:27.299943 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:27.299780 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-jkjxs" event={"ID":"ef741c0f-88eb-4aaf-a1aa-f55329a8695e","Type":"ContainerDied","Data":"7fcf59f01f56030d4b82f3a0c046dbfd87ecadd9fb285c51bbe838eb85398f10"} Nov 03 14:07:27.299943 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:27.299829 5024 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7fcf59f01f56030d4b82f3a0c046dbfd87ecadd9fb285c51bbe838eb85398f10" Nov 03 14:07:35.651110 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:35.651071 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-97b845c85-9wxzm"] Nov 03 14:07:35.651617 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:35.651599 5024 cpu_manager.go:414] "containerMap: removing container" podUID="ef741c0f-88eb-4aaf-a1aa-f55329a8695e" containerName="revoke-keys" Nov 03 14:07:35.651668 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:35.651619 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef741c0f-88eb-4aaf-a1aa-f55329a8695e" containerName="revoke-keys" Nov 03 14:07:35.651711 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:35.651693 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef741c0f-88eb-4aaf-a1aa-f55329a8695e" containerName="revoke-keys" Nov 03 14:07:35.651711 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:35.651707 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="6cb1131d-f786-4f4c-bd32-05ce26bb5411" containerName="istio-proxy" Nov 03 14:07:35.654667 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:35.654644 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-97b845c85-9wxzm" Nov 03 14:07:35.664831 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:35.664806 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 14:07:35.664962 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:35.664888 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-p5qd5\"" Nov 03 14:07:35.665477 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:35.665444 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 14:07:35.665791 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:35.665709 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 03 14:07:35.665916 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:35.665751 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Nov 03 14:07:35.668506 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:35.668484 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-97b845c85-9wxzm"] Nov 03 14:07:35.675879 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:35.675851 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-7d6dc67969-zphs9"] Nov 03 14:07:35.680061 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:35.680037 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7d6dc67969-zphs9" Nov 03 14:07:35.694311 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:35.694256 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7d6dc67969-zphs9"] Nov 03 14:07:35.776409 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:35.776361 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zzb95\" (UniqueName: \"kubernetes.io/projected/96223074-b4c8-4855-88a9-34f3851c8e8b-kube-api-access-zzb95\") pod \"maas-api-7d6dc67969-zphs9\" (UID: \"96223074-b4c8-4855-88a9-34f3851c8e8b\") " pod="odh-ai-gateway-infra/maas-api-7d6dc67969-zphs9" Nov 03 14:07:35.776409 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:35.776412 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zbqn5\" (UniqueName: \"kubernetes.io/projected/ca508912-8228-4c1e-a2d2-4e6d2751c3d4-kube-api-access-zbqn5\") pod \"maas-api-97b845c85-9wxzm\" (UID: \"ca508912-8228-4c1e-a2d2-4e6d2751c3d4\") " pod="odh-ai-gateway-infra/maas-api-97b845c85-9wxzm" Nov 03 14:07:35.776653 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:35.776440 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/96223074-b4c8-4855-88a9-34f3851c8e8b-openshift-service-ca\") pod \"maas-api-7d6dc67969-zphs9\" (UID: \"96223074-b4c8-4855-88a9-34f3851c8e8b\") " pod="odh-ai-gateway-infra/maas-api-7d6dc67969-zphs9" Nov 03 14:07:35.776653 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:35.776596 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca508912-8228-4c1e-a2d2-4e6d2751c3d4-openshift-service-ca\") pod \"maas-api-97b845c85-9wxzm\" (UID: \"ca508912-8228-4c1e-a2d2-4e6d2751c3d4\") " pod="odh-ai-gateway-infra/maas-api-97b845c85-9wxzm" Nov 03 14:07:35.776727 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:35.776662 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ca508912-8228-4c1e-a2d2-4e6d2751c3d4-maas-api-tls\") pod \"maas-api-97b845c85-9wxzm\" (UID: \"ca508912-8228-4c1e-a2d2-4e6d2751c3d4\") " pod="odh-ai-gateway-infra/maas-api-97b845c85-9wxzm" Nov 03 14:07:35.776727 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:35.776721 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca508912-8228-4c1e-a2d2-4e6d2751c3d4-metrics-certs\") pod \"maas-api-97b845c85-9wxzm\" (UID: \"ca508912-8228-4c1e-a2d2-4e6d2751c3d4\") " pod="odh-ai-gateway-infra/maas-api-97b845c85-9wxzm" Nov 03 14:07:35.776814 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:35.776761 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/96223074-b4c8-4855-88a9-34f3851c8e8b-maas-api-tls\") pod \"maas-api-7d6dc67969-zphs9\" (UID: \"96223074-b4c8-4855-88a9-34f3851c8e8b\") " pod="odh-ai-gateway-infra/maas-api-7d6dc67969-zphs9" Nov 03 14:07:35.776889 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:35.776868 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/96223074-b4c8-4855-88a9-34f3851c8e8b-metrics-certs\") pod \"maas-api-7d6dc67969-zphs9\" (UID: \"96223074-b4c8-4855-88a9-34f3851c8e8b\") " pod="odh-ai-gateway-infra/maas-api-7d6dc67969-zphs9" Nov 03 14:07:35.877873 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:35.877820 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/96223074-b4c8-4855-88a9-34f3851c8e8b-maas-api-tls\") pod \"maas-api-7d6dc67969-zphs9\" (UID: \"96223074-b4c8-4855-88a9-34f3851c8e8b\") " pod="odh-ai-gateway-infra/maas-api-7d6dc67969-zphs9" Nov 03 14:07:35.878076 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:35.877913 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/96223074-b4c8-4855-88a9-34f3851c8e8b-metrics-certs\") pod \"maas-api-7d6dc67969-zphs9\" (UID: \"96223074-b4c8-4855-88a9-34f3851c8e8b\") " pod="odh-ai-gateway-infra/maas-api-7d6dc67969-zphs9" Nov 03 14:07:35.878076 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:35.877981 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zzb95\" (UniqueName: \"kubernetes.io/projected/96223074-b4c8-4855-88a9-34f3851c8e8b-kube-api-access-zzb95\") pod \"maas-api-7d6dc67969-zphs9\" (UID: \"96223074-b4c8-4855-88a9-34f3851c8e8b\") " pod="odh-ai-gateway-infra/maas-api-7d6dc67969-zphs9" Nov 03 14:07:35.878076 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:35.878007 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zbqn5\" (UniqueName: \"kubernetes.io/projected/ca508912-8228-4c1e-a2d2-4e6d2751c3d4-kube-api-access-zbqn5\") pod \"maas-api-97b845c85-9wxzm\" (UID: \"ca508912-8228-4c1e-a2d2-4e6d2751c3d4\") " pod="odh-ai-gateway-infra/maas-api-97b845c85-9wxzm" Nov 03 14:07:35.878076 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:35.878045 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/96223074-b4c8-4855-88a9-34f3851c8e8b-openshift-service-ca\") pod \"maas-api-7d6dc67969-zphs9\" (UID: \"96223074-b4c8-4855-88a9-34f3851c8e8b\") " pod="odh-ai-gateway-infra/maas-api-7d6dc67969-zphs9" Nov 03 14:07:35.878076 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:35.878076 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca508912-8228-4c1e-a2d2-4e6d2751c3d4-openshift-service-ca\") pod \"maas-api-97b845c85-9wxzm\" (UID: \"ca508912-8228-4c1e-a2d2-4e6d2751c3d4\") " pod="odh-ai-gateway-infra/maas-api-97b845c85-9wxzm" Nov 03 14:07:35.878395 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:35.878106 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ca508912-8228-4c1e-a2d2-4e6d2751c3d4-maas-api-tls\") pod \"maas-api-97b845c85-9wxzm\" (UID: \"ca508912-8228-4c1e-a2d2-4e6d2751c3d4\") " pod="odh-ai-gateway-infra/maas-api-97b845c85-9wxzm" Nov 03 14:07:35.878395 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:35.878168 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca508912-8228-4c1e-a2d2-4e6d2751c3d4-metrics-certs\") pod \"maas-api-97b845c85-9wxzm\" (UID: \"ca508912-8228-4c1e-a2d2-4e6d2751c3d4\") " pod="odh-ai-gateway-infra/maas-api-97b845c85-9wxzm" Nov 03 14:07:35.879312 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:35.879106 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca508912-8228-4c1e-a2d2-4e6d2751c3d4-openshift-service-ca\") pod \"maas-api-97b845c85-9wxzm\" (UID: \"ca508912-8228-4c1e-a2d2-4e6d2751c3d4\") " pod="odh-ai-gateway-infra/maas-api-97b845c85-9wxzm" Nov 03 14:07:35.879312 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:35.879250 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/96223074-b4c8-4855-88a9-34f3851c8e8b-openshift-service-ca\") pod \"maas-api-7d6dc67969-zphs9\" (UID: \"96223074-b4c8-4855-88a9-34f3851c8e8b\") " pod="odh-ai-gateway-infra/maas-api-7d6dc67969-zphs9" Nov 03 14:07:35.881693 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:35.881661 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca508912-8228-4c1e-a2d2-4e6d2751c3d4-metrics-certs\") pod \"maas-api-97b845c85-9wxzm\" (UID: \"ca508912-8228-4c1e-a2d2-4e6d2751c3d4\") " pod="odh-ai-gateway-infra/maas-api-97b845c85-9wxzm" Nov 03 14:07:35.881802 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:35.881700 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ca508912-8228-4c1e-a2d2-4e6d2751c3d4-maas-api-tls\") pod \"maas-api-97b845c85-9wxzm\" (UID: \"ca508912-8228-4c1e-a2d2-4e6d2751c3d4\") " pod="odh-ai-gateway-infra/maas-api-97b845c85-9wxzm" Nov 03 14:07:35.882969 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:35.882944 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/96223074-b4c8-4855-88a9-34f3851c8e8b-metrics-certs\") pod \"maas-api-7d6dc67969-zphs9\" (UID: \"96223074-b4c8-4855-88a9-34f3851c8e8b\") " pod="odh-ai-gateway-infra/maas-api-7d6dc67969-zphs9" Nov 03 14:07:35.883767 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:35.883741 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/96223074-b4c8-4855-88a9-34f3851c8e8b-maas-api-tls\") pod \"maas-api-7d6dc67969-zphs9\" (UID: \"96223074-b4c8-4855-88a9-34f3851c8e8b\") " pod="odh-ai-gateway-infra/maas-api-7d6dc67969-zphs9" Nov 03 14:07:35.888492 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:35.888464 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zzb95\" (UniqueName: \"kubernetes.io/projected/96223074-b4c8-4855-88a9-34f3851c8e8b-kube-api-access-zzb95\") pod \"maas-api-7d6dc67969-zphs9\" (UID: \"96223074-b4c8-4855-88a9-34f3851c8e8b\") " pod="odh-ai-gateway-infra/maas-api-7d6dc67969-zphs9" Nov 03 14:07:35.889072 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:35.889052 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zbqn5\" (UniqueName: \"kubernetes.io/projected/ca508912-8228-4c1e-a2d2-4e6d2751c3d4-kube-api-access-zbqn5\") pod \"maas-api-97b845c85-9wxzm\" (UID: \"ca508912-8228-4c1e-a2d2-4e6d2751c3d4\") " pod="odh-ai-gateway-infra/maas-api-97b845c85-9wxzm" Nov 03 14:07:35.972549 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:35.972463 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-97b845c85-9wxzm" Nov 03 14:07:36.002931 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:36.002888 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7d6dc67969-zphs9" Nov 03 14:07:36.259034 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:36.259007 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-97b845c85-9wxzm"] Nov 03 14:07:36.260838 ip-10-0-131-36 kubenswrapper[5024]: W1103 14:07:36.260804 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca508912_8228_4c1e_a2d2_4e6d2751c3d4.slice/crio-7895d955fe6b86c80e1c621a584133bb7cc93d3dedad23339faef54b886687fb WatchSource:0}: Error finding container 7895d955fe6b86c80e1c621a584133bb7cc93d3dedad23339faef54b886687fb: Status 404 returned error can't find the container with id 7895d955fe6b86c80e1c621a584133bb7cc93d3dedad23339faef54b886687fb Nov 03 14:07:36.323331 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:36.323301 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7d6dc67969-zphs9"] Nov 03 14:07:36.324783 ip-10-0-131-36 kubenswrapper[5024]: W1103 14:07:36.324744 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96223074_b4c8_4855_88a9_34f3851c8e8b.slice/crio-996f06b3d680e3a6a437947458cdf266d7e7c122f721691f642792487691725f WatchSource:0}: Error finding container 996f06b3d680e3a6a437947458cdf266d7e7c122f721691f642792487691725f: Status 404 returned error can't find the container with id 996f06b3d680e3a6a437947458cdf266d7e7c122f721691f642792487691725f Nov 03 14:07:36.373362 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:36.373323 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-97b845c85-9wxzm" event={"ID":"ca508912-8228-4c1e-a2d2-4e6d2751c3d4","Type":"ContainerStarted","Data":"7895d955fe6b86c80e1c621a584133bb7cc93d3dedad23339faef54b886687fb"} Nov 03 14:07:36.374833 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:36.374805 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7d6dc67969-zphs9" event={"ID":"96223074-b4c8-4855-88a9-34f3851c8e8b","Type":"ContainerStarted","Data":"996f06b3d680e3a6a437947458cdf266d7e7c122f721691f642792487691725f"} Nov 03 14:07:37.383055 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:37.383012 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7d6dc67969-zphs9" event={"ID":"96223074-b4c8-4855-88a9-34f3851c8e8b","Type":"ContainerStarted","Data":"fde3c0204ef40b9fceede889fb674b49fce34f14b0166716eeaf0d2d94aa6991"} Nov 03 14:07:37.383580 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:37.383088 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-7d6dc67969-zphs9" Nov 03 14:07:37.384783 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:37.384752 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-97b845c85-9wxzm" event={"ID":"ca508912-8228-4c1e-a2d2-4e6d2751c3d4","Type":"ContainerStarted","Data":"8ff4c6aeceae566236875fe97bf3c13cfaf9d2ba49265bf8e00637f5b6809c99"} Nov 03 14:07:37.384949 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:37.384810 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-97b845c85-9wxzm" Nov 03 14:07:37.405147 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:37.405101 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-7d6dc67969-zphs9" podStartSLOduration=1.8448971809999999 podStartE2EDuration="2.405087773s" totalImagesPullingTime="560.190592ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:07:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:07:37.403275666 +0000 UTC m=+3537.411466971" watchObservedRunningTime="2026-11-03 14:07:37.405087773 +0000 UTC m=+3537.413279050" Nov 03 14:07:37.424937 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:37.424881 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-97b845c85-9wxzm" podStartSLOduration=1.9203176069999999 podStartE2EDuration="2.424867383s" totalImagesPullingTime="504.549776ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:07:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:07:37.423171221 +0000 UTC m=+3537.431362508" watchObservedRunningTime="2026-11-03 14:07:37.424867383 +0000 UTC m=+3537.433058683" Nov 03 14:07:39.507742 ip-10-0-131-36 kubenswrapper[5024]: E1103 14:07:39.507699 5024 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d7mwz" containerName="extractor" Nov 03 14:07:43.396316 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:43.396253 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-97b845c85-9wxzm" Nov 03 14:07:43.396863 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:43.396683 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-7d6dc67969-zphs9" Nov 03 14:07:43.471600 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:43.471549 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-97b845c85-9wxzm"] Nov 03 14:07:43.472139 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:43.471788 5024 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-97b845c85-9wxzm" podUID="ca508912-8228-4c1e-a2d2-4e6d2751c3d4" containerName="maas-api" containerID="cri-o://8ff4c6aeceae566236875fe97bf3c13cfaf9d2ba49265bf8e00637f5b6809c99" gracePeriod=30 Nov 03 14:07:43.723897 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:43.723868 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-97b845c85-9wxzm" Nov 03 14:07:43.863160 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:43.863080 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ca508912-8228-4c1e-a2d2-4e6d2751c3d4-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca508912-8228-4c1e-a2d2-4e6d2751c3d4-metrics-certs\") pod \"ca508912-8228-4c1e-a2d2-4e6d2751c3d4\" (UID: \"ca508912-8228-4c1e-a2d2-4e6d2751c3d4\") " Nov 03 14:07:43.863160 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:43.863133 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ca508912-8228-4c1e-a2d2-4e6d2751c3d4-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca508912-8228-4c1e-a2d2-4e6d2751c3d4-openshift-service-ca\") pod \"ca508912-8228-4c1e-a2d2-4e6d2751c3d4\" (UID: \"ca508912-8228-4c1e-a2d2-4e6d2751c3d4\") " Nov 03 14:07:43.863519 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:43.863264 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca508912-8228-4c1e-a2d2-4e6d2751c3d4-kube-api-access-zbqn5\" (UniqueName: \"kubernetes.io/projected/ca508912-8228-4c1e-a2d2-4e6d2751c3d4-kube-api-access-zbqn5\") pod \"ca508912-8228-4c1e-a2d2-4e6d2751c3d4\" (UID: \"ca508912-8228-4c1e-a2d2-4e6d2751c3d4\") " Nov 03 14:07:43.863519 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:43.863345 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ca508912-8228-4c1e-a2d2-4e6d2751c3d4-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ca508912-8228-4c1e-a2d2-4e6d2751c3d4-maas-api-tls\") pod \"ca508912-8228-4c1e-a2d2-4e6d2751c3d4\" (UID: \"ca508912-8228-4c1e-a2d2-4e6d2751c3d4\") " Nov 03 14:07:43.863633 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:43.863576 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca508912-8228-4c1e-a2d2-4e6d2751c3d4-openshift-service-ca" pod "ca508912-8228-4c1e-a2d2-4e6d2751c3d4" (UID: "ca508912-8228-4c1e-a2d2-4e6d2751c3d4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:07:43.865440 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:43.865409 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ca508912-8228-4c1e-a2d2-4e6d2751c3d4-metrics-certs" pod "ca508912-8228-4c1e-a2d2-4e6d2751c3d4" (UID: "ca508912-8228-4c1e-a2d2-4e6d2751c3d4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:07:43.865617 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:43.865599 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca508912-8228-4c1e-a2d2-4e6d2751c3d4-kube-api-access-zbqn5" pod "ca508912-8228-4c1e-a2d2-4e6d2751c3d4" (UID: "ca508912-8228-4c1e-a2d2-4e6d2751c3d4"). InnerVolumeSpecName "kube-api-access-zbqn5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:07:43.865902 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:43.865873 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ca508912-8228-4c1e-a2d2-4e6d2751c3d4-maas-api-tls" pod "ca508912-8228-4c1e-a2d2-4e6d2751c3d4" (UID: "ca508912-8228-4c1e-a2d2-4e6d2751c3d4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:07:43.964707 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:43.964625 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zbqn5\" (UniqueName: \"kubernetes.io/projected/ca508912-8228-4c1e-a2d2-4e6d2751c3d4-kube-api-access-zbqn5\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 14:07:43.964707 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:43.964659 5024 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ca508912-8228-4c1e-a2d2-4e6d2751c3d4-maas-api-tls\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 14:07:43.964707 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:43.964671 5024 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca508912-8228-4c1e-a2d2-4e6d2751c3d4-metrics-certs\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 14:07:43.964707 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:43.964681 5024 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca508912-8228-4c1e-a2d2-4e6d2751c3d4-openshift-service-ca\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 14:07:44.432895 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:44.432856 5024 generic.go:356] "Generic (PLEG): container finished" podID="ca508912-8228-4c1e-a2d2-4e6d2751c3d4" containerID="8ff4c6aeceae566236875fe97bf3c13cfaf9d2ba49265bf8e00637f5b6809c99" exitCode=0 Nov 03 14:07:44.433482 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:44.433067 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-97b845c85-9wxzm" event={"ID":"ca508912-8228-4c1e-a2d2-4e6d2751c3d4","Type":"ContainerDied","Data":"8ff4c6aeceae566236875fe97bf3c13cfaf9d2ba49265bf8e00637f5b6809c99"} Nov 03 14:07:44.433482 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:44.433228 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-97b845c85-9wxzm" event={"ID":"ca508912-8228-4c1e-a2d2-4e6d2751c3d4","Type":"ContainerDied","Data":"7895d955fe6b86c80e1c621a584133bb7cc93d3dedad23339faef54b886687fb"} Nov 03 14:07:44.433482 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:44.433297 5024 scope.go:122] "RemoveContainer" containerID="8ff4c6aeceae566236875fe97bf3c13cfaf9d2ba49265bf8e00637f5b6809c99" Nov 03 14:07:44.433682 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:44.433512 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-97b845c85-9wxzm" Nov 03 14:07:44.445619 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:44.445593 5024 scope.go:122] "RemoveContainer" containerID="8ff4c6aeceae566236875fe97bf3c13cfaf9d2ba49265bf8e00637f5b6809c99" Nov 03 14:07:44.445950 ip-10-0-131-36 kubenswrapper[5024]: E1103 14:07:44.445926 5024 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ff4c6aeceae566236875fe97bf3c13cfaf9d2ba49265bf8e00637f5b6809c99\": container with ID starting with 8ff4c6aeceae566236875fe97bf3c13cfaf9d2ba49265bf8e00637f5b6809c99 not found: ID does not exist" containerID="8ff4c6aeceae566236875fe97bf3c13cfaf9d2ba49265bf8e00637f5b6809c99" Nov 03 14:07:44.446035 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:44.445964 5024 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ff4c6aeceae566236875fe97bf3c13cfaf9d2ba49265bf8e00637f5b6809c99"} err="failed to get container status \"8ff4c6aeceae566236875fe97bf3c13cfaf9d2ba49265bf8e00637f5b6809c99\": rpc error: code = NotFound desc = could not find container \"8ff4c6aeceae566236875fe97bf3c13cfaf9d2ba49265bf8e00637f5b6809c99\": container with ID starting with 8ff4c6aeceae566236875fe97bf3c13cfaf9d2ba49265bf8e00637f5b6809c99 not found: ID does not exist" Nov 03 14:07:44.459913 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:44.459865 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-97b845c85-9wxzm"] Nov 03 14:07:44.463417 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:44.463386 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-97b845c85-9wxzm"] Nov 03 14:07:45.023614 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:45.023569 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-cmhcd"] Nov 03 14:07:45.035517 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:45.033923 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-cmhcd"] Nov 03 14:07:46.122740 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:46.122591 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="789b5ef9-0394-4398-a707-13dabe0fa3b6" path="/var/lib/kubelet/pods/789b5ef9-0394-4398-a707-13dabe0fa3b6/volumes" Nov 03 14:07:46.123999 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:46.123747 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ca508912-8228-4c1e-a2d2-4e6d2751c3d4" path="/var/lib/kubelet/pods/ca508912-8228-4c1e-a2d2-4e6d2751c3d4/volumes" Nov 03 14:07:46.647061 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:46.647021 5024 scope.go:122] "RemoveContainer" containerID="2da0b595bad46a280bd5c933ce4672f3ad648f65a249c180abc480e83f43289e" Nov 03 14:07:46.719335 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:46.719300 5024 scope.go:122] "RemoveContainer" containerID="eb5d0e5d2fa1622740dce4eeb5553fb9eef197f0f0ab331a2ee64cb10956d234" Nov 03 14:07:46.801112 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:46.800665 5024 scope.go:122] "RemoveContainer" containerID="bea008ac902a22ee255a9ea60b1adc49a988f80f58bfd420537048edb25720dc" Nov 03 14:07:54.575943 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:54.575902 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ff7c956f-xvmcg"] Nov 03 14:07:54.576483 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:54.576409 5024 cpu_manager.go:414] "containerMap: removing container" podUID="ca508912-8228-4c1e-a2d2-4e6d2751c3d4" containerName="maas-api" Nov 03 14:07:54.576483 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:54.576423 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca508912-8228-4c1e-a2d2-4e6d2751c3d4" containerName="maas-api" Nov 03 14:07:54.576608 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:54.576491 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca508912-8228-4c1e-a2d2-4e6d2751c3d4" containerName="maas-api" Nov 03 14:07:54.581590 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:54.581566 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff7c956f-xvmcg" Nov 03 14:07:54.588202 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:54.588174 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 14:07:54.588422 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:54.588345 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w6vqv\"" Nov 03 14:07:54.589241 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:54.589217 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ff7c956f-xvmcg"] Nov 03 14:07:54.621711 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:54.621677 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68c04a19-7b8d-4ed7-990a-455598d4514f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ff7c956f-xvmcg\" (UID: \"68c04a19-7b8d-4ed7-990a-455598d4514f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff7c956f-xvmcg" Nov 03 14:07:54.621852 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:54.621715 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5kkss\" (UniqueName: \"kubernetes.io/projected/68c04a19-7b8d-4ed7-990a-455598d4514f-kube-api-access-5kkss\") pod \"maas-api-revoke-sub-ff7c956f-xvmcg\" (UID: \"68c04a19-7b8d-4ed7-990a-455598d4514f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff7c956f-xvmcg" Nov 03 14:07:54.621852 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:54.621806 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68c04a19-7b8d-4ed7-990a-455598d4514f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ff7c956f-xvmcg\" (UID: \"68c04a19-7b8d-4ed7-990a-455598d4514f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff7c956f-xvmcg" Nov 03 14:07:54.723001 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:54.722973 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68c04a19-7b8d-4ed7-990a-455598d4514f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ff7c956f-xvmcg\" (UID: \"68c04a19-7b8d-4ed7-990a-455598d4514f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff7c956f-xvmcg" Nov 03 14:07:54.723147 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:54.723010 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5kkss\" (UniqueName: \"kubernetes.io/projected/68c04a19-7b8d-4ed7-990a-455598d4514f-kube-api-access-5kkss\") pod \"maas-api-revoke-sub-ff7c956f-xvmcg\" (UID: \"68c04a19-7b8d-4ed7-990a-455598d4514f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff7c956f-xvmcg" Nov 03 14:07:54.723147 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:54.723047 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68c04a19-7b8d-4ed7-990a-455598d4514f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ff7c956f-xvmcg\" (UID: \"68c04a19-7b8d-4ed7-990a-455598d4514f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff7c956f-xvmcg" Nov 03 14:07:54.723691 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:54.723667 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68c04a19-7b8d-4ed7-990a-455598d4514f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ff7c956f-xvmcg\" (UID: \"68c04a19-7b8d-4ed7-990a-455598d4514f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff7c956f-xvmcg" Nov 03 14:07:54.725299 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:54.725266 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68c04a19-7b8d-4ed7-990a-455598d4514f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ff7c956f-xvmcg\" (UID: \"68c04a19-7b8d-4ed7-990a-455598d4514f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff7c956f-xvmcg" Nov 03 14:07:54.732588 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:54.732558 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5kkss\" (UniqueName: \"kubernetes.io/projected/68c04a19-7b8d-4ed7-990a-455598d4514f-kube-api-access-5kkss\") pod \"maas-api-revoke-sub-ff7c956f-xvmcg\" (UID: \"68c04a19-7b8d-4ed7-990a-455598d4514f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff7c956f-xvmcg" Nov 03 14:07:54.896796 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:54.896758 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff7c956f-xvmcg" Nov 03 14:07:55.174967 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:55.174933 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ff7c956f-xvmcg"] Nov 03 14:07:55.176611 ip-10-0-131-36 kubenswrapper[5024]: W1103 14:07:55.176578 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68c04a19_7b8d_4ed7_990a_455598d4514f.slice/crio-86d8363315c0f03c233e94f47dd13c5b135c565acb65d29ea7a6d29392c73556 WatchSource:0}: Error finding container 86d8363315c0f03c233e94f47dd13c5b135c565acb65d29ea7a6d29392c73556: Status 404 returned error can't find the container with id 86d8363315c0f03c233e94f47dd13c5b135c565acb65d29ea7a6d29392c73556 Nov 03 14:07:55.548353 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:55.548318 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff7c956f-xvmcg" event={"ID":"68c04a19-7b8d-4ed7-990a-455598d4514f","Type":"ContainerStarted","Data":"317b0a3fe6c631b65b3aea04fc3058b0df65082c7c2f85ac5b07b66e5316ef32"} Nov 03 14:07:55.548510 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:55.548371 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff7c956f-xvmcg" event={"ID":"68c04a19-7b8d-4ed7-990a-455598d4514f","Type":"ContainerStarted","Data":"86d8363315c0f03c233e94f47dd13c5b135c565acb65d29ea7a6d29392c73556"} Nov 03 14:07:55.568758 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:55.568709 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff7c956f-xvmcg" podStartSLOduration=1.568697462 podStartE2EDuration="1.568697462s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:07:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:07:55.567235631 +0000 UTC m=+3555.575426911" watchObservedRunningTime="2026-11-03 14:07:55.568697462 +0000 UTC m=+3555.576888739" Nov 03 14:07:56.555975 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:56.555934 5024 generic.go:356] "Generic (PLEG): container finished" podID="68c04a19-7b8d-4ed7-990a-455598d4514f" containerID="317b0a3fe6c631b65b3aea04fc3058b0df65082c7c2f85ac5b07b66e5316ef32" exitCode=0 Nov 03 14:07:56.556406 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:56.555989 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff7c956f-xvmcg" event={"ID":"68c04a19-7b8d-4ed7-990a-455598d4514f","Type":"ContainerDied","Data":"317b0a3fe6c631b65b3aea04fc3058b0df65082c7c2f85ac5b07b66e5316ef32"} Nov 03 14:07:57.686082 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:57.686057 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff7c956f-xvmcg" Nov 03 14:07:57.751989 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:57.751958 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/68c04a19-7b8d-4ed7-990a-455598d4514f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68c04a19-7b8d-4ed7-990a-455598d4514f-maas-api-openshift-ca\") pod \"68c04a19-7b8d-4ed7-990a-455598d4514f\" (UID: \"68c04a19-7b8d-4ed7-990a-455598d4514f\") " Nov 03 14:07:57.752123 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:57.752001 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68c04a19-7b8d-4ed7-990a-455598d4514f-kube-api-access-5kkss\" (UniqueName: \"kubernetes.io/projected/68c04a19-7b8d-4ed7-990a-455598d4514f-kube-api-access-5kkss\") pod \"68c04a19-7b8d-4ed7-990a-455598d4514f\" (UID: \"68c04a19-7b8d-4ed7-990a-455598d4514f\") " Nov 03 14:07:57.752123 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:57.752043 5024 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/68c04a19-7b8d-4ed7-990a-455598d4514f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68c04a19-7b8d-4ed7-990a-455598d4514f-maas-api-xks-ca\") pod \"68c04a19-7b8d-4ed7-990a-455598d4514f\" (UID: \"68c04a19-7b8d-4ed7-990a-455598d4514f\") " Nov 03 14:07:57.752343 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:57.752319 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68c04a19-7b8d-4ed7-990a-455598d4514f-maas-api-openshift-ca" pod "68c04a19-7b8d-4ed7-990a-455598d4514f" (UID: "68c04a19-7b8d-4ed7-990a-455598d4514f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:07:57.754034 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:57.754014 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/68c04a19-7b8d-4ed7-990a-455598d4514f-maas-api-xks-ca" pod "68c04a19-7b8d-4ed7-990a-455598d4514f" (UID: "68c04a19-7b8d-4ed7-990a-455598d4514f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:07:57.754107 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:57.754083 5024 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68c04a19-7b8d-4ed7-990a-455598d4514f-kube-api-access-5kkss" pod "68c04a19-7b8d-4ed7-990a-455598d4514f" (UID: "68c04a19-7b8d-4ed7-990a-455598d4514f"). InnerVolumeSpecName "kube-api-access-5kkss". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:07:57.853639 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:57.853612 5024 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68c04a19-7b8d-4ed7-990a-455598d4514f-maas-api-openshift-ca\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 14:07:57.853753 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:57.853641 5024 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5kkss\" (UniqueName: \"kubernetes.io/projected/68c04a19-7b8d-4ed7-990a-455598d4514f-kube-api-access-5kkss\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 14:07:57.853753 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:57.853651 5024 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68c04a19-7b8d-4ed7-990a-455598d4514f-maas-api-xks-ca\") on node \"ip-10-0-131-36.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 14:07:58.568219 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:58.568129 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff7c956f-xvmcg" event={"ID":"68c04a19-7b8d-4ed7-990a-455598d4514f","Type":"ContainerDied","Data":"86d8363315c0f03c233e94f47dd13c5b135c565acb65d29ea7a6d29392c73556"} Nov 03 14:07:58.568219 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:58.568193 5024 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="86d8363315c0f03c233e94f47dd13c5b135c565acb65d29ea7a6d29392c73556" Nov 03 14:07:58.568219 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:07:58.568149 5024 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff7c956f-xvmcg" Nov 03 14:08:01.021736 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:08:01.021702 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-4xdw9"] Nov 03 14:08:01.024339 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:08:01.024314 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-4xdw9"] Nov 03 14:08:02.106687 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:08:02.106655 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e0e844e0-9520-47c9-84a0-5bf65c116a54" path="/var/lib/kubelet/pods/e0e844e0-9520-47c9-84a0-5bf65c116a54/volumes" Nov 03 14:08:09.509114 ip-10-0-131-36 kubenswrapper[5024]: E1103 14:08:09.509077 5024 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d7mwz" containerName="extractor" Nov 03 14:08:17.026352 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:08:17.026316 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-w9vnh"] Nov 03 14:08:17.032949 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:08:17.032923 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-w9vnh"] Nov 03 14:08:18.107394 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:08:18.107355 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a1deed4-a72f-4ac9-9911-9213c507b151" path="/var/lib/kubelet/pods/9a1deed4-a72f-4ac9-9911-9213c507b151/volumes" Nov 03 14:08:32.022296 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:08:32.022244 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-s79rx"] Nov 03 14:08:32.025614 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:08:32.025588 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-s79rx"] Nov 03 14:08:32.106958 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:08:32.106922 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="99703a90-80bd-4e1f-87f6-60541fa89cb2" path="/var/lib/kubelet/pods/99703a90-80bd-4e1f-87f6-60541fa89cb2/volumes" Nov 03 14:08:39.507432 ip-10-0-131-36 kubenswrapper[5024]: E1103 14:08:39.507383 5024 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d7mwz" containerName="extractor" Nov 03 14:08:41.561063 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:08:41.561030 5024 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6htht_cd91de08-084e-4667-b13f-2e07b3cff624/console-operator/1.log" Nov 03 14:08:41.572826 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:08:41.572802 5024 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6htht_cd91de08-084e-4667-b13f-2e07b3cff624/console-operator/1.log" Nov 03 14:08:47.031172 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:08:47.031136 5024 scope.go:122] "RemoveContainer" containerID="ed19b009b9dba53c066928dcec16837b50544670cc8a26c7c0e64b97c600e902" Nov 03 14:08:47.040272 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:08:47.040250 5024 scope.go:122] "RemoveContainer" containerID="7f17286da1c3782e96f75a1a79312d92b9302a703355e2c632e7c54fdc91bf4f" Nov 03 14:08:47.049161 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:08:47.049139 5024 scope.go:122] "RemoveContainer" containerID="dd8911be1bdaf8fda5c87d9f6133d109b50174b989d17e2f88f669bd08ca7009" Nov 03 14:09:09.506588 ip-10-0-131-36 kubenswrapper[5024]: E1103 14:09:09.506489 5024 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d7mwz" containerName="extractor" Nov 03 14:09:23.064606 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:09:23.064564 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-v4dgr"] Nov 03 14:09:23.071120 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:09:23.071088 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-v4dgr"] Nov 03 14:09:24.106857 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:09:24.106823 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9dc93a95-b62a-4887-8b2b-3ff84104a3ea" path="/var/lib/kubelet/pods/9dc93a95-b62a-4887-8b2b-3ff84104a3ea/volumes" Nov 03 14:09:33.023014 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:09:33.022972 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-2ncfv"] Nov 03 14:09:33.026311 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:09:33.026268 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-2ncfv"] Nov 03 14:09:34.106951 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:09:34.106914 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b39ad908-ed87-45cc-9658-d5a2151a1c3e" path="/var/lib/kubelet/pods/b39ad908-ed87-45cc-9658-d5a2151a1c3e/volumes" Nov 03 14:09:39.508180 ip-10-0-131-36 kubenswrapper[5024]: E1103 14:09:39.508137 5024 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d7mwz" containerName="extractor" Nov 03 14:09:47.081846 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:09:47.081801 5024 scope.go:122] "RemoveContainer" containerID="9bfc2f749f207ae75018b48ad3757e6091aae4bd3c40632dbf1ef93929be9748" Nov 03 14:09:47.090660 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:09:47.090633 5024 scope.go:122] "RemoveContainer" containerID="a7c61b0a6e258cf284b2069a319d7c7c8e059fc7ebf65ee6e09371172ea5cfcb" Nov 03 14:09:47.105672 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:09:47.105429 5024 scope.go:122] "RemoveContainer" containerID="768ef937bfe02a72c017e6e7efe1090563ef520f51fce78c152c3c1a4ed8f824" Nov 03 14:09:47.118501 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:09:47.117993 5024 scope.go:122] "RemoveContainer" containerID="bf55f05d0c3d793d5fd614d78363905e3f74990f87abef2d3fe489d5e722d9dc" Nov 03 14:10:09.507802 ip-10-0-131-36 kubenswrapper[5024]: E1103 14:10:09.507765 5024 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d7mwz" containerName="extractor" Nov 03 14:10:39.506895 ip-10-0-131-36 kubenswrapper[5024]: E1103 14:10:39.506813 5024 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d7mwz" containerName="extractor" Nov 03 14:10:47.215556 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:10:47.215520 5024 scope.go:122] "RemoveContainer" containerID="e2e4e596c0abde1f96a61125780b8b7e85496545012767a9a71e87f67eebe3d2" Nov 03 14:10:47.224254 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:10:47.224226 5024 scope.go:122] "RemoveContainer" containerID="de2f02d4a7e670630a2c5e7c247283884d282128375428d9956fd026fb70693a" Nov 03 14:10:47.231850 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:10:47.231831 5024 scope.go:122] "RemoveContainer" containerID="a37f2f00ca631d1290d5eace140d67b46db2c820465cc1bf75f4aa4797421285" Nov 03 14:10:47.241246 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:10:47.241227 5024 scope.go:122] "RemoveContainer" containerID="75bdbbbd68f417e0264e4af6dfcccf3e824a654fe429b7b6465a8e102c07bfb0" Nov 03 14:11:09.507828 ip-10-0-131-36 kubenswrapper[5024]: E1103 14:11:09.507788 5024 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d7mwz" containerName="extractor" Nov 03 14:11:39.507613 ip-10-0-131-36 kubenswrapper[5024]: E1103 14:11:39.507574 5024 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d7mwz" containerName="extractor" Nov 03 14:12:07.028633 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:12:07.028590 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fmzgf"] Nov 03 14:12:07.033745 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:12:07.033712 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fmzgf"] Nov 03 14:12:08.107189 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:12:08.107114 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="26dd913b-c146-4745-9c18-3dbafa15c763" path="/var/lib/kubelet/pods/26dd913b-c146-4745-9c18-3dbafa15c763/volumes" Nov 03 14:12:09.507680 ip-10-0-131-36 kubenswrapper[5024]: E1103 14:12:09.507632 5024 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d7mwz" containerName="extractor" Nov 03 14:12:27.040441 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:12:27.040400 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-jkjxs"] Nov 03 14:12:27.050389 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:12:27.050359 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-jkjxs"] Nov 03 14:12:28.107150 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:12:28.107114 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef741c0f-88eb-4aaf-a1aa-f55329a8695e" path="/var/lib/kubelet/pods/ef741c0f-88eb-4aaf-a1aa-f55329a8695e/volumes" Nov 03 14:12:39.523274 ip-10-0-131-36 kubenswrapper[5024]: E1103 14:12:39.523196 5024 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d7mwz" containerName="extractor" Nov 03 14:12:41.293472 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:12:41.293418 5024 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-92vh9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.173:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 14:12:41.294177 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:12:41.294134 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-92vh9" podUID="825941b6-3ed3-47f2-a859-c891c4f29434" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.173:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 14:12:41.310432 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:12:41.310403 5024 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-zndxf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.172:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 14:12:41.310741 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:12:41.310671 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-zndxf" podUID="73f8f24f-b955-433b-95ed-5f441f7fceb2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.172:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 14:12:41.340698 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:12:41.340652 5024 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-k4c4j container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.174:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 14:12:41.340959 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:12:41.340914 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-k4c4j" podUID="4b70e366-6fba-4bfa-ac3e-0f0c7b0b64d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.174:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 14:12:58.026316 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:12:58.026266 5024 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ff7c956f-xvmcg"] Nov 03 14:12:58.031247 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:12:58.031219 5024 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ff7c956f-xvmcg"] Nov 03 14:12:58.107924 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:12:58.107888 5024 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="68c04a19-7b8d-4ed7-990a-455598d4514f" path="/var/lib/kubelet/pods/68c04a19-7b8d-4ed7-990a-455598d4514f/volumes" Nov 03 14:13:09.509044 ip-10-0-131-36 kubenswrapper[5024]: E1103 14:13:09.508997 5024 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d7mwz" containerName="extractor" Nov 03 14:13:26.306453 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:13:26.306397 5024 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-92vh9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.173:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 14:13:26.307261 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:13:26.307223 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-92vh9" podUID="825941b6-3ed3-47f2-a859-c891c4f29434" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.173:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 14:13:26.311315 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:13:26.309368 5024 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-zndxf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.172:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 14:13:26.311315 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:13:26.309454 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-zndxf" podUID="73f8f24f-b955-433b-95ed-5f441f7fceb2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.172:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 14:13:26.354310 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:13:26.353397 5024 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-k4c4j container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.174:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 14:13:26.354310 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:13:26.353500 5024 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-k4c4j" podUID="4b70e366-6fba-4bfa-ac3e-0f0c7b0b64d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.174:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 14:13:39.506487 ip-10-0-131-36 kubenswrapper[5024]: E1103 14:13:39.506404 5024 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d7mwz" containerName="extractor" Nov 03 14:13:41.599656 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:13:41.599621 5024 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6htht_cd91de08-084e-4667-b13f-2e07b3cff624/console-operator/1.log" Nov 03 14:13:41.605845 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:13:41.605822 5024 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6htht_cd91de08-084e-4667-b13f-2e07b3cff624/console-operator/1.log" Nov 03 14:13:47.298267 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:13:47.298231 5024 scope.go:122] "RemoveContainer" containerID="c4ce8bbd3cfb4f9304c5b2c0da2fbcb1b4a6ad4d8c5ebb0d8a255a80e83a781e" Nov 03 14:13:47.307039 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:13:47.307016 5024 scope.go:122] "RemoveContainer" containerID="544fede72341b96ce68d93c195908b724ed64037e3276ad9ebcf631c00a9aa3b" Nov 03 14:13:59.432771 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:13:59.432735 5024 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-796d65f566-65br2_795bfdb5-d443-4daf-a0d1-e0bb7567416c/authorino/0.log" Nov 03 14:14:09.508400 ip-10-0-131-36 kubenswrapper[5024]: E1103 14:14:09.508355 5024 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d7mwz" containerName="extractor" Nov 03 14:14:10.880949 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:14:10.880913 5024 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-6f5b4db54-wxw9w_9af794f7-e3d1-4049-9828-499e3a49a290/manager/0.log" Nov 03 14:14:11.785301 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:14:11.785254 5024 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-bc865c477-q4c9z_75c9ed72-4fa6-4d71-a8c9-6453fb9e2c1f/manager/0.log" Nov 03 14:14:13.435571 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:14:13.435536 5024 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-7d6dc67969-zphs9_96223074-b4c8-4855-88a9-34f3851c8e8b/maas-api/0.log" Nov 03 14:14:16.603780 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:14:16.603750 5024 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-796d65f566-65br2_795bfdb5-d443-4daf-a0d1-e0bb7567416c/authorino/0.log" Nov 03 14:14:17.458743 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:14:17.458713 5024 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-gpqgh_6178a6a9-df55-4825-9df9-32a7f2bccad8/manager/0.log" Nov 03 14:14:19.197030 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:14:19.196995 5024 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-84d8c65c8f-j592j_24d19d80-a3b9-468e-9aff-035a62a578fe/kube-auth-proxy/0.log" Nov 03 14:14:19.734600 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:14:19.734570 5024 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-84587756dc-92vh9_825941b6-3ed3-47f2-a859-c891c4f29434/istio-proxy/0.log" Nov 03 14:14:19.936906 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:14:19.936871 5024 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-84587756dc-k4c4j_4b70e366-6fba-4bfa-ac3e-0f0c7b0b64d3/istio-proxy/0.log" Nov 03 14:14:20.674779 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:14:20.674746 5024 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-84587756dc-vrbqc_d12a0b88-93e3-4542-90b0-56a37dd91387/istio-proxy/0.log" Nov 03 14:14:21.242672 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:14:21.242637 5024 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-84587756dc-zndxf_73f8f24f-b955-433b-95ed-5f441f7fceb2/istio-proxy/0.log" Nov 03 14:14:21.610370 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:14:21.610343 5024 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-67f556869f-kv4cs_bad6aa6e-a56a-4955-9237-6f254c967c37/payload-pre-processing/0.log" Nov 03 14:14:21.772797 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:14:21.772768 5024 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-776d6d99b-q7jqc_b0918520-a9f4-4ebf-8c93-301ac3d62e8f/payload-processing/0.log" Nov 03 14:14:21.943815 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:14:21.943740 5024 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7f579888d9-ntt6m_898bd0b5-0d3e-4577-b101-2cc7d75283cb/router/0.log" Nov 03 14:14:22.906590 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:14:22.906557 5024 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-xpkhb_b0c8b366-13df-493b-a519-10797d27e14c/main/0.log" Nov 03 14:14:23.091513 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:14:23.091471 5024 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-q6cdk_510862f1-3997-4a93-b4a9-75e3855154dc/main/0.log" Nov 03 14:14:23.609909 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:14:23.609875 5024 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8h566d_6428a275-67b4-43e6-be9c-a16728516866/main/0.log" Nov 03 14:14:23.783489 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:14:23.783457 5024 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-ghn7b_0cf9c918-efa8-490a-8a25-069337ee09b0/main/0.log" Nov 03 14:14:34.395215 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:14:34.395178 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-tqgz9/must-gather-rh8xx"] Nov 03 14:14:34.395838 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:14:34.395709 5024 cpu_manager.go:414] "containerMap: removing container" podUID="68c04a19-7b8d-4ed7-990a-455598d4514f" containerName="revoke-keys" Nov 03 14:14:34.395838 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:14:34.395727 5024 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68c04a19-7b8d-4ed7-990a-455598d4514f" containerName="revoke-keys" Nov 03 14:14:34.395838 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:14:34.395823 5024 memory_manager.go:365] "RemoveStaleState removing state" podUID="68c04a19-7b8d-4ed7-990a-455598d4514f" containerName="revoke-keys" Nov 03 14:14:34.399092 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:14:34.399073 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-tqgz9/must-gather-rh8xx" Nov 03 14:14:34.406691 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:14:34.406669 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-tqgz9/must-gather-rh8xx"] Nov 03 14:14:34.408186 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:14:34.408162 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-tqgz9\"/\"kube-root-ca.crt\"" Nov 03 14:14:34.408336 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:14:34.408204 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-tqgz9\"/\"openshift-service-ca.crt\"" Nov 03 14:14:34.410245 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:14:34.410226 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-tqgz9\"/\"default-dockercfg-nxzft\"" Nov 03 14:14:34.461053 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:14:34.461017 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vkp8\" (UniqueName: \"kubernetes.io/projected/66c6e2dc-d38e-44fc-88a5-40be5e6edaa5-kube-api-access-5vkp8\") pod \"must-gather-rh8xx\" (UID: \"66c6e2dc-d38e-44fc-88a5-40be5e6edaa5\") " pod="openshift-must-gather-tqgz9/must-gather-rh8xx" Nov 03 14:14:34.461223 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:14:34.461171 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/66c6e2dc-d38e-44fc-88a5-40be5e6edaa5-must-gather-output\") pod \"must-gather-rh8xx\" (UID: \"66c6e2dc-d38e-44fc-88a5-40be5e6edaa5\") " pod="openshift-must-gather-tqgz9/must-gather-rh8xx" Nov 03 14:14:34.484142 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:14:34.484112 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-vtpt6/must-gather-82m2s"] Nov 03 14:14:34.487702 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:14:34.487684 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-vtpt6/must-gather-82m2s" Nov 03 14:14:34.495231 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:14:34.495204 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-vtpt6/must-gather-82m2s"] Nov 03 14:14:34.508803 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:14:34.508604 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-vtpt6\"/\"openshift-service-ca.crt\"" Nov 03 14:14:34.508803 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:14:34.508626 5024 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-vtpt6\"/\"kube-root-ca.crt\"" Nov 03 14:14:34.510720 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:14:34.510700 5024 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-vtpt6\"/\"default-dockercfg-vkdhx\"" Nov 03 14:14:34.562366 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:14:34.562334 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/9ec158cc-9bbb-4718-859b-6b9e3afce55a-must-gather-output\") pod \"must-gather-82m2s\" (UID: \"9ec158cc-9bbb-4718-859b-6b9e3afce55a\") " pod="openshift-must-gather-vtpt6/must-gather-82m2s" Nov 03 14:14:34.562533 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:14:34.562391 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/66c6e2dc-d38e-44fc-88a5-40be5e6edaa5-must-gather-output\") pod \"must-gather-rh8xx\" (UID: \"66c6e2dc-d38e-44fc-88a5-40be5e6edaa5\") " pod="openshift-must-gather-tqgz9/must-gather-rh8xx" Nov 03 14:14:34.562533 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:14:34.562417 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-57sb8\" (UniqueName: \"kubernetes.io/projected/9ec158cc-9bbb-4718-859b-6b9e3afce55a-kube-api-access-57sb8\") pod \"must-gather-82m2s\" (UID: \"9ec158cc-9bbb-4718-859b-6b9e3afce55a\") " pod="openshift-must-gather-vtpt6/must-gather-82m2s" Nov 03 14:14:34.562667 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:14:34.562554 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vkp8\" (UniqueName: \"kubernetes.io/projected/66c6e2dc-d38e-44fc-88a5-40be5e6edaa5-kube-api-access-5vkp8\") pod \"must-gather-rh8xx\" (UID: \"66c6e2dc-d38e-44fc-88a5-40be5e6edaa5\") " pod="openshift-must-gather-tqgz9/must-gather-rh8xx" Nov 03 14:14:34.562797 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:14:34.562773 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/66c6e2dc-d38e-44fc-88a5-40be5e6edaa5-must-gather-output\") pod \"must-gather-rh8xx\" (UID: \"66c6e2dc-d38e-44fc-88a5-40be5e6edaa5\") " pod="openshift-must-gather-tqgz9/must-gather-rh8xx" Nov 03 14:14:34.572567 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:14:34.572545 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vkp8\" (UniqueName: \"kubernetes.io/projected/66c6e2dc-d38e-44fc-88a5-40be5e6edaa5-kube-api-access-5vkp8\") pod \"must-gather-rh8xx\" (UID: \"66c6e2dc-d38e-44fc-88a5-40be5e6edaa5\") " pod="openshift-must-gather-tqgz9/must-gather-rh8xx" Nov 03 14:14:34.663454 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:14:34.663370 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/9ec158cc-9bbb-4718-859b-6b9e3afce55a-must-gather-output\") pod \"must-gather-82m2s\" (UID: \"9ec158cc-9bbb-4718-859b-6b9e3afce55a\") " pod="openshift-must-gather-vtpt6/must-gather-82m2s" Nov 03 14:14:34.663454 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:14:34.663428 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57sb8\" (UniqueName: \"kubernetes.io/projected/9ec158cc-9bbb-4718-859b-6b9e3afce55a-kube-api-access-57sb8\") pod \"must-gather-82m2s\" (UID: \"9ec158cc-9bbb-4718-859b-6b9e3afce55a\") " pod="openshift-must-gather-vtpt6/must-gather-82m2s" Nov 03 14:14:34.663738 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:14:34.663715 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/9ec158cc-9bbb-4718-859b-6b9e3afce55a-must-gather-output\") pod \"must-gather-82m2s\" (UID: \"9ec158cc-9bbb-4718-859b-6b9e3afce55a\") " pod="openshift-must-gather-vtpt6/must-gather-82m2s" Nov 03 14:14:34.673165 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:14:34.673131 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-57sb8\" (UniqueName: \"kubernetes.io/projected/9ec158cc-9bbb-4718-859b-6b9e3afce55a-kube-api-access-57sb8\") pod \"must-gather-82m2s\" (UID: \"9ec158cc-9bbb-4718-859b-6b9e3afce55a\") " pod="openshift-must-gather-vtpt6/must-gather-82m2s" Nov 03 14:14:34.719213 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:14:34.719182 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-tqgz9/must-gather-rh8xx" Nov 03 14:14:34.809363 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:14:34.804637 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-vtpt6/must-gather-82m2s" Nov 03 14:14:34.999510 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:14:34.999480 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-tqgz9/must-gather-rh8xx"] Nov 03 14:14:35.000813 ip-10-0-131-36 kubenswrapper[5024]: W1103 14:14:35.000784 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod66c6e2dc_d38e_44fc_88a5_40be5e6edaa5.slice/crio-6c84a7ba5e606bd6e21d37519eb66575778e53bf06fcdae9a5fd410a4582e0e1 WatchSource:0}: Error finding container 6c84a7ba5e606bd6e21d37519eb66575778e53bf06fcdae9a5fd410a4582e0e1: Status 404 returned error can't find the container with id 6c84a7ba5e606bd6e21d37519eb66575778e53bf06fcdae9a5fd410a4582e0e1 Nov 03 14:14:35.002700 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:14:35.002683 5024 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 14:14:35.082082 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:14:35.082054 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-vtpt6/must-gather-82m2s"] Nov 03 14:14:35.083960 ip-10-0-131-36 kubenswrapper[5024]: W1103 14:14:35.083934 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9ec158cc_9bbb_4718_859b_6b9e3afce55a.slice/crio-f8dac449e5d241bb86789458ab82fca52ec4e7367c5f56e36ef91b89d31b6a89 WatchSource:0}: Error finding container f8dac449e5d241bb86789458ab82fca52ec4e7367c5f56e36ef91b89d31b6a89: Status 404 returned error can't find the container with id f8dac449e5d241bb86789458ab82fca52ec4e7367c5f56e36ef91b89d31b6a89 Nov 03 14:14:35.814545 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:14:35.814500 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-vtpt6/must-gather-82m2s" event={"ID":"9ec158cc-9bbb-4718-859b-6b9e3afce55a","Type":"ContainerStarted","Data":"f8dac449e5d241bb86789458ab82fca52ec4e7367c5f56e36ef91b89d31b6a89"} Nov 03 14:14:35.815955 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:14:35.815919 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-tqgz9/must-gather-rh8xx" event={"ID":"66c6e2dc-d38e-44fc-88a5-40be5e6edaa5","Type":"ContainerStarted","Data":"6c84a7ba5e606bd6e21d37519eb66575778e53bf06fcdae9a5fd410a4582e0e1"} Nov 03 14:14:36.842346 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:14:36.842298 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-vtpt6/must-gather-82m2s" event={"ID":"9ec158cc-9bbb-4718-859b-6b9e3afce55a","Type":"ContainerStarted","Data":"bf1a06010e9759b4fbba07591693aef1128cc59153b6bf68d4ee58d8191abb8c"} Nov 03 14:14:36.842848 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:14:36.842366 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-vtpt6/must-gather-82m2s" event={"ID":"9ec158cc-9bbb-4718-859b-6b9e3afce55a","Type":"ContainerStarted","Data":"d5947c08a846bd6e7f0025baff96fefe62ea70994358281e655fe7e1214839c0"} Nov 03 14:14:38.593969 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:14:38.593912 5024 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-5jz7t_7e280277-f29f-4a2b-8b85-2491529e5ef2/global-pull-secret-syncer/0.log" Nov 03 14:14:38.750112 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:14:38.749997 5024 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-hslrf_b8fe3220-d285-487f-9fc0-597438982de7/connection-checker/0.log" Nov 03 14:14:38.784690 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:14:38.784506 5024 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-vl8fr_07a7e2de-0c09-4819-9670-2be9460a155d/connection-checker/0.log" Nov 03 14:14:38.819675 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:14:38.819646 5024 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-w5cql_c82d9f4b-8c00-4c31-8aa2-99cdc0d87e52/connection-checker/0.log" Nov 03 14:14:38.955528 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:14:38.955493 5024 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-bdg96_246971bc-ed35-4fc7-b5f8-941327b90a94/konnectivity-agent/0.log" Nov 03 14:14:39.078069 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:14:39.078039 5024 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-131-36.us-east-2.compute.internal_5ee88558fb38141d796a31757de861ab/haproxy/0.log" Nov 03 14:14:39.519423 ip-10-0-131-36 kubenswrapper[5024]: E1103 14:14:39.519381 5024 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d7mwz" containerName="extractor" Nov 03 14:14:46.633682 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:14:46.633640 5024 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-796d65f566-65br2_795bfdb5-d443-4daf-a0d1-e0bb7567416c/authorino/0.log" Nov 03 14:14:47.331151 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:14:47.331024 5024 scope.go:122] "RemoveContainer" containerID="317b0a3fe6c631b65b3aea04fc3058b0df65082c7c2f85ac5b07b66e5316ef32" Nov 03 14:14:47.904836 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:14:47.904644 5024 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-gpqgh_6178a6a9-df55-4825-9df9-32a7f2bccad8/manager/0.log" Nov 03 14:14:49.781848 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:14:49.781808 5024 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-884576c57-tm4nb_a3042b60-062f-4f9b-8406-f884c95a8729/cluster-monitoring-operator/0.log" Nov 03 14:14:49.915712 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:14:49.915676 5024 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-86b957855b-5k7lz_bddb955a-a11f-4cb4-b25e-ef2ed77c98ba/metrics-server/0.log" Nov 03 14:14:49.956791 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:14:49.956720 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-vtpt6/must-gather-82m2s" podStartSLOduration=15.144083648 podStartE2EDuration="15.956696905s" totalImagesPullingTime="812.613257ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:14:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:14:36.866454583 +0000 UTC m=+3956.874645885" watchObservedRunningTime="2026-11-03 14:14:49.956696905 +0000 UTC m=+3969.964888183" Nov 03 14:14:49.957632 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:14:49.957611 5024 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-vtpt6/perf-node-gather-daemonset-9vxzp"] Nov 03 14:14:49.999915 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:14:49.999875 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-vtpt6/perf-node-gather-daemonset-9vxzp"] Nov 03 14:14:50.000110 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:14:50.000008 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-vtpt6/perf-node-gather-daemonset-9vxzp" Nov 03 14:14:50.095996 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:14:50.095901 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vt9bb\" (UniqueName: \"kubernetes.io/projected/9814e948-258e-487b-995e-96edfa5d8e50-kube-api-access-vt9bb\") pod \"perf-node-gather-daemonset-9vxzp\" (UID: \"9814e948-258e-487b-995e-96edfa5d8e50\") " pod="openshift-must-gather-vtpt6/perf-node-gather-daemonset-9vxzp" Nov 03 14:14:50.095996 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:14:50.095965 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/9814e948-258e-487b-995e-96edfa5d8e50-podres\") pod \"perf-node-gather-daemonset-9vxzp\" (UID: \"9814e948-258e-487b-995e-96edfa5d8e50\") " pod="openshift-must-gather-vtpt6/perf-node-gather-daemonset-9vxzp" Nov 03 14:14:50.096212 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:14:50.096102 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/9814e948-258e-487b-995e-96edfa5d8e50-proc\") pod \"perf-node-gather-daemonset-9vxzp\" (UID: \"9814e948-258e-487b-995e-96edfa5d8e50\") " pod="openshift-must-gather-vtpt6/perf-node-gather-daemonset-9vxzp" Nov 03 14:14:50.096212 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:14:50.096134 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9814e948-258e-487b-995e-96edfa5d8e50-lib-modules\") pod \"perf-node-gather-daemonset-9vxzp\" (UID: \"9814e948-258e-487b-995e-96edfa5d8e50\") " pod="openshift-must-gather-vtpt6/perf-node-gather-daemonset-9vxzp" Nov 03 14:14:50.096323 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:14:50.096249 5024 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9814e948-258e-487b-995e-96edfa5d8e50-sys\") pod \"perf-node-gather-daemonset-9vxzp\" (UID: \"9814e948-258e-487b-995e-96edfa5d8e50\") " pod="openshift-must-gather-vtpt6/perf-node-gather-daemonset-9vxzp" Nov 03 14:14:50.134494 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:14:50.134451 5024 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-r9pzz_dbe43d42-0886-402c-bc38-080acd180803/node-exporter/0.log" Nov 03 14:14:50.165228 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:14:50.165179 5024 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-r9pzz_dbe43d42-0886-402c-bc38-080acd180803/kube-rbac-proxy/0.log" Nov 03 14:14:50.196438 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:14:50.196406 5024 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-r9pzz_dbe43d42-0886-402c-bc38-080acd180803/init-textfile/0.log" Nov 03 14:14:50.197358 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:14:50.197164 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/9814e948-258e-487b-995e-96edfa5d8e50-podres\") pod \"perf-node-gather-daemonset-9vxzp\" (UID: \"9814e948-258e-487b-995e-96edfa5d8e50\") " pod="openshift-must-gather-vtpt6/perf-node-gather-daemonset-9vxzp" Nov 03 14:14:50.197526 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:14:50.197243 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/9814e948-258e-487b-995e-96edfa5d8e50-podres\") pod \"perf-node-gather-daemonset-9vxzp\" (UID: \"9814e948-258e-487b-995e-96edfa5d8e50\") " pod="openshift-must-gather-vtpt6/perf-node-gather-daemonset-9vxzp" Nov 03 14:14:50.197526 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:14:50.197508 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/9814e948-258e-487b-995e-96edfa5d8e50-proc\") pod \"perf-node-gather-daemonset-9vxzp\" (UID: \"9814e948-258e-487b-995e-96edfa5d8e50\") " pod="openshift-must-gather-vtpt6/perf-node-gather-daemonset-9vxzp" Nov 03 14:14:50.197676 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:14:50.197551 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9814e948-258e-487b-995e-96edfa5d8e50-lib-modules\") pod \"perf-node-gather-daemonset-9vxzp\" (UID: \"9814e948-258e-487b-995e-96edfa5d8e50\") " pod="openshift-must-gather-vtpt6/perf-node-gather-daemonset-9vxzp" Nov 03 14:14:50.197676 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:14:50.197622 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9814e948-258e-487b-995e-96edfa5d8e50-sys\") pod \"perf-node-gather-daemonset-9vxzp\" (UID: \"9814e948-258e-487b-995e-96edfa5d8e50\") " pod="openshift-must-gather-vtpt6/perf-node-gather-daemonset-9vxzp" Nov 03 14:14:50.197793 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:14:50.197717 5024 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vt9bb\" (UniqueName: \"kubernetes.io/projected/9814e948-258e-487b-995e-96edfa5d8e50-kube-api-access-vt9bb\") pod \"perf-node-gather-daemonset-9vxzp\" (UID: \"9814e948-258e-487b-995e-96edfa5d8e50\") " pod="openshift-must-gather-vtpt6/perf-node-gather-daemonset-9vxzp" Nov 03 14:14:50.197971 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:14:50.197949 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9814e948-258e-487b-995e-96edfa5d8e50-lib-modules\") pod \"perf-node-gather-daemonset-9vxzp\" (UID: \"9814e948-258e-487b-995e-96edfa5d8e50\") " pod="openshift-must-gather-vtpt6/perf-node-gather-daemonset-9vxzp" Nov 03 14:14:50.198127 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:14:50.198101 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9814e948-258e-487b-995e-96edfa5d8e50-sys\") pod \"perf-node-gather-daemonset-9vxzp\" (UID: \"9814e948-258e-487b-995e-96edfa5d8e50\") " pod="openshift-must-gather-vtpt6/perf-node-gather-daemonset-9vxzp" Nov 03 14:14:50.198239 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:14:50.198115 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/9814e948-258e-487b-995e-96edfa5d8e50-proc\") pod \"perf-node-gather-daemonset-9vxzp\" (UID: \"9814e948-258e-487b-995e-96edfa5d8e50\") " pod="openshift-must-gather-vtpt6/perf-node-gather-daemonset-9vxzp" Nov 03 14:14:50.206696 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:14:50.206673 5024 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vt9bb\" (UniqueName: \"kubernetes.io/projected/9814e948-258e-487b-995e-96edfa5d8e50-kube-api-access-vt9bb\") pod \"perf-node-gather-daemonset-9vxzp\" (UID: \"9814e948-258e-487b-995e-96edfa5d8e50\") " pod="openshift-must-gather-vtpt6/perf-node-gather-daemonset-9vxzp" Nov 03 14:14:50.358565 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:14:50.358544 5024 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-vtpt6/perf-node-gather-daemonset-9vxzp" Nov 03 14:14:50.661711 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:14:50.661676 5024 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-vtpt6/perf-node-gather-daemonset-9vxzp"] Nov 03 14:14:50.666977 ip-10-0-131-36 kubenswrapper[5024]: W1103 14:14:50.666941 5024 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod9814e948_258e_487b_995e_96edfa5d8e50.slice/crio-1a32f54744b8084574926d2b59e0e9bf551035d98014c7f30b9c45fe378123fc WatchSource:0}: Error finding container 1a32f54744b8084574926d2b59e0e9bf551035d98014c7f30b9c45fe378123fc: Status 404 returned error can't find the container with id 1a32f54744b8084574926d2b59e0e9bf551035d98014c7f30b9c45fe378123fc Nov 03 14:14:51.032352 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:14:51.031984 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-vtpt6/perf-node-gather-daemonset-9vxzp" event={"ID":"9814e948-258e-487b-995e-96edfa5d8e50","Type":"ContainerStarted","Data":"0da4915338e6b8edfefa82a57f2fb3fb66912c6ad3b78aebc6b49a12d99e6358"} Nov 03 14:14:51.032352 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:14:51.032059 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-vtpt6/perf-node-gather-daemonset-9vxzp" event={"ID":"9814e948-258e-487b-995e-96edfa5d8e50","Type":"ContainerStarted","Data":"1a32f54744b8084574926d2b59e0e9bf551035d98014c7f30b9c45fe378123fc"} Nov 03 14:14:51.032352 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:14:51.032215 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-vtpt6/perf-node-gather-daemonset-9vxzp" Nov 03 14:14:51.037302 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:14:51.034074 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-tqgz9/must-gather-rh8xx" event={"ID":"66c6e2dc-d38e-44fc-88a5-40be5e6edaa5","Type":"ContainerStarted","Data":"d76aea3d74ba06e490873b01c147affc366212ed72ab2bfab760a2d902979818"} Nov 03 14:14:51.037302 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:14:51.034118 5024 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-tqgz9/must-gather-rh8xx" event={"ID":"66c6e2dc-d38e-44fc-88a5-40be5e6edaa5","Type":"ContainerStarted","Data":"7bd69c24b697540ff9949f17791bb14da6923ca46e1c88fa287c7fdb7b505345"} Nov 03 14:14:51.052354 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:14:51.052298 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-vtpt6/perf-node-gather-daemonset-9vxzp" podStartSLOduration=2.05226872 podStartE2EDuration="2.05226872s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 14:14:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:14:51.050674607 +0000 UTC m=+3971.058865886" watchObservedRunningTime="2026-11-03 14:14:51.05226872 +0000 UTC m=+3971.060460016" Nov 03 14:14:51.070980 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:14:51.070918 5024 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-tqgz9/must-gather-rh8xx" podStartSLOduration=1.846412399 podStartE2EDuration="17.070899066s" totalImagesPullingTime="15.224486667s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:14:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:14:51.068168613 +0000 UTC m=+3971.076359891" watchObservedRunningTime="2026-11-03 14:14:51.070899066 +0000 UTC m=+3971.079090345" Nov 03 14:14:52.053780 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:14:52.053748 5024 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-6f5b4db54-wxw9w_9af794f7-e3d1-4049-9828-499e3a49a290/manager/0.log" Nov 03 14:14:52.510757 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:14:52.510664 5024 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-86c478bb88-rz4n9_fa297a30-6da7-4a92-98fb-201323490bb0/networking-console-plugin/0.log" Nov 03 14:14:53.165060 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:14:53.165030 5024 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6htht_cd91de08-084e-4667-b13f-2e07b3cff624/console-operator/1.log" Nov 03 14:14:53.180904 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:14:53.180874 5024 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6htht_cd91de08-084e-4667-b13f-2e07b3cff624/console-operator/2.log" Nov 03 14:14:53.231734 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:14:53.231703 5024 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-bc865c477-q4c9z_75c9ed72-4fa6-4d71-a8c9-6453fb9e2c1f/manager/0.log" Nov 03 14:14:54.550171 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:14:54.550138 5024 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-58c9f5f46d-p6cbj_021e9e00-a973-4726-8f48-02a52c4bae14/volume-data-source-validator/0.log" Nov 03 14:14:56.253832 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:14:56.253760 5024 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-vgnv8_3fdeb9f3-6c67-4390-8073-17d4d5f802b9/dns/0.log" Nov 03 14:14:56.283528 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:14:56.283497 5024 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-vgnv8_3fdeb9f3-6c67-4390-8073-17d4d5f802b9/kube-rbac-proxy/0.log" Nov 03 14:14:56.368641 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:14:56.368587 5024 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-6f5b4db54-wxw9w_9af794f7-e3d1-4049-9828-499e3a49a290/manager/0.log" Nov 03 14:14:56.387837 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:14:56.387807 5024 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-v8gcq_5ea5cf61-ed1c-4f5e-b1a0-5c46c5c3ee18/dns-node-resolver/0.log" Nov 03 14:14:57.057108 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:14:57.057079 5024 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-vtpt6/perf-node-gather-daemonset-9vxzp" Nov 03 14:14:57.169528 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:14:57.169496 5024 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-4tn4j_39e11989-e598-4cea-87d9-26b3cf78a66b/node-ca/0.log" Nov 03 14:14:57.491120 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:14:57.491083 5024 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-bc865c477-q4c9z_75c9ed72-4fa6-4d71-a8c9-6453fb9e2c1f/manager/0.log" Nov 03 14:15:01.063271 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:15:01.063234 5024 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-6f5b4db54-wxw9w_9af794f7-e3d1-4049-9828-499e3a49a290/manager/0.log" Nov 03 14:15:02.766609 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:15:02.766569 5024 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-bc865c477-q4c9z_75c9ed72-4fa6-4d71-a8c9-6453fb9e2c1f/manager/0.log" Nov 03 14:15:02.803381 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:15:02.803343 5024 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-84d8c65c8f-j592j_24d19d80-a3b9-468e-9aff-035a62a578fe/kube-auth-proxy/0.log" Nov 03 14:15:02.943204 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:15:02.943159 5024 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-84587756dc-92vh9_825941b6-3ed3-47f2-a859-c891c4f29434/istio-proxy/0.log" Nov 03 14:15:02.989844 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:15:02.989811 5024 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-84587756dc-k4c4j_4b70e366-6fba-4bfa-ac3e-0f0c7b0b64d3/istio-proxy/0.log" Nov 03 14:15:03.178584 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:15:03.178555 5024 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-84587756dc-vrbqc_d12a0b88-93e3-4542-90b0-56a37dd91387/istio-proxy/0.log" Nov 03 14:15:03.301692 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:15:03.301657 5024 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-84587756dc-zndxf_73f8f24f-b955-433b-95ed-5f441f7fceb2/istio-proxy/0.log" Nov 03 14:15:03.381068 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:15:03.381025 5024 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-67f556869f-kv4cs_bad6aa6e-a56a-4955-9237-6f254c967c37/payload-pre-processing/0.log" Nov 03 14:15:03.420740 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:15:03.420698 5024 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-776d6d99b-q7jqc_b0918520-a9f4-4ebf-8c93-301ac3d62e8f/payload-processing/0.log" Nov 03 14:15:03.468703 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:15:03.468629 5024 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7f579888d9-ntt6m_898bd0b5-0d3e-4577-b101-2cc7d75283cb/router/0.log" Nov 03 14:15:05.421538 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:15:05.421503 5024 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-d8f6l_50d68cb7-b766-4674-b295-8f3229880994/serve-healthcheck-canary/0.log" Nov 03 14:15:05.910616 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:15:05.910588 5024 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-84d8c65c8f-j592j_24d19d80-a3b9-468e-9aff-035a62a578fe/kube-auth-proxy/0.log" Nov 03 14:15:05.989421 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:15:05.989386 5024 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-84587756dc-92vh9_825941b6-3ed3-47f2-a859-c891c4f29434/istio-proxy/0.log" Nov 03 14:15:06.019377 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:15:06.019339 5024 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-84587756dc-k4c4j_4b70e366-6fba-4bfa-ac3e-0f0c7b0b64d3/istio-proxy/0.log" Nov 03 14:15:06.120564 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:15:06.120532 5024 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-84587756dc-vrbqc_d12a0b88-93e3-4542-90b0-56a37dd91387/istio-proxy/0.log" Nov 03 14:15:06.196245 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:15:06.196174 5024 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-84587756dc-zndxf_73f8f24f-b955-433b-95ed-5f441f7fceb2/istio-proxy/0.log" Nov 03 14:15:06.244987 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:15:06.244955 5024 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-67f556869f-kv4cs_bad6aa6e-a56a-4955-9237-6f254c967c37/payload-pre-processing/0.log" Nov 03 14:15:06.268633 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:15:06.268603 5024 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-776d6d99b-q7jqc_b0918520-a9f4-4ebf-8c93-301ac3d62e8f/payload-processing/0.log" Nov 03 14:15:06.307960 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:15:06.305917 5024 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7f579888d9-ntt6m_898bd0b5-0d3e-4577-b101-2cc7d75283cb/router/0.log" Nov 03 14:15:06.378535 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:15:06.378489 5024 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-7c4986d449-4t8lj_9fa6d32d-b30b-4b76-9462-5f3f09443d02/insights-operator/0.log" Nov 03 14:15:06.380451 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:15:06.380414 5024 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-7c4986d449-4t8lj_9fa6d32d-b30b-4b76-9462-5f3f09443d02/insights-operator/1.log" Nov 03 14:15:06.421120 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:15:06.417519 5024 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-d7mwz_fff35d1b-e7b2-4dd3-9824-8b64fad02506/kube-rbac-proxy/0.log" Nov 03 14:15:06.466208 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:15:06.466131 5024 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-d7mwz_fff35d1b-e7b2-4dd3-9824-8b64fad02506/exporter/0.log" Nov 03 14:15:06.505726 ip-10-0-131-36 kubenswrapper[5024]: I1103 14:15:06.505692 5024 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-d7mwz_fff35d1b-e7b2-4dd3-9824-8b64fad02506/extractor/0.log"